Why pilots stall
A pilot is built to impress. Production is built to be boring. The gap between the two is mostly unglamorous work: integrations, permissions, error handling, monitoring and the people who will own the system on an ordinary Tuesday.
In the session we walked through the patterns we see most. The pilot answers questions in a chat window but can't act in the systems where work happens. Nobody measured the process beforehand, so nobody can prove the pilot helped. And the sponsor sits in IT, while the people whose work would change were never in the room.
Pick a job, not a technology
Pilots that graduate start with a job description. “Draft first replies to tier-one billing tickets” is a job. “Use GenAI in customer service” is a slogan.
A narrow job gives you a test set, a baseline and an obvious owner. It also makes the risk conversation shorter, because you can say exactly what the system will and won't touch.
Measure before you build
Spend the first week timing the current process: how many tickets, how long each takes, how often the answer is wrong. Without those numbers the pilot review turns into a debate about impressions.
Write the success threshold down before any code exists. For example: 60 percent of drafts sent with light edits, and average handling time down by a third.
Design for production from week one
You don't need production infrastructure on day one, but you should know what it will look like. Where will the model run? How will it authenticate to the CRM? Who gets paged when it fails?
Treat prompts, retrieval settings and evaluation sets as code. Version them, review them and run the evaluation on every change.
- Version prompts and evaluation sets alongside the application code
- Log every model call with its inputs, outputs and cost
- Write down what the system may do without approval
- Name the business owner who signs off on releases
Hand ownership to the business
This is the step most often skipped. Production AI needs an owner in the business who reviews quality, decides on scope changes and pushes adoption with their team. IT keeps it running. The business decides what it's for.
