Success and kill criteria: how to decide an AI pilot’s fate

What success and kill criteria are in AI pilots

Success criteria describe when an AI pilot is good enough to continue or scale. Kill criteria describe when you stop or redesign the pilot because it is not worth more time, money, or risk. You define both before the pilot starts, together with business, IT, and compliance. This keeps decisions focused on facts instead of opinions or hype. As an AI operations manager, your job is to turn vague expectations into clear, measurable conditions that everyone understands.

Why clear criteria protect your time and budget

AI pilots easily drift into endless experiments. Stakeholders keep hoping that one more tweak will save a weak pilot. Clear success and kill criteria protect you from this. They set expectations early, reduce emotional debates, and help you say no without drama. They also protect users and customers, because you stop pilots that are unsafe or confusing. With good criteria, you can compare pilots fairly and invest in the ones that actually help the business instead of the ones that sound most exciting.

How to define simple, measurable success criteria

Start from the business problem, for example slow response times in customer support. Then choose 1 to 3 core metrics that show improvement, such as average handling time, first contact resolution, or customer satisfaction. Set a realistic target range, for example reduce handling time by 20 to 30 percent without lowering satisfaction. Define a minimum sample size and a fixed test period so you do not cherry pick results. Finally, write down what scaling means in practice, such as roll out to two more teams or integrate into the main tool.

How to set fair and firm kill criteria

Kill criteria should be clear, specific, and agreed in writing. Cover at least four areas: performance, cost, risk, and adoption. Example thresholds: after six weeks, if quality stays below human baseline, or if manual rework exceeds 40 percent of outputs, or if users reject the tool and use workarounds, you stop or redesign. Also include hard safety and compliance rules, for example zero tolerance for leaking personal data. Decide in advance who has the authority to pull the plug so you avoid blame games later.

Common mistakes and how to avoid them

Many teams define criteria that are too vague, like make customer service better, or impossible, like cut costs by 80 percent in one month. Others ignore non functional aspects such as data protection, transparency, or the time needed for human review. A frequent trap is to move the goalposts during the pilot when results are disappointing. Prevent this by writing a short pilot charter that contains the problem, metrics, success criteria, kill criteria, timeframe, and decision owners. Share it with all stakeholders before you start.

Useful links

• OECD AI principles — High level guidance on trustworthy and beneficial AI use
• NIST artificial intelligence — Standards and risk management resources for AI systems

Keep learning