AI deployment checklist

Do your tests cover real user tasks, difficult requests and known failures?

Have you set measurable quality requirements that each release must meet?

Do you rerun quality tests when prompts, models, document search or tools change?

Have people who know the subject checked that your test scores reflect useful, correct answers?

Does the system fail safely when a model, data source or tool is unavailable?

Are there limits on retries, time spent waiting and the number of steps an agent can take?

Can the system decline a request or pass it to a person when it cannot answer safely?

Have you checked that source data is accurate, up to date and complete enough for user requests?

Have you tested whether document search finds the right source information, or confirmed that your system does not need it?

Can you trace factual answers to trusted sources, or check them against a reliable reference?

Do you measure the total cost of a task, including retries, tools and model calls?

Have you measured response times for typical and slow requests against an agreed target?

Have you tested the traffic you expect and set usage or spending limits?

Does your application enforce each user's and customer's access permissions whenever the system reads data or calls a tool?

Do actions such as changing records, sending information or spending money require human approval or follow explicit application rules?

Have you tested attempts to override the system's instructions through user input, documents and tool responses?

Have you reviewed which sensitive data reaches models, logs and providers, and how long it is kept?

Can you follow a failed request through document search, model calls and tool actions without exposing sensitive data?

Do you monitor quality, errors, response time and spending, with someone responsible for alerts?

Can your team restore a previous release and follow written instructions for incidents and handover?