AI Systems Test Strategy

Oct 20, 2026 · @Santosh Karad

Purpose and scope

This strategy defines how QA verifies every AI-powered system we build or buy: no AI change reaches production without passing a versioned evaluation suite and a security review. It replaces exact-match test cases with statistical, rubric-based evaluation and extends testing into production.

System under test	What we own and test	Out of scope
AI models (foundation, fine-tuned, ML classifiers)	Fitness for our use cases, bias, robustness, version drift	Vendor pre-training and vendor-internal safety work
Chatbots and RAG assistants	Answers, retrieval, grounding, multi-turn behaviour, safety	Underlying model weights
AI agents (tool-using, multi-step)	Plans, tool calls, side effects, permissions, cost	Third-party APIs the agent calls (mocked)
AI voice agents	All chatbot checks plus speech, latency, turn-taking, telephony	Carrier network quality
AI-written code	Correctness, security, maintainability of code generated by assistants	The coding assistant product itself

Applies to all teams shipping AI features, internal tools, and vendor AI products before procurement sign-off.

Testing principles

AI output is non-deterministic and open-ended, so we test behaviour as a distribution and accept a measured pass rate, not a single correct answer.

Traditional testing	AI testing (this strategy)
Exact expected output	Rubric: what a good answer must and must not contain
Pass / fail per case	Pass rate per suite, with a threshold
One run per case	Multiple runs per case; variance tracked
Test cases in a tool	Versioned golden datasets in the repo
Testing ends at release	Production traffic is sampled and scored continuously
Tester judges output	Code checks, calibrated AI judges and humans, layered

Guiding rules:

Test the system, not the model. Prompts, retrieval, tools and guardrails change behaviour as much as the model does.
Every change is a regression risk. A prompt edit, model upgrade or new document in the knowledge base triggers the eval suite.
Measure the cost of being wrong. Thresholds are set per use case by business impact, not one global number.
Assume adversarial users. Security and misuse testing is part of every release, not a one-off audit.
Humans stay in the loop for high-impact decisions and for calibrating automated graders.

Evaluation framework

Every AI feature ships with a golden dataset and three layers of graders; this is the backbone all later sections reuse.

Golden datasets

100–500 labelled examples per use case to start, drawn from real user queries (anonymised), SME-written cases and known past failures.
Each record holds: input, context (documents, conversation history, tool state), expected behaviour as a rubric, tags (intent, language, risk level, persona).
Split into a regression set (stable, gates releases) and an exploration set (new and hard cases, tracked but not gating).
Stored in the repo, versioned with the prompt and config they test. No dataset change without review.
Coverage target: every intent, every supported language, edge and adversarial cases at ≥ 15% of the set.

Grader layers

Layer	Checks	Speed and cost	Use for
Code-based	Schema, JSON validity, regex, banned terms, length, tool-call arguments, exact facts	Fast, free, deterministic	Everything that can be checked mechanically
AI judge (LLM-as-judge)	Rubric scoring: correctness, groundedness, tone, completeness, policy adherence	Medium	Open-ended quality at scale
Human review	Ambiguous cases, judge disagreements, high-risk domains	Slow, costly	Calibration and final sign-off

AI judge rules

One criterion per judge prompt; binary or 1–5 scale with written anchors for each score.
Judge must cite the evidence for its verdict before scoring.
Calibrate against ≥ 100 human-labelled cases; require ≥ 85% agreement before the judge can gate a release. Re-calibrate when the judge model changes.
Never use the same model as both system under test and sole judge.

Run policy

Run each case 3–5 times; report mean pass rate and worst-case. Flag cases that pass sometimes as flaky for review.
Pin model version, temperature and seed where the provider allows, and record them with every run.

Approach: AI models

We select and accept models on our own domain data; public leaderboards inform the shortlist, never the decision.

Test area	What we check	How
Task performance	Accuracy, F1, or rubric pass rate on our golden set	Offline eval, compared across candidate models
Robustness	Stable output under typos, paraphrase, casing, code-mixed language (e.g. Hinglish), long inputs	Perturbation suites generated from the golden set
Calibration	Confidence matches correctness; model says "I don't know" when it should	Unanswerable-question set; abstention rate
Bias and fairness	Equal quality across gender, region, language, age-related cues	Slice metrics; counterfactual pairs that swap only one attribute
Version drift	No regression when the provider updates or we fine-tune	Full regression run on every model version; side-by-side diff
Data and training (fine-tuned / ML)	No train–test leakage, label quality, data lineage	Dedup checks, label audits, holdout sets never seen in training
Non-functional	Latency p50/p95, throughput, cost per 1,000 requests, context limits	Load tests against the serving endpoint

Model acceptance criterion: a new or upgraded model must match or beat the current model on the regression set, with no slice dropping more than 3 points.

Approach: chatbots and RAG systems

We test retrieval and generation separately, because a wrong answer from a wrong document needs a different fix than a wrong answer from the right one.

Retrieval layer

Context recall: did the top results include the document that holds the answer?
Context precision: how much of what was retrieved is relevant?
Freshness: superseded documents are not retrieved after a knowledge-base update.
Access control: users never receive chunks from documents they are not entitled to see.

Generation layer

Groundedness (faithfulness): every claim is supported by retrieved context; unsupported claims count as hallucinations.
Answer correctness and completeness against the rubric.
Citation accuracy: cited source actually contains the cited claim.
Correct abstention when the knowledge base has no answer.

Conversation behaviour

Multi-turn: remembers earlier facts, handles corrections ("no, I meant the other account"), resolves pronouns.
Refusals: no over-refusal of legitimate requests, no under-refusal of out-of-policy ones.
Tone and persona consistency; brand and regulatory wording.
Handoff to a human when confidence is low or the user asks.
Languages and scripts we support, including code-mixed input.
Privacy: no PII echoed from other sessions, no system-prompt disclosure.

Test design: script multi-turn conversations with simulated users (an AI playing defined personas and goals) and score the whole transcript, not only the last reply.

Approach: AI agents

For agents we grade the trajectory (every step it took) as well as the outcome, because an agent can reach the right answer by an unsafe or expensive path.

Level	What we check	Example assertion
Outcome	Task completed; end state correct in the target system	Ticket created with correct priority and assignee
Tool selection	Right tool chosen, wrong tools not called	Uses `refund_lookup`, never `refund_issue` for a status query
Tool arguments	Arguments valid and grounded in user input	Amount equals the amount the user confirmed
Ordering and planning	Prerequisite steps happen first	Verifies identity before changing account data
Efficiency	Step count, tokens, cost, wall-clock time within budget	≤ 8 steps and ≤ ₹5 per task (set per use case)
Termination	Stops when done or stuck; no infinite loops	Exits with a clear message after 3 failed retries
Permissions	Asks for confirmation before irreversible or high-value actions	Payment, deletion and external email always need approval

Environment: agents run in a sandbox with mocked or staging tools. Mocks inject failures — timeouts, 500 errors, empty results, malformed responses, rate limits — to test recovery.

Key scenarios

Ambiguous instructions: does it ask rather than guess on high-impact actions?
Conflicting instructions between user, system prompt and tool output.
Indirect prompt injection planted in tool results, web pages, emails or files the agent reads.
Multi-agent handoffs: context and state passed correctly; no duplicated actions.
Idempotency: re-running after a crash does not double-charge or double-send.

Every agent run in test and production emits a full trace (prompts, tool calls, results, decisions) so failures can be replayed.

Approach: AI voice agents

Voice agents get every chatbot and agent check plus a speech pipeline layer, tested end to end with simulated callers over real telephony.

Pipeline stage	What we check	Metric
Speech-to-text (ASR)	Accuracy across accents, regional languages, code-mixing, background noise, poor lines	Word error rate per slice; entity accuracy for names, numbers, dates, IDs
Understanding and reasoning	Intent, slot filling, policy, tool use (same as chatbot and agent sections)	Task success rate
Text-to-speech (TTS)	Pronunciation of names, amounts, dates, abbreviations; natural prosody	Human MOS-style rating on a sample
Turn-taking	Barge-in (caller interrupts), long pauses, silence, crosstalk, "hello? are you there?"	Interrupt handling pass rate
Latency	Time from caller finishing speech to first agent audio	p50 and p95 against target set by product
Telephony	DTMF input, call transfer to human, hold, voicemail detection, dropped calls	Scenario pass rate

Test method

Synthetic callers: AI personas with defined goals, accents, speaking speed, emotional state (angry, confused, elderly) and background noise (traffic, office, TV).
Replay of anonymised real calls with consent, scored on transcripts and audio.
Load test concurrent calls; latency must hold at expected peak.
Compliance: call recording disclosure, consent capture, identity verification steps, no reading back full card or account numbers.

Always score at two levels: per turn (did this reply make sense?) and per call (did the caller achieve their goal?).

Approach: AI-written code

AI-generated code goes through the same pipeline as human code, plus extra gates for the failure modes AI makes more often: invented dependencies, plausible-but-wrong logic, and tests that pass without testing anything.

Mandatory gates

Human review by an engineer who understands the change; the author stays accountable regardless of who or what wrote it.
Static analysis and security scanning (SAST, secrets detection) on every PR.
Dependency check: every new package must exist in the official registry, have real maintainers and pass SCA. AI can invent package names that attackers then register.
Licence check for code that may resemble copied open-source snippets.
Test-quality check: mutation testing on AI-written tests; a test suite that survives most mutations is not testing anything.
Separation: AI must not write both the code and its only tests. At least one side is human-written or human-reviewed against the requirement.

What testers focus on

Edge cases and error paths, which AI often skips: nulls, empty lists, time zones, currency rounding, concurrency.
Behaviour against the original requirement, not against what the code appears to do.
Silent changes: AI refactors that alter behaviour outside the requested scope.
Performance regressions from naive generated queries or loops.

Tracking: tag AI-assisted PRs and compare their defect escape rate, review rework and incident rate with other PRs each quarter.

Security and red teaming

Every AI release runs an automated attack suite, and every high-risk release also gets manual red teaming; we map findings to the OWASP Top 10 for LLM Applications.

Threat	Applies to	Test
Direct prompt injection and jailbreaks	All	Attack library (role-play, encoding, multi-turn escalation, other languages)
Indirect prompt injection	RAG, agents, voice	Malicious instructions planted in documents, web pages, emails, tool output
System prompt and config leakage	All	Extraction attempts; check no secrets are in prompts at all
Sensitive data disclosure	All	Attempts to extract other users' data, PII, internal documents
Excessive agency	Agents	Coax agent into unapproved actions, privilege escalation, data exfiltration via tools
Insecure output handling	All	Output rendered as HTML/SQL/shell is sanitised; XSS and injection payloads in replies
Denial of wallet	All	Inputs designed to cause long loops, huge outputs or runaway token cost
Voice spoofing	Voice	Cloned or synthetic voices against voice-based verification
Supply chain	Models, code	Model and package provenance; untrusted weights, plugins and MCP servers

Cadence: automated suite in CI on every change; manual red team before first launch and quarterly for high-risk systems; findings become permanent regression cases.

Responsible AI, bias and compliance

Each AI system gets a risk tier at design time, and the tier decides how much fairness, explainability and compliance evidence QA must produce before launch.

Risk tier	Examples	Required evidence
High	Credit, hiring, health, legal, identity verification	Full slice-level fairness report, human review of decisions, explainability check, legal sign-off, quarterly re-test
Medium	Customer support, sales assistants, internal agents with write access	Fairness spot checks, harmful-content suite, escalation path tested
Low	Internal summarisation, drafting aids with human edit	Harmful-content suite, basic accuracy eval

Checks

Fairness: quality metrics compared across demographic and language slices; counterfactual tests swap one attribute (name, gender, region) and expect equivalent outcomes.
Harmful content: toxicity, self-harm handling, dangerous advice, misinformation in our domain.
Transparency: users are told they are talking to AI; AI-generated content is labelled where required.
Privacy: data minimisation, consent, retention and deletion tested against applicable law (e.g. India's DPDP Act, GDPR, and the EU AI Act where we serve EU users). Legal confirms the current obligations; QA produces the evidence.
Audit trail: every eval run, dataset version and sign-off is stored for audit.

Environments, test data and tooling

We run evals in three environments and standardise on one eval framework and one tracing platform across teams, chosen after a two-week proof of concept.

Environments

Dev / CI: fast regression subset (≈ 50–100 cases) on every PR; mocked tools.
Staging: full regression and security suites; staging tools and a staging copy of the knowledge base; voice tests over real telephony.
Production: shadow mode for new versions, A/B or canary release, and continuous scoring of sampled traffic.

Test data

Production samples are anonymised before entering any dataset; no raw PII in test repos.
Synthetic data (AI-generated cases and personas) is allowed but reviewed by an SME before it gates releases.
Knowledge-base snapshots are versioned with the dataset so results are reproducible.

Tooling shortlist (open-source examples to evaluate, not mandates)

Need	Candidate tools
Eval framework and CI gating	promptfoo, DeepEval, OpenAI Evals, Inspect
RAG metrics	Ragas, DeepEval
Tracing and observability	Langfuse, Arize Phoenix, LangSmith (commercial)
Red teaming	garak, PyRIT, promptfoo red-team
Voice agent simulation	Vendor voice-testing platforms; in-house caller simulator with TTS personas
AI-written code	Existing SAST/SCA (e.g. Semgrep, CodeQL, Snyk), mutation testing (e.g. Stryker, PIT, mutmut)

Selection criteria: CI integration, custom metrics, dataset versioning, self-hosting for data privacy, and cost at our volume.

Entry/exit criteria and quality gates

A release passes only when all exit criteria for its risk tier are met; thresholds below are defaults that product and QA tune per use case and record in the test plan.

Entry criteria (testing starts when)

Use case, risk tier and success definition agreed with product
Golden dataset reviewed by an SME and versioned
Rubrics written; AI judges calibrated to ≥ 85% human agreement
Model version, prompts and config pinned and recorded
Tracing enabled in the test environment

Exit criteria (release allowed when)

Gate	High risk	Medium risk	Low risk
Task success on regression set	≥ 95%	≥ 90%	≥ 85%
Hallucination / ungrounded rate	≤ 1%	≤ 3%	≤ 5%
Regression vs current version	No drop > 1 pt	No drop > 2 pts	No drop > 3 pts
Critical security findings	0 open	0 open	0 open
Safety policy violations	0	≤ 0.5%	≤ 1%
Latency p95 and cost per task	Within agreed budget	Within agreed budget	Within agreed budget
Human review sample	100 cases signed off	50 cases	20 cases

Suspension criteria: stop testing and return to development if task success drops more than 10 points from baseline, a critical security issue is found, or the eval environment cannot reproduce results.

Metrics and reporting

We report six headline metrics per AI system on one dashboard, broken down by use case, slice and version.

Metric	Definition	Audience
Task success rate	% of cases or sessions where the user goal is achieved	Leadership, product
Hallucination rate	% of answers with unsupported or false claims	Product, QA, compliance
Safety violation rate	% of responses breaching content or action policy	Compliance, security
Escalation rate	% of sessions handed to a human, and why	Operations, product
Latency p95	95th percentile response time (voice: time to first audio)	Engineering
Cost per task	Model, tool and infra cost per completed task	Finance, engineering

Supporting QA metrics: judge–human agreement, dataset coverage by intent and language, flaky-case count, security findings open and closed, defect escape rate for AI-assisted code.

Reports: an eval report on every release (pass rates vs thresholds, diff against last version, top failure clusters, sign-offs); a weekly production quality summary; a quarterly AI quality review with leadership.

Production monitoring and feedback loop

Production is the final test environment: we score sampled live traffic every day, and every confirmed failure becomes a permanent regression case.

Canary releasesmall share of traffic first
Sample live traffictraces and user feedback
Score automaticallycode checks and AI judges
Triage failurescluster, then human review
Add to golden seteach failure becomes a test
Fix and re-run evalsmust pass gates to ship

↺ Passing the gates starts the next canary release.

Every production failure becomes a permanent regression test.

Coverage: trace 100% of sessions; score all high-risk sessions and a daily sample of the rest.
Signals: low judge scores, thumbs-down, escalations, repeated questions, abandoned calls, tool errors, cost spikes.
Alerts: page on-call QA and engineering when a headline metric crosses its exit threshold over a rolling 24 hours.
Drift: compare live topics, languages and intents with the golden set weekly; new intents trigger dataset updates.
Rollback: model, prompt and knowledge-base versions roll back together in one step.

Roles and responsibilities (RACI)

QA owns the evaluation datasets and quality gates; product owns the thresholds; engineering owns fixes. R = responsible, A = accountable, C = consulted, I = informed.

Activity	QA lead	QA / eval engineers	Product owner	AI/ML engineers	Domain SMEs	Security	Compliance / legal
Risk tiering	C	I	A	C	C	C	R
Golden datasets	A	R	C	C	R	I	I
Rubrics and judge calibration	A	R	C	C	R	I	I
Thresholds and exit criteria	R	C	A	C	C	C	C
Eval automation in CI	A	R	I	C	I	I	I
Red teaming	C	R	I	C	I	A	I
Fairness and compliance evidence	R	R	C	C	C	I	A
Release sign-off	R	C	A	C	I	C	C
Production monitoring	A	R	I	R	C	C	I

New skills for the QA team: rubric and eval design, prompt engineering, basic Python for eval pipelines, reading traces, statistics for pass rates and confidence intervals, and AI security testing.

Risks and mitigations

The biggest risk to this strategy is false confidence: green eval dashboards built on datasets or judges that do not reflect real users.

Risk	Impact	Mitigation
Golden set does not reflect real traffic	Passes in test, fails in production	Refresh monthly from production samples; track coverage by intent
AI judge drifts or is biased	Wrong release decisions	Quarterly re-calibration; human spot checks of 5% of judged cases
Provider silently updates model	Unexplained regressions	Pin versions; nightly canary eval on pinned and latest
Non-determinism hides real failures	Flaky gates, ignored results	Multiple runs per case; flaky cases reviewed, not skipped
Eval cost grows with suite size	Teams skip evals	Tiered suites: small in PR, full nightly and pre-release; cache results
SME time unavailable for labelling	Slow or low-quality datasets	Agree SME capacity at project start; use AI pre-labelling with SME review
New attack techniques	Exploits in production	Update attack library monthly; bug bounty or external red team for high risk
Over-reliance on AI-written tests	Untested code paths	Mutation testing; human-owned acceptance tests

Rollout roadmap

We start with one pilot chatbot and one agent, prove the gates block a bad release, then extend to every AI system within about six months.

Foundation · weeks 1–4Risk-tier AI systemsFirst golden setEval tool PoC
Suite runs in CI
Pilot · weeks 5–101 chatbot, 1 agentJudges calibratedFirst red team
Gates block a release
Scale · months 3–6All AI systems gatedVoice + code gatesLive monitoring
All AI releases gated
Sustain · month 6 onwardQuarterly red teamAudit-ready evidenceCost tuning

Every AI release is gated within about six months.

Each phase moves on only when its gate is met. Durations are estimates to confirm with team capacity and SME availability.

Open questions

Which product is the pilot?
Which eval framework and tracing platform win the PoC?
How many SME hours per week can each domain commit to labelling?
Budget for eval compute and external red teaming?