You launched an AI agent. It's answering WhatsApp messages, handling support tickets, or booking calls. Most guides on LLM evals for business skip the real question: how do you know if this thing is working, without hiring an engineer to tell you?
You don't need a data scientist. You need three questions, a spreadsheet, and one hour a month.
Key Takeaways
- Evals are a business task, not engineering — Domain experts who know the right answer, not developers, score outputs as yes, no, or partial in a spreadsheet, no code required.
- 85 percent of GenAI projects fail on testing — Gartner traces this failure rate to bad data or insufficient testing, not weak models. A recurring eval habit is the fix, not a better LLM.
- Sample 1 to 5 percent of outputs monthly — For an SME with a few hundred conversations a week, that rate keeps the review to a bounded monthly commitment for one non-technical eval owner, not a new hire.
- 85 to 90 percent agreement unlocks automated judging — Run the judge on the same sample a human already scored. Once agreement clears that range, the judge can run at scale without a human re-checking every output.
- Most failures are a prompt gap, not model — A stale knowledge-base article, a missed prompt case, or a tool call returning the wrong field explains most failures, and each is a same-day fix, not a rebuild.
Why Untested Agents Are a Business Risk
Skip evals and you inherit two failures at once: silent inaccuracy and drift you cannot see. Gartner reported that 85 percent of GenAI projects fail because of bad data or insufficient testing.
The failure rate at the business level is worse. 95% of enterprise GenAI pilots produce no measurable P&L return, according to MIT Media Lab's Project NANDA. That gap between "shipped" and "actually working" is what evals close.
In the UAE the risk is more immediate than in most markets. WhatsApp is the primary customer channel here, so a wrong answer reaches the customer in seconds, with no support-desk buffer to catch it. Ongoing oversight of a live agent is what the discipline of AgentOps is built around, and evals are its central signal.
Verizon's annual Data Breach Investigations Report is the standard reference for how breaches actually begin.
Model Evals vs System Evals: What Business Teams Actually Need to Care About
Photo: RDNE Stock project on Pexels
Two different tests. Model evals check the underlying LLM in isolation, using academic benchmarks. System evals check whether your specific agent, with your prompts, knowledge base, tools, and conversation flow, does the job you built it for.
Business teams should ignore model evals almost entirely. The right question is not "is GPT-4 good?" It's "does our agent answer a Dubai customer's return query correctly, in the right language, using the right policy?"
Traditional software tests miss this. Standard QA can confirm an API call succeeded and the response came back non-empty. It cannot tell you the answer was factually wrong, politely rude, or missed a compliance rule. Industry writeups typically catalogue seven categories of agent failure that conventional testing does not catch, and every one of them is a business problem, not an engineering one.
That reframes who owns evals. Your domain experts, the people who actually know what a correct answer looks like, are the right reviewers. Not your dev team.
Three Questions That Replace a Dashboard Full of Metrics
Forget F1, BLEU, ROUGE, and perplexity. These are the three questions that matter for a business rubric:
- Is the answer factually correct for this use case? Did the agent get the price, the policy, the opening hours, the shipping window right?
- Is it on-brand and compliant? Outputs that touch personal data must respect Federal Decree-Law No. 45 of 2021, the UAE Personal Data Protection Law. A policy breach is a different severity from an off-tone reply, so score them separately.
- Did it complete the task the user actually needed? Task completion is the metric benchmarks and leaderboards systematically miss. A confident, articulate, technically correct answer that leaves the customer's problem unsolved is still a failure.
These three questions translate directly to the technical metrics catalogued by IBM, SuperAnnotate, and every other framework. You are not simplifying evaluation. You are doing the same evaluation, in language your team can act on.
PwC Middle East publishes comparable regional work, and reading both is a cheap way to spot where a single number is being over-quoted.
How to Sample Your Agent's Output Without Writing a Line of Code
You are not evaluating every conversation. Common industry practice is to sample 1 to 5 percent of random production outputs. For an SME with a few hundred conversations a week, that is a monthly workload one person can handle.
The process is boring on purpose:
- Export a random sample of last month's agent conversations to a spreadsheet.
- Add three columns, one per business question. Each cell is yes, no, or partial.
- Give the spreadsheet to a subject-matter expert on your team, whoever actually knows the correct answers.
- Tally the results. A running fail rate per question tells you where the agent is weakest.
One UAE-specific detail your reviewer must handle: bilingual reality. Customers write in Arabic, English, and code-switched mixes of both in the same message, sometimes Arabizi. If your reviewer only reads one language, your eval misses half the failure surface. Pick someone who works comfortably in both.
LLM-as-a-Judge: Automating the Checks Your Team Cannot Do at Scale
Photo: KATRIN BOLOVTSOVA on Pexels
Once your rubric is stable, you can hand scoring to another LLM. The judge is a second model, prompted with your rubric and a sample output. It scores the response the way your human reviewer would, at volume, without fatigue.
The trust rule is straightforward. Run the judge on the same sample your human reviewer just scored. If the two agree above 85 to 90 percent, the judge is reliable enough to run automatically. If they don't, fix the judge's prompt, not the results.
Be honest about the limit: the judge can share the same blindspots as the agent it's scoring. If both models struggle with UAE trade licence procedures, DIFC regulations, or a specific Arabic phrasing your customers use, the judge will approve wrong answers with confidence. Keep a human in the loop for the categories where the risk is highest.
This is where the shape of the operations work starts to diverge from a traditional engineering pipeline. The differences are the whole subject of AgentOps vs MLOps vs DevOps: who owns the eval, where it runs, and what triggers a change.
If you're not sure whether your current agent is set up to be evaluated at all, that is worth a conversation. Book a free 30-minute consultation with Lenoo AI and we'll help you map the rubric that fits your specific workflow.
The US National Institute of Standards and Technology organises its AI Risk Management Framework around four functions: govern, map, measure and manage.
What to Do When an Eval Surfaces a Problem
Most eval failures are not "the model is broken." They are a prompt that missed a case, a knowledge-base article that never got updated, or a tool call that returns the wrong field. All same-day fixes.
Triage by failure type before touching anything:
- Wrong facts: the knowledge base is stale or missing a document. Update the source, not the prompt.
- Wrong tone: the system prompt or persona instructions need tightening.
- Incomplete task: the agent gave up early or missed a step, usually a prompt or flow issue.
- Policy breach: a compliance or PDPL failure. Escalate immediately; a single one is one too many.
Set a threshold. If failures in a sample cross whatever error rate you have agreed with the team, escalate to the person who owns the agent instead of patching outputs one by one. Individual fixes are how technical debt accumulates fast.
This is the tight version of the loop covered in The Post-Launch Loop: Monitor, Evaluate, Update, Redeploy. Turning a one-off fix into a repeatable process is what stops the same failure from recurring next month.
Each failure type calls for a different fix, so triage before you touch anything.
| Failure Type | What It Usually Means | The Fix |
|---|---|---|
| Wrong facts | Knowledge base is stale or missing a document | Update the source, not the prompt |
| Wrong tone | System prompt or persona instructions are too loose | Tighten the persona instructions |
| Incomplete task | Agent gave up early or missed a step | Fix the prompt or conversation flow |
| Policy breach | Compliance or PDPL failure | Escalate immediately, don't patch quietly |
Making Evals a Monthly Habit, Not a Launch-Day Checkbox
Agent performance drifts as customer language, products, and business policies change, and as your knowledge base ages. A monthly sample at 1 to 5 percent of outputs catches drift before customers notice, which is the entire point.
Assign one non-technical team member as eval owner. At that sampling cadence, the role is a real but bounded monthly commitment, not a new hire. Their job is to run the sample, tally the scores, and flag anything that crosses the escalation threshold.
Evals are one signal among several in a well-run monthly cadence. The Monthly AI Health Check covers what else belongs on that review: cost per conversation, latency, edge-case volume, and change logs. Together they feed back into the ongoing AgentOps cycle. Evals tell you when to update the prompt, when to refresh the knowledge base, when to retrain, and when to retire an agent that stopped earning its keep. Without them, the cycle has no input.
Want an outside read on whether your agent is actually working, and which evals matter most for your workflow? Book a free 30-minute consultation with Lenoo AI. We'll give you an honest assessment, including if what you've built isn't working yet.
FAQ
Do I need a data scientist to run evals on my AI agent?
No. System evals, the type that matter for a live business agent, are a domain-expertise task, not a machine-learning one. Anyone on your team who knows what a correct answer looks like can score outputs against a yes/no rubric in a spreadsheet.
How often should a UAE business evaluate its AI agent after launch?
Monthly, at a sample of 1 to 5 percent of production outputs. That cadence catches drift caused by product, policy, and customer-language changes before customers start complaining, and it is small enough for one non-technical person to own alongside their normal work.
What is the difference between testing an AI agent and testing regular software?
Traditional software tests confirm a request completed and the code did not throw an error. Agent evals check whether the answer was factually correct, on-brand, compliant, and actually finished the customer's task. Same-shape success in the logs can still be a business failure.
Can I use an LLM to check whether my own agent's answers are correct?
Yes, once you have a stable rubric and enough human-scored examples to check the judge against. When human and LLM scores agree above 85 to 90 percent on the same sample, the automated judge is reliable enough to run at scale. Keep humans on the highest-risk categories.
What should I do if my agent starts giving wrong answers after a prompt or product update?
Categorise the failure first: wrong facts, wrong tone, incomplete task, or policy breach. Most fixes are same-day changes to the prompt or knowledge base rather than a model rebuild. If failures cross your agreed error rate, escalate to whoever owns the agent instead of patching individual outputs.
How does Federal Decree-Law No. 45 of 2021 affect how I store and review agent conversation logs in the UAE?
Agent conversations often contain personal data, which brings them under the PDPL. Limit reviewer access to the people who need it, keep processing purposes clear, and do not retain logs longer than your stated purpose requires. Involve whoever handles data protection on your side before you set up the eval workflow.
What counts as a passing result when I evaluate my agent against a business rubric rather than a technical benchmark?
You set the pass bar, not a leaderboard. A common starting point is 90 percent yes on factual correctness and task completion, and zero policy breaches in the sample. Track the trend more than the absolute number; a stable score beats a high one that is quietly declining.
