LLM evaluation is the process of testing how a language model or an application using one performs against defined expectations. It helps a team decide whether a change is useful, which failures remain and where human judgment is still needed.
For a company building an AI support assistant, the question is concrete: can the next version answer customer questions accurately from the material that customer is allowed to access? A fluent demonstration does not answer that question. The team needs representative cases, explicit criteria and a record of what happened.
Good evaluation turns a broad impression into a decision that other people can examine. It can reveal that a new version writes more clearly while missing a critical qualification, or that a faster retrieval configuration overlooks a necessary source. The resulting decision might be to ship, revise the system, limit its scope or gather better evidence.
What are you actually evaluating?
The phrase LLM evaluation can refer to several different objects. Naming the object prevents a result from being stretched beyond its scope.
Model evaluation asks how a model performs on selected tasks under defined conditions. A public benchmark can be useful background, but it may have little overlap with a company's documents, permissions or action rules.
Application evaluation includes the instructions, retrieval, source preparation and output handling around the model. It asks whether this assembled system meets the requirements of its intended use.
Agent evaluation also examines the sequence of actions. It matters which tools were called, what information they returned, whether the agent stopped appropriately and whether any claimed external result was confirmed.
A model can produce a good answer when given the correct evidence while an application fails to retrieve that evidence. An agent can choose the right action but apply it to the wrong account. These failures call for different changes.
Start an evaluation record with one sentence describing the object and decision. For example: “Compare the current and proposed support assistant on answering questions from permitted product documentation, with external changes disabled.” That scope is specific enough to test.
Begin with the decision, then choose the measurements
A useful evaluation starts with what the team needs to decide. “Which version should answer these questions?” is more actionable than “What is our AI score?”
For the support assistant, the team might require that the answer preserves the documented conditions of a feature, identifies uncertainty and stays within the requester's access. Writing quality matters, but it cannot compensate for disclosing another customer's information.
Treat these requirements as separate dimensions. A single average can hide the failure that matters most. A release review should expose serious failures even when many routine questions were answered well.
Evaluation services make similar distinctions between measurement methods. Google's evaluation overview describes deterministic metrics and evaluations based on explicit rubrics. A team still needs to decide which criteria are appropriate for its application.
Example: reviewing a support assistant update
Imagine a software company that is revising the instructions and retrieval settings for its support assistant. The assistant drafts answers for people to review; it cannot change customer accounts in this test. This is an illustrative evaluation plan, not a customer case study or a measured comparison.
The proposed version appears more concise in a demonstration. Before releasing it, the team wants to know whether that concision removes information customers need.
Build a representative review set
Select questions that reflect the intended workload. Include ordinary feature questions and cases that require care: an answer split across two documents, a feature with a plan restriction, a superseded policy, an ambiguous request and a question the available material cannot answer.
Each case should include the request, the permitted source material or source snapshot, and the expected behavior. When several answers could be correct, describe the required substance and boundaries. Matching one preferred sentence is usually too narrow.
For a question about enabling an integration, the expectation might be: explain the documented prerequisites, retain the plan restriction and ask for the missing environment information before proposing a specific configuration. An answer that invents a setting should fail even if the rest reads well.
Cases should come from material the team can legitimately use. Remove unnecessary personal information and preserve the access conditions that make the case meaningful.
Establish the comparison conditions
Record the model configuration, instruction version, retrieval settings and source revisions for each candidate. If the application calls tools, record the relevant tool contract and test environment too.
Run both candidates against the same defined cases. Repeated trials can reveal variability that a single response misses. Keep a comparison record that connects every result to the configuration that produced it.
This does not require freezing production forever. It requires knowing what was held stable for the comparison and what changed. If both the documents and instructions changed, the result cannot cleanly establish which change caused the difference.
Examine the failures before the aggregate
Suppose a shorter response omits a plan restriction. The important finding is the missing condition and its consequence. The reviewer should be able to inspect the answer and the exact source passage, then decide whether the proposed version is acceptable for the intended use.
The release decision can be narrower than approving the whole system. The team might revise the instruction, keep the earlier retrieval configuration or limit the assistant to drafting answers that a person checks. The evidence should support that decision without implying a broader safety guarantee.
Use the right kind of check for each requirement
Different questions need different tests. Keeping them separate makes the results more interpretable.
Deterministic checks
Use ordinary software checks for requirements that can be established mechanically: required fields, valid identifiers, allowed tool names, output length limits or whether a returned reference belongs to the permitted test collection.
These checks are valuable because their meaning is explicit. A valid source identifier confirms that a reference exists in the expected set. It does not prove that the source supports the answer's claim.
Evidence and task review
Some requirements need an assessment of meaning. Does the answer preserve the documented limitation? Does it distinguish a confirmed fact from a recommendation? Does it ask a useful clarifying question when the request is incomplete?
A short rubric can make this review more consistent. For the integration question, a reviewer might record the prerequisite that was included or omitted, the source used and the consequence of the omission. That explanation is more useful for fixing the system than an unexplained score.
Model assisted judgment
A model can help assess answers against a rubric, particularly when there are many cases. Its verdict needs validation. The original LLM judge research by Zheng and colleagues identifies limitations including position, verbosity and self enhancement biases. Those findings are a reason to examine the judge, not a universal measurement of every current evaluator.
Compare a sample of automated judgments with knowledgeable human review. Include disagreements and serious failure cases. If two answers are compared, check whether reversing their presentation changes the preference. Keep the rubric and judge configuration in the record.
The output of a judge is another piece of evidence. A persuasive explanation from that judge is not independent proof that its verdict is correct.
What changes when the system is an agent?
An agent can fail between the request and its final answer. Evaluating only the final paragraph can miss the event that caused the problem.
For an assistant allowed to investigate a support issue, inspect whether it selected the correct customer scope, requested the appropriate records and handled an unavailable tool. If it can propose or perform changes, the evaluation should also check the authority boundary and the confirmation of any effect.
A practical review separates four questions:
Did the agent choose a justified next step from the information available?
Was the action permitted for this user and target?
Did the tool or external service actually complete the intended operation?
Did the final report accurately describe the result, including uncertainty?
An agent that receives a timeout should not turn that uncertainty into “completed.” A retry may also be inappropriate when the first attempt could already have changed the target. Test these conditions in a controlled environment before granting consequential access.
The surrounding execution software matters here. Our agent harness guide explains how model decisions become executable work and where application controls belong.
Keep development cases and release evidence distinct
Teams naturally improve a system using the examples in front of them. If the same small set becomes both the development target and the entire release argument, the evidence becomes less informative.
Maintain a regression set for known failures and keep additional cases for checking broader behavior. Track which cases informed a prompt or retrieval change. When a failure becomes a regression test, preserve the reason it matters.
A newly added case does not need an invented definitive answer. An expert may need to document acceptable alternatives, unresolved ambiguity or the conditions under which the system should decline to conclude.
Production feedback can supply new cases, subject to appropriate privacy and access handling. MLflow's evaluation and monitoring documentation describes datasets, expectations, human feedback and evaluators as related parts of this process. The article's support scenario is independent of any particular evaluation tool.
Record cost and time alongside quality
A system can become more accurate on a selected task while creating too much delay or review work for its intended setting. Evaluate those operational consequences alongside the answer.
For the support assistant, useful records could include response time, model and tool usage, the amount of human correction needed and the number of unresolved cases. Define what each measurement includes. A model bill alone does not capture the cost of a person repairing a misleading answer.
Report the number and type of cases behind a result. A result from a narrow internal review should be described at that scope. Avoid turning it into a claim about every customer request, language or operating condition.
An evaluation can also identify a useful boundary: a workflow may be ready for assisted drafting while remaining unsuitable for autonomous account changes. That is a product decision supported by evidence, not a failure to achieve an all purpose score.
How do evaluation and observability fit together?
Evaluation judges behavior against expectations. Observability provides records that help explain what happened. The two support different questions.
A failed case may show that the answer omitted a restriction. The trace may reveal that retrieval supplied an old document or that a tool returned only part of the requested information. That evidence helps the team decide what to change.
Keep the references between the case, the attempted work, its outputs and the judgment. A later reviewer should be able to reconstruct the basis of the decision without relying on someone's recollection. Our AI observability guide covers the records needed to investigate agent work.
Where does Swarm fit in an evaluation process?
A Swarm Space can hold the objective for an evaluation, discussion of the criteria, outputs from attempted work and the human decisions about the result. Tasks describe the saved work; Runs record attempts. Where an evaluation is requested and supported, its saved judgment can remain connected to the relevant evidence.
That record is useful when several people or agents investigate a failure over time. One participant can identify a retrieval problem, another can propose a change and a person can review the result with access to the same permitted context.
Saving a judgment is separate from authorizing an action. An evaluation result is not human approval, and a successful Run does not independently establish that a customer system changed correctly. External effects need their own confirmation.
Swarm does not remove the need to choose cases, run the appropriate tests or validate automated judges. This article also makes no claim that Swarm includes a native MLflow or Google evaluation integration. The useful connection is a persistent, reviewable history of the work and the decisions it supports.
Common questions about LLM evaluation
Is a public benchmark enough to choose a model?
It can inform a shortlist. A release decision also needs evidence from the application's tasks, sources, permissions and operating conditions. Test the assembled system that people will use.
Can another LLM do all the evaluation?
A model can assist with defined judgments, but its behavior needs checking against expert review and meaningful failure cases. Mechanical validation, human assessment and model judgments have different strengths and limits.
Is a good answer enough for an agent to pass?
No. The path matters when an agent accesses information or takes actions. Inspect permissions, tool behavior, external confirmations and whether the final answer faithfully reports what happened.
What should a team do first?
Choose one release decision. Collect representative cases, define acceptable behavior and identify the failures that would block that release. Preserve the results and their configurations so the next change can be evaluated against an understandable baseline.