Building an LLM Eval Framework That Catches Regressions

Every enterprise LLM system we have inherited from another team has the same problem: nobody knows whether the last prompt change made things better or worse. The team ships, somebody notices something is off three days later, somebody else rolls back, and the cycle repeats. There is no eval framework. There is no golden dataset. There is no production sampling. There is just vibes and a Slack channel.

This article is the architecture we put in place when we are brought in to fix this. It is opinionated, it is what works, and it costs less than most teams expect.

Why most LLM teams skip evals (and what it costs)

The objection we hear most often is that LLM outputs are "too subjective" to test. This is half true and entirely irrelevant. The fact that there is no single correct answer to "summarise this contract" does not mean there are no wrong answers. Hallucinated clauses are wrong. Truncated outputs are wrong. PII leakage is wrong. Refusing a legitimate request is wrong.

The cost of skipping evals compounds. By the time the system has been in production for six months, the team is too scared to touch the prompts because they cannot prove the change is safe. The model gets stuck on whatever was deployed in week 2, the underlying provider releases something twice as good and a third the price, and the team cannot adopt it because they cannot validate it.

That is the real cost: not the production incidents, but the frozen pipeline.

The two layers every production system needs

Treat evals as two distinct surfaces, with two distinct purposes.

Layer 1: Offline evals (run in CI before every deploy)

Offline evals run against a versioned dataset of inputs you control. They live in your repository alongside the application code. They run on every pull request that touches a prompt, a model version, a retrieval index, or a parsing rule. They are deterministic enough to be useful as a regression gate.

The output of an offline eval run is a small set of numbers — accuracy, faithfulness, latency p95, cost per request — compared against the last green run on main. If any number regresses beyond a defined threshold, the build fails. The contributor either fixes the regression or explains the trade-off in the PR description.

Layer 2: Online evals (run against sampled production traffic)

Online evals run on a sample of real user traffic, after the fact, in the background. They cannot block anything. Their job is to surface the failures that your offline dataset did not anticipate — the queries you did not predict, the edge cases your test set missed, the drift that happens when user behaviour shifts.

The output of online evals is a dashboard and a set of alerts. When the rolling 24-hour hallucination rate crosses 3%, somebody is paged. When the refusal rate doubles week over week, somebody investigates.

Online evals also feed the offline dataset. The interesting failures from production — particularly the ones humans flag during review — become the next round of test cases. Your dataset gets sharper over time because it gets fed by reality.

Building the golden dataset (the part that matters)

This is where 80% of the value lives, and where 80% of teams skip the work.

A golden dataset is a versioned set of inputs paired with either ground-truth outputs or evaluation criteria. For a RAG system over your support knowledge base, this might be 500 questions with the expected source documents and a description of what a good answer looks like. For a document classifier, it is 2,000 documents labelled with the correct class. For an agentic system, it is a set of tasks and a checklist of steps the agent must complete.

How to build one that does not lie to you

• Source from real traffic. The fastest way to a useless golden dataset is to have engineers write the examples. Engineers underestimate ambiguity and overestimate user literacy. Sample from your actual logs.
• Stratify by use case, not by frequency. If 60% of your traffic is one query template, you do not want 60% of your eval set to be that template. Cover the long tail.
• Include adversarial and edge cases deliberately. PII in inputs, jailbreak attempts, malformed JSON, queries in the wrong language, multi-turn confusion, very long contexts, very short contexts. If you have not tested these, you have not tested.
• Get human ground truth where it matters. Yes, this is expensive. Yes, you still need to do it. For the 100 most critical examples, have a domain expert review the expected output. Those become the anchors that calibrate everything else.
• Version the dataset. Treat it like code. Tag releases. Track which model version was evaluated against which dataset version. Without this, you cannot reason about whether a regression came from the model or the test set.

LLM-as-a-judge: when it works and when it does not

LLM-as-a-judge — using one model to score another model's output — is now the default scoring approach for anything that is not exact-match. It scales. It is cheap. It is broadly usable. It is also wrong about 30% of the time if you do not calibrate it.

The calibration step nobody skips twice

Take your judge prompt. Run it against 200 examples that you have also had a human reviewer score. Compute the inter-rater agreement between the judge and the human. If it is below 0.7 (Cohen's kappa), the judge is unreliable and you need to fix the prompt before trusting any aggregate scores.

Common reasons the judge prompt is wrong:

• The criteria are too vague. "Score 1–5 on quality" produces noise. "Score 1 if the answer contradicts the source document, 2 if it omits a key fact, 3 if it is correct but unclear, 4 if it is correct and clear, 5 if it is correct, clear, and directly cites the source" produces signal.
• The judge is too weak. Using GPT-4-class models to judge GPT-4-class outputs works. Using a small model to judge a frontier model usually does not.
• The judge is biased toward its own family. When choosing between two outputs, models tend to favour outputs from their own family. If you are comparing models from different providers, rotate judges and report a panel score, not a single judge score.

What to actually measure

The temptation is to pick every metric from every framework. Resist it. We track these in production for almost every LLM system we ship:

• Faithfulness / hallucination rate. Does the output stay grounded in the provided context? For RAG, this is the headline metric.
• Answer relevance. Did we actually address what the user asked, or did we answer an adjacent question?
• Refusal rate. How often is the system declining to answer? Sudden movement here usually means a prompt change or a model update.
• Latency p50 and p95. P95 is the one users actually feel. Track it per route, not aggregate.
• Cost per request. The fastest way to break a business case is to silently double your token usage on a prompt iteration.
• Toxicity / safety violations. Lower priority for internal enterprise systems, top priority for anything customer-facing.
• For agents: tool correctness and plan adherence. Did the agent call the right tools with the right arguments, and did it follow the plan it claimed it would follow?

The framework choice (less important than you think)

RAGAS, DeepEval, TruLens, Braintrust, LangSmith, Confident AI, MLflow LLM Evaluate, Arize — they are all viable. The differences matter at the margins. The differences in your dataset and judge prompts matter at the centre.

That said, our default stack for new enterprise builds is:

• DeepEval for offline. Pytest integration, runs in CI, broad metric coverage, open source. Easy to extend with custom metrics.
• Langfuse or Braintrust for online. Production tracing, sampling, judge runs against live traffic, dashboards.
• Your own evaluation harness if you have specific compliance constraints. Regulated industries (finance, healthcare, public sector) often cannot ship traces to third-party SaaS. A 200-line in-house harness is usually fine for offline; online observability is the harder build.

Where to start this week

If your team is shipping an LLM feature with no eval framework today, here is the sequence:

• Day 1–2. Sample 200 representative inputs from production logs (or, if you are pre-launch, from manual testing). Write down what a "good" output looks like for each. This is your seed dataset.
• Day 3–4. Pick a framework, write a judge prompt for your top 2 metrics, run it. Validate against 50 human-labelled examples. Iterate on the judge prompt until agreement is over 0.7.
• Day 5. Wire offline eval into CI as a non-blocking check first. Watch for two weeks. Once the signal is stable, promote it to blocking.
• Week 2. Add production tracing. Sample 5% of live traffic to start. Run the same judge metrics nightly. Build one dashboard with three panels: hallucination rate, refusal rate, latency p95.
• Week 3 onwards. Expand the dataset weekly with interesting production examples. Add metrics one at a time as you understand what they actually move.

That is a 3-week investment for a system that will let you iterate confidently for the next two years. We have not yet met a team that regretted the time.

The biggest unlock in production AI in 2026 is not a better model. It is the team that can change its prompts on a Friday afternoon and prove on Monday morning that nothing broke.

If you are building or rescuing an enterprise LLM system and the eval surface is missing or broken, this is the kind of work our AI consultancy does day in and day out. We have shipped eval frameworks for FTSE-250 banks, global telcos, and regulated healthcare platforms. We would be happy to talk.