How to Evaluate LLM Outputs in Production (Without Going Insane)

Shipping an LLM feature without evaluation is flying blind. Here's a practical system that catches regressions before your users do.

The Evaluation Pyramid

LLM evals have layers, similar to the test pyramid:

1. Unit evals -- fast, deterministic checks on small inputs
2. Integration evals -- end-to-end flows with real prompts
3. Human evals -- slow, expensive, ground truth
4. Production monitoring -- continuous sampling of live traffic

Most teams only do #3 and wonder why they keep shipping regressions.

Automated Metrics

Start with things you can measure without an LLM:

def evaluate_response(prompt: str, response: str, expected: dict) -> dict:
    scores = {}

    scores["too_short"] = len(response.split()) < 10
    scores["too_long"] = len(response.split()) > 2000

    for keyword in expected.get("must_contain", []):
        scores[f"contains_{keyword}"] = keyword.lower() in response.lower()

    if expected.get("format") == "json":
        try:
            import json
            json.loads(response)
            scores["valid_json"] = True
        except:
            scores["valid_json"] = False

    refusal_phrases = ["I cannot", "I'm not able to", "I don't have access"]
    scores["no_refusal"] = not any(p in response for p in refusal_phrases)

    return scores

LLM-as-Judge

For quality checks that need reasoning, use a second LLM:

import anthropic

judge = anthropic.Anthropic()

JUDGE_PROMPT = """
You are evaluating an AI assistant's response. Score it on each criterion from 1-5.

User question: {question}
AI response: {response}

Evaluate:
1. Accuracy -- is the information correct?
2. Completeness -- does it fully answer the question?
3. Clarity -- is it easy to understand?
4. Conciseness -- does it avoid unnecessary padding?

Return JSON: {"accuracy": N, "completeness": N, "clarity": N, "conciseness": N, "overall": N, "explanation": "..."}
"""

def judge_response(question: str, response: str) -> dict:
    result = judge.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=512,
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(question=question, response=response)
        }]
    )
    import json
    return json.loads(result.content[0].text)

Use a cheaper/faster model for judging. You'll run thousands of evals.

Building a Test Suite

from dataclasses import dataclass
from typing import Optional

@dataclass
class EvalCase:
    id: str
    input: str
    expected_contains: list[str]
    expected_format: Optional[str]
    tags: list[str]

def run_eval_suite(model_fn, suite: list[EvalCase]) -> dict:
    results = []
    for case in suite:
        response = model_fn(case.input)
        scores = evaluate_response(case.input, response, {
            "must_contain": case.expected_contains,
            "format": case.expected_format
        })
        judge_scores = judge_response(case.input, response)
        results.append({
            "id": case.id,
            "tags": case.tags,
            "automated": scores,
            "judge": judge_scores,
            "passed": all(scores.values()) and judge_scores["overall"] >= 3
        })

    pass_rate = sum(r["passed"] for r in results) / len(results)
    return {"pass_rate": pass_rate, "results": results}

Regression Testing on Prompt Changes

Every prompt change should run the full eval suite:

baseline_scores = load_baseline_scores()
new_scores = run_eval_suite(new_prompt_fn, eval_suite)

degraded = [
    r for r in new_scores["results"]
    if r["judge"]["overall"] < baseline_scores[r["id"]]["overall"] - 0.5
]

if degraded:
    print(f"REGRESSION: {len(degraded)} cases degraded")
    for r in degraded:
        print(f"  {r['id']}: {baseline_scores[r['id']]['overall']} -> {r['judge']['overall']}")
    exit(1)

Production Sampling

Log and evaluate a random sample of live traffic:

import random

def log_for_eval(prompt: str, response: str, user_id: str):
    if random.random() < 0.05:  # 5% sample rate
        eval_queue.push({
            "prompt": prompt,
            "response": response,
            "user_id": user_id,
            "timestamp": time.time()
        })

Review flagged samples daily. Anything scoring below 3 in the judge goes into your eval suite as a regression test.

What Good Looks Like

A mature eval system catches: prompt injection attempts, hallucinated facts, format drift after model upgrades, latency regressions (track p95 not average).

Track your pass rate over time. Below 90%? Stop shipping features and fix the evals.