Offline Evals as a CI Gate

Nearly half of production AI teams are shipping without any offline evaluations. The exact number, from a recent industry survey, is 47.6%. No golden dataset. No regression check. No CI gate. Just vibes and hope that the next prompt change did not break anything.

I find this stunning, but also completely understandable. When you are moving fast, evals feel like infrastructure work you will "add later." The problem is later never comes, and by then you've shipped three regressions you did not notice until users complained.

Table of Contents#

• Why Most Teams Treat Evals as a Ceremony
• Start Small: 20 Examples, Binary Pass/Fail
• What Goes in the Dataset
• Three Sources for Dataset Content
• The Evaluation Tier
• Dataset Maintenance
• GitHub Actions Setup with Promptfoo
• What to Do This Week
• How I Would Run This in Production
• What I Would Measure
• Where This Connects
• FAQ

This post is about making evals part of CI the same way tests are. Not a one-time audit before launch. Not a manual "let's run evals before this release." A gate that fails the PR if quality drops.

Why Most Teams Treat Evals as a Ceremony#

The data on this is damning. 93.28% of published agent evaluations happen only pre-deployment, as a one-time check. 70.90% of teams treat evaluation as a checkpoint, not a continuous process.

That pattern makes sense historically. Evals came from the ML research world where you train a model, evaluate it once, and ship it. But LLM applications are not trained artifacts. They change with every prompt edit, every library update, every context window tweak. A system that was "good" last week might behave completely differently after you added three sentences to the system prompt.

If evals do not run on every change, they do not exist. Running them manually means they'll be skipped when you are under pressure, which is exactly when you need them most.

Start Small: 20 Examples, Binary Pass/Fail#

The main reason teams do not have evals is they think it requires a massive upfront investment. It does not.

Start with 20 to 50 manually labeled production traces. Pull real inputs from your logs. Label each one with the correct output or correct behavior. Write a binary pass/fail check. That is it. Wire it into CI. You now have a regression suite.

The statistical floor for meaningful signal is 246 samples per scenario, assuming you are targeting an 80% pass rate with a 5% margin of error at 95% confidence. But that is not where you start. You start at 20 examples to build the habit, learn where your system breaks, and get something running in the pipeline. Expand from there.

The working minimum for a production system is around 100 high-quality golden examples. For most enterprise use cases, 500 to 1,000 is sufficient.

What Goes in the Dataset#

Dataset composition matters. A pure success case collection will hide failures in edge conditions. A reasonable breakdown:

• 50 to 60% success and typical cases: the bread and butter queries your system handles well
• 25 to 35% complex multi-step scenarios: the chains of reasoning that are easy to break
• 10 to 20% edge cases and adversarial inputs: prompt injection, PII leakage, jailbreaks, fraud attempts

The adversarial bucket is the one most teams skip. It is also the one that bites them.

Two requirements that teams often miss. First, every golden example needs a rationale alongside the correct answer. Not just "this response is correct" but "this response is correct because X, and the failure mode we are guarding against is Y." Rationale-less sets cannot calibrate LLM-as-judge scoring, and they cannot support audits. Second, run near-duplicate pruning via embedding similarity clustering before finalizing the dataset. Duplicates inflate pass rates and give you false confidence.

Also: decontaminate your dataset against your model's training corpora if you know what they contain. Training data leakage silently inflates benchmark scores. You'll think your system is performing well when you are just measuring memorization.

Three Sources for Dataset Content#

A robust golden dataset draws from three places, in combination.

Production logs are the most valuable source because they represent real failures, real edge cases, and actual user behavior. These are the examples your system will actually face.

Human-curated edge cases catch the things that do not show up in logs yet but you know are coming. PII handling, refusal behaviors, format edge cases.

Synthetic data fills coverage gaps. If you have a scenario you want to test but do not have enough real examples, generate them. The catch: synthetic items get promoted to the gold tier only after domain expert review. Synthetic data that looks plausible but is subtly wrong is worse than having no data.

One sequencing note on the evaluator itself. Write the evaluator after labeling examples, not before. The correct sequence is: name the failure mode you care about, label 20 or more examples that demonstrate it, then write the evaluator to match your labeled intuitions. Writing the evaluator first leads to measuring what is easy to measure, not what matters.

The Evaluation Tier#

Not all checks cost the same. Structure them in order of increasing cost and decreasing certainty.

Deterministic checks run first: regex matching, JSON schema validation, format compliance. These are fast, free, and unambiguous. If the output does not match the required JSON schema, fail immediately without involving a model.

Heuristic scoring runs second: semantic similarity, ROUGE scores, structural overlap. Slower but still cheap.

LLM-as-judge runs third: the model evaluates a subset of outputs for quality, coherence, appropriateness. More expensive and requires calibration against your rationales.

Human review runs last, for calibration only. Not for every PR, but on a sample weekly to make sure your LLM-as-judge hasn't drifted.

The thresholds that matter in CI: minimum 90% success rate on the full golden dataset, accuracy above 0.85 on labeled examples. You can tune these per scenario. The point is that the gate has teeth. A PR that drops you from 94% to 86% should not merge.

Dataset Maintenance#

A golden dataset that does not get updated becomes a museum of old bugs. The maintenance cadence that actually works:

Pull fresh production samples weekly or bi-weekly. Do a comprehensive audit monthly: check for label drift, prune stale examples, add new failure modes you've observed. Do a major structural refresh quarterly: reassess coverage, revisit the composition balance, consider whether your scenario taxonomy still matches the system.

Version-control your dataset paired with your prompt version. If you change the prompt, the dataset version that corresponds to that prompt should be traceable. When you debug a regression, you need to know which dataset was passing at the time the regression was introduced.

GitHub Actions Setup with Promptfoo#

My current toolchain recommendation depends on what you need. Braintrust has native GitHub Actions integration and posts metric deltas as PR comments, which makes regression analysis visible in the review flow. Promptfoo is OSS, supports GitHub Actions and GitLab CI, and has red-teaming built in. Langfuse is the self-hosted option for teams that cannot send data to third parties.

Here is a minimal Promptfoo CI gate using GitHub Actions:

name: Eval Gate

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'src/**'

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Promptfoo
        run: npm install -g promptfoo

      - name: Run evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          promptfoo eval \
            --config evals/promptfooconfig.yaml \
            --output evals/results.json \
            --max-concurrency 4

      - name: Check pass rate
        run: |
          PASS_RATE=$(jq '.results.stats.passRate' evals/results.json)
          echo "Pass rate: $PASS_RATE"
          if (( $(echo "$PASS_RATE < 0.90" | bc -l) )); then
            echo "Eval pass rate $PASS_RATE below threshold 0.90. Blocking merge."
            exit 1
          fi

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: evals/results.json

The promptfooconfig.yaml references your golden dataset, your prompt templates, and your assertion definitions. The check step reads the output JSON and fails the job if the pass rate drops below threshold. This runs on every PR that touches prompts or source code.

If your eval suite takes more than a few minutes, run the deterministic and heuristic tiers on every PR and gate the LLM-as-judge tier to main branch merges only. Fast feedback on deterministic failures is more valuable than waiting for a full evaluation before you even know the format is right.

What to Do This Week#

1. Pull 20 production traces from your logs. Label them manually with correct outputs and rationale. Save them in a versioned file next to your prompt.

2. Write one deterministic check per output requirement (JSON schema, required fields, character limits). These take 30 minutes and catch the obvious regressions immediately.

3. Add a Promptfoo or Braintrust config file to your repo with those 20 examples wired up. Run it locally. Make sure it passes on current state.

4. Add the GitHub Actions job above, or an equivalent. Set the threshold to 90%. Watch it run on the next PR.

5. Schedule a calendar block once a month to review and expand the dataset. Treat it like a database migration: mechanical, necessary, not optional.

The teams that have offline evals in CI are not doing something exotic. They built a small dataset, wired it into their pipeline, and did not turn it off. That is the whole thing. The teams without evals are running experiments in production with their users as the test suite. That is the worse option.

How I Would Run This in Production#

I would begin with a painfully small dataset. Twenty examples is enough to expose whether the team understands the task. If you cannot agree on the expected answer for twenty representative cases, a two-hundred-example suite will only create more noise.

The first version should mix boring cases, boundary cases, and recent failures. Boring cases prevent regressions in the main path. Boundary cases reveal whether the prompt handles ambiguity. Recent failures keep the suite connected to reality instead of turning into a museum of old assumptions.

I would avoid making every grader an LLM judge. Use exact checks for JSON shape, required fields, forbidden actions, citations, and tool parameters. Use an LLM judge only when the quality dimension is semantic, such as whether a summary preserved the actual decision or whether a support response answered the customer without inventing policy.

The CI gate should be strict on safety and compatibility and a little more tolerant on subjective quality while the suite is young. A broken schema should block immediately. A one-point drop in style quality should trigger review. Treating every signal the same is how teams either ignore the gate or make it too noisy to use.

What I Would Measure#

The dashboard should show pass rate, fail count by category, flaky examples, average judge score, and examples added from production. Flaky examples deserve special attention because they often mean the expected answer is under-specified or the model temperature is too high for the task.

Dataset freshness matters more than dataset size. A stale eval suite creates confidence without coverage. I would require every material production incident to produce either a new eval case or an explicit decision that the incident is not worth guarding against.

The other metric is developer friction. If the eval suite takes twenty minutes and fails without a useful explanation, engineers will route around it. Fast feedback with concrete failing examples is what makes the gate part of the development loop.

Where This Connects#

This post is one piece of the production agent lifecycle. The adjacent pieces matter because the failure usually does not stay inside one layer. A tracing problem becomes an eval problem. An eval problem becomes an ownership problem. A routing problem becomes a cost and security problem if nobody can see what changed.

• Why You Need Offline and Online Evals
• Your Agent Passes Every Test and Still Gets the Date Wrong
• Evaluating AI Agent Skills with Skill Eval
• Who Actually Owns Eval Quality
• Why Your Traces Are Lying to You

If you are using this as a checklist, read those posts as dependencies rather than as optional background. The stack only becomes reliable when the evidence loop, the release loop, and the security loop all point at the same production behavior.

FAQ#

How many examples do I need for an offline eval?#

Start with 20 to 50 high-quality examples. Coverage and clarity matter more than volume at the beginning. Add cases from production failures over time so the dataset grows in response to real risk.

Should offline evals block every prompt change?#

They should block changes that break safety, schema compatibility, tool behavior, or core task success. Softer quality signals can start as warnings until the rubric is calibrated and the team trusts the scores.

What belongs in a golden dataset?#

Include common successful paths, known failure modes, edge cases, adversarial inputs, and domain-specific examples that only a knowledgeable reviewer would catch. Avoid synthetic-only datasets because they miss the weirdness of real usage.

Can I use an LLM judge for all checks?#

You can, but you should not. Deterministic checks are cheaper and more reliable for exact requirements. Use LLM judges for open-ended quality dimensions that cannot be captured by schema or string checks.

How do I handle flaky eval results?#

Label flaky cases, inspect them manually, and decide whether the expected output is ambiguous or the model behavior is unstable. Flaky evals should not silently remain in the blocking path.

When should I update the dataset?#

Update it after production incidents, policy changes, product behavior changes, and major model migrations. A dataset that does not change while the product changes is no longer measuring the right system.

Implementation Review Checklist#

Before I would call this production-ready, I would ask five questions in the review. First, does the team know which artifact changed because of this lesson? A trace field, eval case, prompt file, routing rule, dashboard, or security policy should be visible in the repo or the runbook. If the only output is agreement, the lesson has not landed yet.

Second, can the team reproduce the failure mode in a lower environment? Production AI work gets messy when the only evidence is a screenshot, a Slack thread, or a vague user complaint. A reproducible example gives the team something concrete to test against after the fix.

Third, does the fix have an owner after the first merge? Most agent systems decay slowly. Datasets go stale, prompts drift, routing rules stop matching traffic, and permissions expand. The owner is the person who notices that drift before it becomes an incident.

Fourth, is rollback obvious? If a prompt, model, parser, evaluator, or security rule makes behavior worse, the team should know how to return to the last known-good version without rewriting the system under pressure.

Fifth, did the incident or improvement strengthen the loop? A good production process compounds. The trace becomes an eval. The eval becomes a gate. The gate becomes a safer release. That is the practical standard I would use for every idea in this series.

I would also ask whether the change makes the system easier to explain to a new teammate. Production AI stacks fail when the important behavior lives in someone s memory. If the reason for a prompt rule, routing threshold, eval example, or tool permission is not written down, the next person will eventually delete it while cleaning up what looks like accidental complexity.

The review should include one negative example. Show the input that used to fail, the trace or eval result that made the failure visible, and the current behavior after the change. This keeps the team grounded in evidence. It also stops the review from becoming abstract, which is where AI engineering discussions often drift.

For agent systems, I would check the cost impact and the quality impact together. A fix that improves quality by calling a frontier model three extra times may be correct for a regulated workflow and wrong for a low-margin support workflow. A cost reduction that removes useful context may look good for a week and then show up as worse user outcomes. The trade-off has to be explicit.

The last check is operational ownership. If the change creates a new dashboard, who looks at it? If it creates a new eval, who updates it? If it creates a new permission boundary, who approves exceptions? The difference between a strong production practice and a temporary cleanup is usually whether the maintenance path is obvious.

None of this needs to be heavy. A short pull request note, one linked trace, one eval case, and one owner are enough for many changes. The point is to leave a trail that future engineering work can build on. That is how AI systems become more reliable over time instead of slowly collecting unexplained rules.