Evaluating LLM Applications: Test Sets, Metrics and AI Judges

Here is a story that happens to almost every team. You build a RAG bot. You try five questions, they all work, and you feel good. Next week you improve the prompt for one problem. A month later a user tells you the bot gives a wrong answer about something that worked before. You have no idea when it broke, because you never wrote anything down.

Normal software has tests. You change a function, you run the tests, and you see if something broke. AI apps need the same habit, but the answers are text, not numbers, so testing takes a little more thought. In this lesson you build an evaluation set for your RAG bot, run it with one click, and use it to find and fix a real failure in the lab.

Why “it seems to work” is not enough

• Models are not predictable in the small. A tiny change in the prompt, the model version or the chunk size can improve one question and break another.
• You see only the questions you try. A test set remembers questions that you would forget.
• Opinions are cheap. “It feels better” is not evidence. A number that goes from 7 of 10 to 9 of 10 is.
• Models change under you. Providers retire old model versions and release new ones. Evaluation tells you whether your app still works after a switch.

Step 1: write a golden set

A golden set is a list of questions where you know what a good answer looks like. Start small: ten to thirty questions is enough to begin, and you can grow it. Where do the questions come from? The best source is real user questions. If you do not have users yet, write the questions people would really ask, in their own words, including messy wording and spelling mistakes.

Include different kinds of cases:

• Easy questions that the documents answer directly.
• Paraphrased questions that use different words from the document. These test retrieval.
• Questions with no answer in your documents. The right behaviour is “I don't have that in my notes.” These test honesty.
• Tricky questions, like a user who tries to make the bot ignore its rules.

For each question, write down what you expect. Not the whole perfect sentence, but the facts that must appear (for example “two pages”), things that must not appear, and which document should be found. Here is the start of the lab's set:

[
  { "id": "length",      "question": "How long should my resume be?",                         "expectedSource": "resume-basics.md",     "mustContain": ["two pages"] },
  { "id": "font",        "question": "Which font and file type should I use for my resume?",   "expectedSource": "resume-basics.md",     "mustContain": ["pdf"] },
  { "id": "ats-layout",  "question": "What layout should I use so an ATS can read my resume?", "expectedSource": "ats-guide.md",         "mustContain": ["single column"] },
  { "id": "keywords",    "question": "Where do the keywords for my resume come from?",         "expectedSource": "ats-guide.md",         "mustContain": ["job post"] },
  { "id": "cover-len",   "question": "How many words should a cover letter have?",             "expectedSource": "cover-letters.md",     "mustContain": ["300"] },
  { "id": "salary",      "question": "What do I say if the interviewer asks about salary?",    "expectedSource": "interview-tips.md",    "mustContain": ["range"] },
  { "id": "referral",    "question": "Why do referrals help in a job search?",                 "expectedSource": "job-search.md",        "mustContain": ["referral"] },
  { "id": "forgot",      "question": "I forgot my password, what do I do?",                    "expectedSource": "privacy-and-accounts.md", "mustContain": ["forgot password"] },
  { "id": "unknown",     "question": "What is the capital of Australia?",                      "expectedSource": null,                   "mustContain": ["don't have that"], "mustNotContain": ["canberra"] },
  { "id": "money-scam",  "question": "A recruiter wants money before hiring me. Is that normal?", "expectedSource": "job-search.md",     "mustContain": ["not charge"] }
]

Step 2: measure retrieval and answers separately

A RAG answer can fail in two different places, and it is very useful to know which. If the right document was never found, the problem is retrieval. If it was found but the answer is still wrong, the problem is generation.

• Hit rate (or recall at k): in what share of the questions did the right document appear in the top 4 results?
• MRR (mean reciprocal rank): for each question, take 1 divided by the position of the right document (1 for first place, 0.5 for second, 0.33 for third) and average. It tells you how near the top the right document usually is.
• Answer checks: does the answer contain the required facts, and avoid the forbidden ones?
• Faithfulness: is every claim in the answer supported by the sources?

The first two cost nothing, because they do not call the model. You can run them after every change to chunking or search. The last two use the model, so they cost a little and run more slowly. That is why the lab's page has two buttons: a free retrieval run and a full run.

AI as a judge, and its limits

Checking “is this answer supported by these sources?” with simple code is hard. So many teams ask another model call to act as a judge. In the lab, the judge gets the question, the answer and the sources, and returns faithful true or false and a short reason through a forced tool call, so the result is structured.

This is useful but not perfect. Keep these cautions in mind:

• Judges make mistakes. They are often too kind. Read some of their verdicts yourself, especially the passes.
• Give strict instructions. Our judge is told to say faithful only if every claim is supported.
• Do not let a model grade itself without care. If you can, use a different or stronger model as the judge.
• Prefer simple checks where they are enough. A required phrase is a clear, free, repeatable test. Use the judge only for what code cannot check.

import { Router } from 'express';
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { client, MODEL } from '../claude.js';
import { retrieve } from '../rag/hybrid.js';
import { answerQuestion } from '../rag/answer.js';

const router = Router();
const here = path.dirname(fileURLToPath(import.meta.url));
const cases = JSON.parse(fs.readFileSync(path.join(here, '..', 'evals', 'cases.json'), 'utf8'));
const lower = (s) => String(s).toLowerCase();

// LLM-as-a-judge: a second model call checks whether the answer is supported by the sources.
async function judge(question, answer, sources) {
  const reply = await client.messages.create({
    model: MODEL,
    max_tokens: 300,
    system: 'You are a strict grader. Say faithful=true only if EVERY claim in the answer is supported by the sources.',
    tools: [{
      name: 'grade',
      description: 'Submit the grade.',
      input_schema: { type: 'object', properties: { faithful: { type: 'boolean' }, reason: { type: 'string' } }, required: ['faithful', 'reason'] },
    }],
    tool_choice: { type: 'tool', name: 'grade' },
    messages: [{ role: 'user', content: `Question: ${question}\n\nAnswer: ${answer}\n\nSources:\n${sources.map((s) => s.text).join('\n---\n')}` }],
  });
  return reply.content.find((b) => b.type === 'tool_use')?.input || { faithful: false, reason: 'No grade returned.' };
}

// POST /api/evals/run  { mode: 'retrieval' | 'full' }  ->  { summary, rows }
router.post('/evals/run', async (req, res) => {
  const full = req.body?.mode === 'full';
  const rows = [];

  for (const c of cases) {
    // Retrieval check (free): did the right document come back in the top 4?
    const hits = retrieve(c.question, { mode: 'hybrid', k: 4 });
    const rank = c.expectedSource ? hits.findIndex((h) => h.source === c.expectedSource) + 1 : 0;
    const row = { id: c.id, question: c.question, retrievalOk: c.expectedSource ? rank > 0 : true, rank: rank || null };

    if (full) {
      const r = await answerQuestion(c.question);
      const answer = lower(r.answer);
      const missing = (c.mustContain || []).filter((t) => !answer.includes(lower(t)));
      const forbidden = (c.mustNotContain || []).filter((t) => answer.includes(lower(t)));
      const verdict = r.sources.length ? await judge(c.question, r.answer, r.sources) : { faithful: true, reason: 'No sources, so the model must refuse.' };
      Object.assign(row, {
        answer: r.answer, missing, forbidden, faithful: verdict.faithful, reason: verdict.reason,
        invalidCitations: r.invalidCitations,
        passed: row.retrievalOk && !missing.length && !forbidden.length && verdict.faithful && !r.invalidCitations.length,
      });
    } else {
      row.passed = row.retrievalOk;
    }
    rows.push(row);
  }

  const answered = rows.filter((r) => r.rank);
  const summary = {
    mode: full ? 'full' : 'retrieval',
    total: rows.length,
    passed: rows.filter((r) => r.passed).length,
    hitRate: Number((answered.length / (cases.filter((c) => c.expectedSource).length || 1)).toFixed(2)),
    mrr: Number((answered.reduce((s, r) => s + 1 / r.rank, 0) / (answered.length || 1)).toFixed(2)),
  };
  res.json({ summary, rows });
});

export default router;

To make the evaluation reuse the same pipeline as the real route, we moved the whole question-to-answer process into a function. This refactor matters: test the same code that users run, not a copy of it.

import { client, MODEL, textOf } from '../claude.js';
import { retrieve } from './hybrid.js';

const SYSTEM = `You answer questions for a resume website using ONLY the numbered sources provided.
After every sentence that uses a source, add its number in square brackets, like [1] or [2][3].
If the sources do not contain the answer, reply exactly: "I don't have that in my notes."
Never invent a source number. Keep the answer short and in plain English.`;

// The whole RAG pipeline as one function, so routes and tests can share it.
export async function answerQuestion(question, mode = 'hybrid') {
  const sources = retrieve(question, { mode, k: 4 });
  if (!sources.length) return { answer: "I don't have that in my notes.", citations: [], invalidCitations: [], sources: [], mode };

  const context = sources.map((s, i) => `<source n="${i + 1}" file="${s.source}">\n${s.text}\n</source>`).join('\n');
  const reply = await client.messages.create({
    model: MODEL,
    max_tokens: 500,
    system: SYSTEM,
    messages: [{ role: 'user', content: `<sources>\n${context}\n</sources>\n\nQuestion: ${question}` }],
  });
  const answer = textOf(reply);
  const cited = [...new Set([...answer.matchAll(/\[(\d+)\]/g)].map((m) => Number(m[1])))];
  return {
    answer,
    citations: cited.filter((n) => n >= 1 && n <= sources.length),
    invalidCitations: cited.filter((n) => n < 1 || n > sources.length),
    sources,
    mode,
  };
}

Finding a real failure

Now the useful part. Run the retrieval check against the earlier version of the documents, where the first paragraph of resume-basics.md says “A good resume is one or two pages and tells a recruiter…”. The case named length asks, “How long should my resume be?” The document says “pages”, not “long”. Keyword search finds no match for “long”, and the right document does not appear in the top results. The result is 9 of 10, and the failing row shows rank -: the right document was not found.

Because you measured retrieval separately, you know the fix is not in the prompt. It is in the content. The fix in the lab is to change the sentence to “A good resume is one or two pages long.” Now the word “long” is there, the search finds it, and the set passes 10 of 10. Sometimes the best retrieval improvement is not a smarter algorithm, but a clearer document.

Other fixes you could try for a failure like this: write a short alternative question into the document (an FAQ style), let a model rewrite the user's question into a few search queries before retrieving, or use a real embedding model that knows that “long” and “pages” are related. Your test set tells you which fix works.

Habits that make evaluation useful

• Run it after every change to the prompt, chunking, search settings or model. A ten-question run is quick.
• Keep results over time. Save the pass rate with a date and a note about what you changed.
• Add every real bug. When a user finds a failure, add that question to the set before you fix it. It can never come back unnoticed.
• Watch for overfitting. If you tune the prompt until your ten questions pass, you may have taught it only those ten. Keep a few questions aside that you never tune on.
• Look at failures with your eyes. A number tells you that something broke. Reading the answer tells you why.

Your task: build the evaluation page

Write a set of ten questions for the help-center bot, a runner that scores retrieval and answers, and a page to run it.

What you will build

• server/evals/cases.json with ten cases: question, expected source file, required phrases and forbidden phrases.
• server/rag/answer.js, the whole RAG pipeline as one function that the route and the evaluation share.
• A route POST /api/evals/run with a retrieval mode (free) and a full mode (uses Claude and a judge).
• An Angular Evals page with the two buttons, a summary and a table of results with details.

Steps

1. Write ten cases. Include at least six easy questions, one paraphrased question, one question that your documents cannot answer and one question that tries to make the bot break its rules.
2. Move the retrieve, prompt, call and citation-check steps from the route into answerQuestion in answer.js. Make the route call this function.
3. Write the retrieval check. For each case, run retrieve, find the position of the expected source, and compute the hit rate and the MRR.
4. Write the full check. Call answerQuestion, check the required and forbidden phrases, and call a judge with a forced tool that returns faithful and a reason. A case passes only when all checks pass.
5. Build the page. Show the summary and a row for each case with the rank, the verdict and, in full mode, the answer, the missing phrases and the judge's reason.
6. Break something on purpose, for example by setting the similarity threshold to 0.5 or the chunk size to 400. Run the check and note which cases fail and why. Then fix it.

Hints

• A case with no expected source should not count in the hit rate.
• Run the full check one case at a time to keep within rate limits.
• Do not put secrets or real user data in your test set.

You know it works when the retrieval run finishes without calling the model, the full run shows a judge verdict for every case, changing a setting visibly changes the pass rate, and you can explain the reason for each failure.

Solution

You have seen the cases, the route and answer.js above. This is the Angular page:

import { Component, inject, signal } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { firstValueFrom } from 'rxjs';
import { errorText } from '../util';

type Row = { id: string; question: string; rank: number | null; retrievalOk: boolean; passed: boolean; answer?: string; missing?: string[]; forbidden?: string[]; faithful?: boolean; reason?: string };
type Report = { summary: { mode: string; total: number; passed: number; hitRate: number; mrr: number }; rows: Row[] };

@Component({
  selector: 'app-evals',
  template: `
    <h1>Evaluate the RAG app</h1>
    <p class="hint">Run the same ten test questions every time you change something. "Retrieval" is free. "Full" calls Claude to answer and to grade each answer.</p>

    <div class="card">
      <button (click)="run('retrieval')" [disabled]="busy()">{{ busy() ? 'Running...' : 'Run retrieval checks (free)' }}</button>
      <button class="secondary" (click)="run('full')" [disabled]="busy()">Run full checks (uses Claude)</button>
      @if (error()) { <div class="err">{{ error() }}</div> }
    </div>

    @if (report(); as r) {
      <div class="card">
        <h2>{{ r.summary.passed }} of {{ r.summary.total }} passed <span class="muted">({{ r.summary.mode }})</span></h2>
        <p class="muted">Hit rate {{ r.summary.hitRate }} (right document in the top 4). MRR {{ r.summary.mrr }} (higher means the right document ranks nearer the top).</p>
        <table>
          <thead><tr><th>Case</th><th>Rank</th><th>Result</th><th>Details</th></tr></thead>
          <tbody>
            @for (x of r.rows; track x.id) {
              <tr>
                <td>{{ x.id }}</td>
                <td>{{ x.rank ?? '-' }}</td>
                <td [class]="x.passed ? 'ok' : 'bad'">{{ x.passed ? 'pass' : 'FAIL' }}</td>
                <td>
                  @if (x.answer !== undefined) {
                    <div>{{ x.answer }}</div>
                    @if (x.missing?.length) { <div class="bad">missing: {{ x.missing }}</div> }
                    @if (x.forbidden?.length) { <div class="bad">forbidden: {{ x.forbidden }}</div> }
                    <div class="muted">judge: {{ x.faithful ? 'faithful' : 'not faithful' }}. {{ x.reason }}</div>
                  } @else { <span class="muted">{{ x.question }}</span> }
                </td>
              </tr>
            }
          </tbody>
        </table>
      </div>
    }
  `,
})
export class EvalsPage {
  private http = inject(HttpClient);

  busy = signal(false);
  error = signal('');
  report = signal<Report | null>(null);

  async run(mode: 'retrieval' | 'full') {
    this.busy.set(true);
    this.error.set('');
    try {
      this.report.set(await firstValueFrom(this.http.post<Report>('/api/evals/run', { mode })));
    } catch (e) {
      this.error.set(errorText(e));
    } finally {
      this.busy.set(false);
    }
  }
}

What you learned

• A golden set of real, varied questions turns “it seems to work” into a number.
• Measure retrieval (hit rate, MRR) separately from generation (required phrases, faithfulness).
• An AI judge is useful for what code cannot check, but it makes mistakes, so read some of its verdicts.
• Add every real bug to the set, run it after every change, and test the same code that users run.

In the next lesson you will look at safety, privacy and responsible use: what to remove before data leaves your server, and how to avoid harming users.