Advanced RAG: Hybrid Search, Reranking and Citations

A basic RAG system works well on easy questions and fails on the rest. Sometimes the user asks with different words than your document, and vector search saves you. Sometimes the user searches for an exact name, code or number, like “Form 16” or “Zendesk”, and keyword search saves you. Neither method wins every time. The best systems use both, check the best results again, and make the final answer prove where it came from.

In this lesson you will build four upgrades: hybrid search, a similarity threshold, a simple reranking step, and checked citations. Then you will compare keyword, vector and hybrid results side by side on one page, so you can see the differences with your own eyes.

Why one search method is not enough

• Keyword search is excellent at exact words: names, product codes, error messages, rare terms. It cannot match a different word with the same meaning.
• Vector search is excellent at meaning: with a good embedding model it can find “how many pages” when the question says “how long”. It can be weak on exact rare terms and numbers, and it always returns something, even when nothing is related.
• Hybrid search runs both and merges the lists. It is more robust than either alone, which is why it is the default in many production systems.

BM25: keyword search done properly

In lesson 13 we scored chunks with a simple shared-word score. Search engines use a better formula called BM25. It keeps the useful idea (rare words count more) and fixes two problems:

• Repetition has a ceiling. If a chunk says “resume” fifty times, it is not fifty times better than a chunk that says it twice. BM25 lets extra repeats help less and less.
• Length is balanced. A very long chunk has more chances to contain the words by accident. BM25 reduces its score a little so short, focused chunks are not unfairly beaten.

There are two settings, called k1 and b. The defaults (1.5 and 0.75) work well in most cases, so you rarely change them. Here is the code:

import { tokenize } from './keyword.js';

// BM25: the standard keyword ranking used by search engines.
// It improves plain keyword matching in two ways:
//  - a word repeated many times helps less and less (saturation, k1)
//  - long chunks are not unfairly favoured (length normalisation, b)
export function bm25Search(query, chunks, k = 8, { k1 = 1.5, b = 0.75 } = {}) {
  const docs = chunks.map((c) => tokenize(`${c.title || ''} ${c.text}`));
  const avgLen = docs.reduce((s, d) => s + d.length, 0) / (docs.length || 1);
  const df = new Map();
  for (const d of docs) for (const t of new Set(d)) df.set(t, (df.get(t) || 0) + 1);
  const q = [...new Set(tokenize(query))];

  return chunks
    .map((chunk, i) => {
      const tf = new Map();
      for (const t of docs[i]) tf.set(t, (tf.get(t) || 0) + 1);
      let score = 0;
      for (const t of q) {
        const f = tf.get(t) || 0;
        if (!f) continue;
        const idf = Math.log(1 + (docs.length - df.get(t) + 0.5) / (df.get(t) + 0.5));
        score += idf * ((f * (k1 + 1)) / (f + k1 * (1 - b + b * (docs[i].length / avgLen))));
      }
      return { ...chunk, score: Number(score.toFixed(4)) };
    })
    .filter((c) => c.score > 0)
    .sort((a, b2) => b2.score - a.score)
    .slice(0, k);
}

Merging two rankings: Reciprocal Rank Fusion

BM25 gives scores like 4.7. Cosine similarity gives scores like 0.31. These numbers live on different scales, so you cannot just add them. A simple, strong solution is Reciprocal Rank Fusion (RRF). It ignores the scores and only uses the position in each list. An item that is first in a list gets 1 / (60 + 1), the second gets 1 / (60 + 2), and so on. You add up what each item earns across all the lists. An item that is high in both lists gets the largest total.

Think of two judges ranking singers. You do not add their raw marks, because one judge is harsh and the other is generous. You add up each singer's rank from each judge. The singer who is near the top for both wins. The number 60 is a common constant that stops first place from being too dominant.

The similarity threshold: knowing when nothing is relevant

Vector search has a quiet danger. It always returns the nearest chunks, even when none of them is near. If a user asks “What is the capital of Australia?”, the search will still hand you three chunks about resumes, and the model may try to use them. The fix is a minimum similarity. Chunks below it are dropped, and if nothing remains, the bot says “I don't have that in my notes.” without calling the model.

In the lab the cutoff is 0.1. It was found by trying real and unrelated questions: unrelated queries scored around 0.05 to 0.07, while real matches scored 0.12 or more. Your number will be different with a different embedding model, so measure it on your own data. This is a classic example of why you need a test set (lesson 22).

Reranking: a second, closer look

Search methods are fast but rough. They compare a question with a chunk without reading them together. A reranker is a slower, smarter step that looks at the question and each candidate together and decides how well the chunk really answers the question. The usual pattern is: retrieve about 20 candidates cheaply, rerank them, and keep the best 3 to 5.

Production systems use a trained reranker model, often called a cross-encoder, from a provider or an open model. We do not want an account in this lab, so we use a very small stand-in: for each candidate, we count how many of the question's words it contains, and prefer the ones with the highest coverage. It shows you where the reranker sits in the pipeline, and you can swap it for a real model later.

import { getStore } from './store.js';
import { bm25Search } from './bm25.js';
import { tokenize } from './keyword.js';

// Reciprocal Rank Fusion: combine several ranked lists using only the RANK of each item.
// Scores from different systems (BM25 and cosine) are not comparable, ranks are.
export function rrf(lists, { k = 60 } = {}) {
  const fused = new Map();
  for (const list of lists) {
    list.forEach((item, rank) => {
      const entry = fused.get(item.id) || { ...item, score: 0 };
      entry.score += 1 / (k + rank + 1);
      fused.set(item.id, entry);
    });
  }
  return [...fused.values()].map((x) => ({ ...x, score: Number(x.score.toFixed(5)) })).sort((a, b) => b.score - a.score);
}

// A very small "reranker": after the first search, look at the few best chunks again
// and favour the ones that contain MORE of the question's words in one place.
// Real systems use a cross-encoder model here; this shows where it fits.
export function rerank(query, items) {
  const q = [...new Set(tokenize(query))];
  return items
    .map((it) => {
      const words = new Set(tokenize(it.text));
      const covered = q.filter((t) => words.has(t)).length;
      return { ...it, coverage: q.length ? Number((covered / q.length).toFixed(2)) : 0 };
    })
    .sort((a, b) => b.coverage - a.coverage || b.score - a.score);
}

// Vector search ALWAYS returns the nearest chunks, even when nothing is near.
// Below this similarity a chunk is noise, so we drop it. Tune it on your own data.
export const MIN_SIMILARITY = 0.1;

// mode: 'keyword' | 'vector' | 'hybrid'
export function retrieve(query, { mode = 'hybrid', k = 4 } = {}) {
  const store = getStore();
  const all = [...store.items.values()];
  const keyword = bm25Search(query, all, 8);
  const vector = store.query(query, { k: 8 }).filter((h) => h.score >= MIN_SIMILARITY);
  if (mode === 'keyword') return keyword.slice(0, k);
  if (mode === 'vector') return vector.slice(0, k);
  return rerank(query, rrf([keyword, vector]).slice(0, 6)).slice(0, k);
}

Citations that the server checks

An answer without sources is hard to trust. We ask the model to put a number like [1] after each sentence that uses source 1. This helps users, because they can check the source, and it helps you, because it makes the model show its work.

But a model can write [9] when there are only four sources. So the server checks the citations. It reads all the [n] marks in the answer, keeps only the numbers that exist, and reports the others as invalid. The Angular page shows a red warning for invalid citations and highlights the sources that were really cited.

This is a very good pattern, because it moves a check out of the model's “promise” and into your code, which can be tested. Remember it for the whole course: do not trust, verify.

import { Router } from 'express';
import { client, MODEL, textOf } from '../claude.js';
import { retrieve } from '../rag/hybrid.js';

const router = Router();

const SYSTEM = `You answer questions for a resume website using ONLY the numbered sources provided.
After every sentence that uses a source, add its number in square brackets, like [1] or [2][3].
If the sources do not contain the answer, reply exactly: "I don't have that in my notes."
Never invent a source number. Keep the answer short and in plain English.`;

// POST /api/rag-plus  { question, mode }  ->  { answer, citations, sources, mode }
router.post('/rag-plus', async (req, res) => {
  const question = String(req.body?.question || '').trim().slice(0, 500);
  if (!question) return res.status(400).json({ error: { message: 'Please type a question.' } });
  const mode = ['keyword', 'vector', 'hybrid'].includes(req.body?.mode) ? req.body.mode : 'hybrid';

  const sources = retrieve(question, { mode, k: 4 });
  if (!sources.length) return res.json({ answer: "I don't have that in my notes.", citations: [], sources: [], mode });

  const context = sources.map((s, i) => `<source n="${i + 1}" file="${s.source}">\n${s.text}\n</source>`).join('\n');
  const reply = await client.messages.create({
    model: MODEL,
    max_tokens: 500,
    system: SYSTEM,
    messages: [{ role: 'user', content: `<sources>\n${context}\n</sources>\n\nQuestion: ${question}` }],
  });
  const answer = textOf(reply);

  // Check the citations: keep only numbers that really exist. A made-up [9] is dropped and reported.
  const cited = [...new Set([...answer.matchAll(/\[(\d+)\]/g)].map((m) => Number(m[1])))];
  const citations = cited.filter((n) => n >= 1 && n <= sources.length);
  const invalid = cited.filter((n) => n < 1 || n > sources.length);

  res.json({ answer, citations, invalidCitations: invalid, sources, mode });
});

export default router;

Your task: build the RAG+ page

Build a page where the same question can be answered with keyword, vector or hybrid retrieval, and where citations are checked.

What you will build

• server/rag/bm25.js with a BM25 search.
• server/rag/hybrid.js with rrf, rerank and retrieve, including the minimum similarity.
• A route POST /api/rag-plus that takes a question and a mode, calls Claude with numbered sources, and returns the answer, the valid citations and the invalid ones.
• An Angular page with a mode selector, the answer, and a source list that shows which sources were cited.

Steps

1. Write bm25Search. Compute the average chunk length and how many chunks contain each word. Score each chunk with the BM25 formula and return the best ones that scored above zero.
2. Write rrf. Take a list of ranked lists, add 1 / (60 + rank + 1) to each item for each list it appears in, and sort by the total.
3. Write retrieve. Run BM25 and vector search. Drop vector results below 0.1. For keyword and vector modes return the list directly. For hybrid mode fuse them, rerank the top six and return four.
4. Write the route. Number the sources in the prompt, write the system prompt that asks for [n] after each used sentence, and then check the citations with a regular expression.
5. Build the page. Show the answer, show invalid citations in red, and list the sources with their score and a “cited” label.
6. Compare modes. Ask the same three questions in each mode and write down which mode found the right document. Add one question with an exact rare word and one with different wording.

Hints

• To read the citations, use answer.matchAll(/\[(\d+)\]/g) and remove duplicates with a Set.
• Do not call Claude when there are no sources.
• Add a way to force a bad citation while testing, so you can prove your check works. A test question that makes your fake or mock model write [9] is enough.

You know it works when keyword mode finds exact words, vector mode finds related words, hybrid mode is at least as good as the better of the two, an unrelated question gets “I don't have that in my notes.” without a model call, and an invented citation is flagged.

Solution

The core files are above. This is the Angular page; the zip holds the whole project.

import { Component, inject, signal } from '@angular/core';
import { FormsModule } from '@angular/forms';
import { HttpClient } from '@angular/common/http';
import { firstValueFrom } from 'rxjs';
import { errorText } from '../util';

type Source = { id: string; source: string; text: string; score: number; coverage?: number };
type Reply = { answer: string; citations: number[]; invalidCitations?: number[]; sources: Source[]; mode: string };

@Component({
  selector: 'app-rag-plus',
  imports: [FormsModule],
  template: `
    <h1>Better RAG: hybrid search and citations</h1>
    <p class="hint">Compare keyword, vector and hybrid retrieval. The answer must cite its sources, and the server checks the numbers.</p>

    <div class="card">
      <label for="q">Question</label>
      <input id="q" [(ngModel)]="question" (keyup.enter)="ask()" />
      <label for="m">Retrieval mode</label>
      <select id="m" [(ngModel)]="mode">
        <option value="hybrid">Hybrid (BM25 + vector, fused, reranked)</option>
        <option value="keyword">Keyword only (BM25)</option>
        <option value="vector">Vector only</option>
      </select>
      <button (click)="ask()" [disabled]="busy() || !question.trim()">{{ busy() ? 'Working...' : 'Ask' }}</button>
      @if (error()) { <div class="err">{{ error() }}</div> }
    </div>

    @if (reply(); as r) {
      <div class="card">
        <h2>Answer <span class="muted">({{ r.mode }})</span></h2>
        <div class="answer">{{ r.answer }}</div>
        @if (r.invalidCitations?.length) { <div class="err">The model cited sources that do not exist: {{ r.invalidCitations }}. Do not trust this answer.</div> }
        <h2>Sources</h2>
        @for (s of r.sources; track s.id; let i = $index) {
          <p>
            <strong [class.ok]="r.citations.includes(i + 1)">[{{ i + 1 }}] {{ s.id }}</strong>
            <span class="muted">score {{ s.score }}@if (s.coverage !== undefined) { , coverage {{ s.coverage }} }</span>
            @if (r.citations.includes(i + 1)) { <span class="ok"> cited</span> }
          </p>
          <pre class="out">{{ s.text }}</pre>
        }
      </div>
    }
  `,
})
export class RagPlusPage {
  private http = inject(HttpClient);

  question = 'Which fonts and file type are best for a resume?';
  mode = 'hybrid';
  busy = signal(false);
  error = signal('');
  reply = signal<Reply | null>(null);

  async ask() {
    this.busy.set(true);
    this.error.set('');
    try {
      this.reply.set(await firstValueFrom(this.http.post<Reply>('/api/rag-plus', { question: this.question, mode: this.mode })));
    } catch (e) {
      this.error.set(errorText(e));
    } finally {
      this.busy.set(false);
    }
  }
}

What you learned

• Hybrid search combines keyword strength (exact terms) and vector strength (meaning).
• Reciprocal Rank Fusion merges rankings by position, so you never compare scores from different systems.
• A similarity threshold lets the system say “nothing relevant found”.
• A reranker takes a closer look at a short list. Citations make answers checkable, and the server verifies them.

Retrieval is done. In the next lessons we move from answering questions to taking actions: AI agents.