Running AI in Production: Cost, Reliability and Monitoring

A demo works on your laptop for one person. A product has to work for thousands of people, at any hour, without a surprise bill. The model API is a remote service that costs money for every call, sometimes takes many seconds, and sometimes fails. Production is the work of making that acceptable.

In this lesson you add five production basics to your AI Lab: a cache, a rate limit, a timeout, a fallback and metrics. None of them is complicated. Together they are the difference between a toy and a service you can run.

What does an AI feature cost?

API providers charge by tokens: a price for input tokens (what you send) and a price for output tokens (what the model writes), usually per million tokens. Output tokens often cost more than input tokens. A call that sends 1,500 tokens of context and gets 200 tokens back costs a small amount, but multiply it by 100,000 calls a month and it is a real number.

To estimate your cost, you need four numbers: calls per day, input tokens per call, output tokens per call and the current price. Write them down before you launch. Prices and models change, so always read the provider's pricing page for the current values. In the lab the prices are set in the .env file, so the page can show an estimate.

Ways to spend less without hurting quality:

• Cache. If many users ask the same question, answer it once.
• Send less. Three good chunks instead of ten. A shorter system prompt.
• Limit the output. Set a sensible max_tokens and ask for short answers.
• Use a smaller model for easy steps, and a larger one only where needed. Your evaluation set (lesson 22) tells you if the cheaper model is good enough.
• Use prompt caching for a long, repeated prefix, as in the Claude API course.
• Stop wasteful loops with step limits and token budgets (lesson 17).

The cache

A cache stores the answer to a question for a while. When the same question comes again, you return the stored answer immediately. It costs no tokens and takes a few milliseconds.

• Normalise the key. Make the question lower case and collapse spaces, so “How do I...” and “ how do I...” count as the same.
• Set an expiry. Our cache keeps answers for 10 minutes. If your documents change often, use a shorter time or clear the cache when they change.
• Set a size limit. A cache that grows forever is a memory leak. Ours keeps at most 200 entries and drops the oldest.
• Be careful with personal answers. Only cache answers that are the same for everyone. Never cache an answer that depends on one user's private data under a shared key.

The rate limit

Without a limit, one user (or one bug, or one attacker) can send thousands of requests and burn your budget in an hour. A rate limit allows each client a fixed number of requests per time window. Ours allows 10 per minute for each IP address. When the limit is passed we return status 429 with a Retry-After header that says how many seconds to wait. The Angular page shows the friendly message.

In a real product you would limit by signed-in user as well as by IP, set a daily cost limit for each user, and keep the counters in a shared store like Redis when you run more than one server. Our in-memory version is correct for one process and good for learning.

Timeouts, retries and fallbacks

Remote services fail. They return errors, they slow down, or they hang. Plan for it:

• Timeout. Never wait forever. We pass timeout: 15000 so a call is cancelled after 15 seconds. A slow answer is often worse than a clear fallback.
• Limited retries. A short retry helps with temporary errors, but too many retries make an outage worse. We allow one.
• Fallback. If the AI fails, show something useful instead of an error page. Our route returns the best passages from the search with a note, “The AI assistant is busy right now. These are the most relevant notes.” The search part works even when the model does not.

A fallback is a product decision. Ask: if the AI is down for an hour, what is the most helpful thing we can still show? For a resume tool it might be saved templates or a checklist.

Metrics and logs

You cannot manage what you cannot see. Our small ops.js records, for each request: how long it took, how many tokens it used, whether it came from the cache, whether it fell back and whether there was an error. From these it calculates the number of requests, the cache hit rate, the average time, the total tokens and the estimated cost, and the page shows them.

Two rules about logs. First, log numbers, not text. A log file full of user questions is a privacy risk (lesson 23). Second, decide in advance what you will alert on: a rise in errors, a drop in the cache hit rate, a sudden jump in tokens or cost. In a larger system you would send these numbers to a monitoring tool and set alerts, but the habit starts with counting.

import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';

const here = path.dirname(fileURLToPath(import.meta.url));
const LOG = path.join(here, 'runtime', 'usage.jsonl');

// ---- 1. Response cache: the same question within 10 minutes costs nothing. ----
const cache = new Map();
const TTL = 10 * 60 * 1000;
const MAX_ENTRIES = 200;
export const cacheKey = (...parts) => parts.map((p) => String(p).toLowerCase().replace(/\s+/g, ' ').trim()).join('|');
export const cacheGet = (key) => {
  const hit = cache.get(key);
  if (!hit) return null;
  if (hit.expires < Date.now()) { cache.delete(key); return null; }
  return hit.value;
};
export const cacheSet = (key, value) => {
  cache.set(key, { value, expires: Date.now() + TTL });
  if (cache.size > MAX_ENTRIES) cache.delete(cache.keys().next().value); // drop the oldest
};

// ---- 2. Rate limit: at most `limit` calls per client per minute. ----
const windows = new Map();
export function rateLimit({ limit = 10, windowMs = 60_000 } = {}) {
  return (req, res, next) => {
    const now = Date.now();
    const ip = req.ip || 'unknown';
    const w = windows.get(ip);
    if (!w || w.reset < now) {
      windows.set(ip, { count: 1, reset: now + windowMs });
      return next();
    }
    if (++w.count > limit) {
      res.setHeader('Retry-After', Math.ceil((w.reset - now) / 1000));
      return res.status(429).json({ error: { message: 'Too many requests. Please wait a minute and try again.', status: 429 } });
    }
    next();
  };
}

// ---- 3. Metrics and a usage log. ----
const m = { requests: 0, cacheHits: 0, fallbacks: 0, errors: 0, inputTokens: 0, outputTokens: 0, totalMs: 0 };

const price = () => ({
  input: Number(process.env.PRICE_INPUT_PER_MTOK) || 0,
  output: Number(process.env.PRICE_OUTPUT_PER_MTOK) || 0,
  currency: process.env.PRICE_CURRENCY || 'USD',
});

export function record({ route, ms, usage = { input: 0, output: 0 }, cached = false, fallback = false, error = false }) {
  m.requests++;
  m.totalMs += ms;
  if (cached) m.cacheHits++;
  if (fallback) m.fallbacks++;
  if (error) m.errors++;
  m.inputTokens += usage.input;
  m.outputTokens += usage.output;
  try {
    fs.mkdirSync(path.dirname(LOG), { recursive: true });
    // Log numbers only. Never write the user's text to a log file.
    fs.appendFileSync(LOG, JSON.stringify({ at: new Date().toISOString(), route, ms, cached, fallback, error, ...usage }) + '\n');
  } catch { /* logging must never break a request */ }
}

export function metrics() {
  const p = price();
  const cost = (m.inputTokens * p.input + m.outputTokens * p.output) / 1_000_000;
  return {
    ...m,
    avgMs: m.requests ? Math.round(m.totalMs / m.requests) : 0,
    cacheHitRate: m.requests ? Number((m.cacheHits / m.requests).toFixed(2)) : 0,
    estimatedCost: p.input || p.output ? Number(cost.toFixed(4)) : null,
    currency: p.currency,
  };
}

import { client, MODEL, textOf } from '../claude.js';
import { retrieve } from './hybrid.js';

const SYSTEM = `You answer questions for a resume website using ONLY the numbered sources provided.
After every sentence that uses a source, add its number in square brackets, like [1] or [2][3].
If the sources do not contain the answer, reply exactly: "I don't have that in my notes."
Never invent a source number. Keep the answer short and in plain English.`;

// Lesson 24 changes: the call now has a timeout and the result reports token usage.
export async function answerQuestion(question, mode = 'hybrid', { timeoutMs = 20000 } = {}) {
  const sources = retrieve(question, { mode, k: 4 });
  const usage = { input: 0, output: 0 };
  if (!sources.length) return { answer: "I don't have that in my notes.", citations: [], invalidCitations: [], sources: [], mode, usage };

  const context = sources.map((s, i) => `<source n="${i + 1}" file="${s.source}">\n${s.text}\n</source>`).join('\n');
  const reply = await client.messages.create(
    {
      model: MODEL,
      max_tokens: 500,
      system: SYSTEM,
      messages: [{ role: 'user', content: `<sources>\n${context}\n</sources>\n\nQuestion: ${question}` }],
    },
    { timeout: timeoutMs, maxRetries: 1 }, // fail fast: a slow answer is worse than a fallback
  );
  usage.input = reply.usage?.input_tokens || 0;
  usage.output = reply.usage?.output_tokens || 0;
  const answer = textOf(reply);
  const cited = [...new Set([...answer.matchAll(/\[(\d+)\]/g)].map((m) => Number(m[1])))];
  return {
    answer,
    citations: cited.filter((n) => n >= 1 && n <= sources.length),
    invalidCitations: cited.filter((n) => n < 1 || n > sources.length),
    sources,
    mode,
    usage,
  };
}

import { Router } from 'express';
import { answerQuestion } from '../rag/answer.js';
import { retrieve } from '../rag/hybrid.js';
import { cacheGet, cacheSet, cacheKey, rateLimit, record, metrics } from '../ops.js';

const router = Router();

// POST /api/ops/ask  { question }
// Cached, rate limited, timed, measured, and it degrades gracefully when the AI is unavailable.
router.post('/ops/ask', rateLimit({ limit: 10 }), async (req, res) => {
  const started = Date.now();
  const question = String(req.body?.question || '').trim().slice(0, 500);
  if (!question) return res.status(400).json({ error: { message: 'Please type a question.' } });

  const key = cacheKey('ask', question);
  const cached = cacheGet(key);
  if (cached) {
    record({ route: 'ask', ms: Date.now() - started, cached: true });
    return res.json({ ...cached, cached: true });
  }

  try {
    const result = await answerQuestion(question, 'hybrid', { timeoutMs: 15000 });
    cacheSet(key, result);
    record({ route: 'ask', ms: Date.now() - started, usage: result.usage });
    return res.json({ ...result, cached: false });
  } catch (e) {
    // Fallback: the AI failed or timed out, so show the best passages instead of an error page.
    record({ route: 'ask', ms: Date.now() - started, fallback: true, error: true });
    console.error('[ops] AI failed:', e?.status || '', e?.message);
    const sources = retrieve(question, { mode: 'hybrid', k: 3 });
    return res.json({
      answer: 'The AI assistant is busy right now. These are the most relevant notes:',
      sources, citations: [], invalidCitations: [], fallback: true, cached: false, usage: { input: 0, output: 0 },
    });
  }
});

// GET /api/ops/metrics
router.get('/ops/metrics', (req, res) => res.json(metrics()));

export default router;

Other things to plan

• Secrets. Keep API keys in environment variables or a secret manager, never in code or in the browser. Rotate a key if it ever leaks.
• Streaming. For long answers, stream the text so users see progress (as in the Claude API course).
• Queues. For slow jobs, such as processing a large PDF, put the work in a queue and show progress, rather than holding a web request open.
• Prompt versions. Keep prompts in version control and record which version produced which result.
• Model changes. Models are retired on a schedule. Keep the model name in one setting and run your evaluation set before you switch.
• Release checks. Run the evaluation set automatically before each release.
• A kill switch. A setting that turns the AI feature off quickly when something goes wrong.

Your task: add production basics

Add a cache, a rate limit, a timeout, a fallback and metrics to the question-answering route, and a page that shows what they do.

What you will build

• server/ops.js with a cache (with expiry and a size limit), a rate limit middleware and metrics with a usage log.
• An updated server/rag/answer.js that accepts a timeout and returns token usage.
• A route POST /api/ops/ask that uses all five, and GET /api/ops/metrics.
• An Angular Ops page that shows the answer, whether it was cached or a fallback, and the server metrics.

Steps

1. Write the cache with a Map. Each entry has a value and an expiry time. Return nothing when the entry has expired and remove the oldest entry when you pass 200.
2. Write the rate limit as an Express middleware. Count requests per IP in a time window. After the limit, return 429 with a Retry-After header.
3. Change answerQuestion to pass a timeout and one retry to the SDK call, and return the token usage in the result.
4. Write the route. Check the cache first. Otherwise call answerQuestion inside try/catch. On failure, record the failure and return the best passages as a fallback.
5. Write record and metrics. Count requests, cache hits, fallbacks, errors, tokens and time. Append one line of numbers (no text) to a log file. Calculate the estimated cost from the prices in .env.
6. Build the page. Ask the same question twice and check that the second is cached. Send more than ten questions in a minute and check the 429 message. Stop the AI (use a wrong key) and check the fallback.

Hints

• Normalise the cache key before you look it up.
• Write the log in a try/catch. Logging must never break a request.
• Store generated files like the usage log in a folder that is not committed, such as server/runtime.

You know it works when a repeated question is served from the cache with no tokens, the eleventh request in a minute gets a 429 with a wait time, a failed AI call returns passages and a clear message instead of an error, and the metrics change as you use the page.

Solution

You have read ops.js, answer.js and the route above. This is the Angular page.

import { Component, inject, signal, OnInit } from '@angular/core';
import { FormsModule } from '@angular/forms';
import { HttpClient } from '@angular/common/http';
import { firstValueFrom } from 'rxjs';
import { errorText } from '../util';

type Metrics = { requests: number; cacheHits: number; fallbacks: number; errors: number; inputTokens: number; outputTokens: number; avgMs: number; cacheHitRate: number; estimatedCost: number | null; currency: string };
type Reply = { answer: string; cached: boolean; fallback?: boolean; sources: { id: string; text: string }[] };

@Component({
  selector: 'app-ops',
  imports: [FormsModule],
  template: `
    <h1>Production basics</h1>
    <p class="hint">Ask the same question twice to see the cache work. Ask more than 10 questions in a minute to see the rate limit. The numbers below come from the server.</p>

    <div class="card">
      <label for="q">Question</label>
      <input id="q" [(ngModel)]="question" (keyup.enter)="ask()" />
      <button (click)="ask()" [disabled]="busy() || !question.trim()">{{ busy() ? 'Asking...' : 'Ask' }}</button>
      @if (error()) { <div class="err">{{ error() }}</div> }
      @if (reply(); as r) {
        <div class="answer">{{ r.answer }}</div>
        <p>
          @if (r.cached) { <span class="ok">served from cache (no AI cost)</span> }
          @if (r.fallback) { <span class="bad">fallback: AI unavailable</span> }
        </p>
        @if (r.fallback) { @for (s of r.sources; track s.id) { <pre class="out">{{ s.text }}</pre> } }
      }
    </div>

    @if (metrics(); as m) {
      <div class="card">
        <h2>Server metrics <button class="secondary" style="margin-left:8px" (click)="load()">Refresh</button></h2>
        <table>
          <tbody>
            <tr><td>Requests</td><td>{{ m.requests }}</td></tr>
            <tr><td>Cache hits</td><td>{{ m.cacheHits }} ({{ m.cacheHitRate * 100 }}%)</td></tr>
            <tr><td>Fallbacks / errors</td><td>{{ m.fallbacks }} / {{ m.errors }}</td></tr>
            <tr><td>Average time</td><td>{{ m.avgMs }} ms</td></tr>
            <tr><td>Tokens in / out</td><td>{{ m.inputTokens }} / {{ m.outputTokens }}</td></tr>
            <tr><td>Estimated cost</td><td>{{ m.estimatedCost === null ? 'set PRICE_* in .env' : m.estimatedCost + ' ' + m.currency }}</td></tr>
          </tbody>
        </table>
      </div>
    }
  `,
})
export class OpsPage implements OnInit {
  private http = inject(HttpClient);

  question = 'How should I answer a salary question in an interview?';
  busy = signal(false);
  error = signal('');
  reply = signal<Reply | null>(null);
  metrics = signal<Metrics | null>(null);

  ngOnInit() {
    this.load();
  }

  async load() {
    this.metrics.set(await firstValueFrom(this.http.get<Metrics>('/api/ops/metrics')));
  }

  async ask() {
    this.busy.set(true);
    this.error.set('');
    try {
      this.reply.set(await firstValueFrom(this.http.post<Reply>('/api/ops/ask', { question: this.question })));
    } catch (e) {
      this.error.set(errorText(e));
    } finally {
      this.busy.set(false);
      this.load();
    }
  }
}

What you learned

• Cost is calls times tokens times price. Estimate it before you launch and cut it with caching, smaller prompts and the right model.
• A cache saves money and time, a rate limit protects your budget, and a timeout and fallback keep the feature useful when the AI is slow or down.
• Measure requests, speed, tokens, cache hits, errors and cost. Log numbers, not user text.
• Keep secrets safe, keep prompts versioned and run your evaluation set before each release.

You now have all the pieces. In the final lesson you plan what to learn next and choose a project to build.