How Large Language Models Work

A large language model, or LLM, is a very big neural network trained on a very large amount of text to predict the next token. You met that idea in the last lesson. In this lesson we open the box a little. You will learn what a token is, what the context window limits, how the model is trained in stages, and how settings such as temperature change the way the model picks its next word. By the end you will be able to explain why the same question can give different answers, and you will have a page where you can see the effect of every setting.

Tokens: the pieces a model reads

A model does not read letters or whole words. It reads tokens. A token is a piece of text, often a common word, part of a longer word, or a bit of punctuation. A simple word such as “job” is usually one token. A rare word such as “unemployability” may be split into several pieces. The splitting method is learned from data so that common pieces get their own token, which keeps text short.

As a rough guide for English, one token is about four characters, or three quarters of a word. So 100 words are around 130 tokens. Other languages often need more tokens for the same meaning, because the tokenizer saw less text in them. Numbers, code and unusual names can use many tokens too.

Tokens matter for three reasons. They decide the cost, because most AI services charge by the token for input and output. They decide the limit, because a model can only handle a fixed number of tokens at once. And they explain some odd behaviour: a model that sees “strawberry” as a few pieces rather than ten letters can find it hard to count letters.

The context window

The context window is the maximum number of tokens the model can look at in one go. It includes everything: your instructions, earlier messages in the chat, any documents you pasted, and the answer being written. Think of it as the model’s desk. Anything on the desk can be used. Anything that does not fit is simply not there.

Windows have grown from a few thousand tokens to hundreds of thousands, and some models handle more than a million. But a bigger window is not magic. Longer input costs more and runs slower. Models can also pay less attention to the middle of very long text. A good habit is to put only what the model needs into the window, and to place the most important instructions clearly, near the start or the end.

The model has no memory between separate conversations unless the application stores and re-sends earlier messages. When a chat seems to “remember” you, the app is putting the earlier text back into the context window each time.

How a model is trained

1. Pre-training. The model reads a huge collection of text (books, websites, code, articles) and learns to predict the next token. It makes a guess, compares it with the real next token, and adjusts its weights, billions of times. This is the most expensive stage and builds the broad knowledge and language skill.
2. Instruction tuning. The pre-trained model only continues text. It is then trained on examples of questions with good answers, so that it learns to follow instructions and to behave like an assistant.
3. Preference training. People compare answers and say which one is better. The model is adjusted towards answers people prefer: more helpful, more honest, and less harmful. Methods in this stage include reinforcement learning from human feedback.

After training, the weights are fixed. When you use the model, it does not learn from your chat. This is why a model cannot know about last week’s news, unless a tool provides it.

Choosing the next token

At each step the model produces one raw score for every token in its vocabulary, which can have 100 thousand entries or more. Softmax (lesson 2) turns the scores into probabilities. Then the program must pick one token. How it picks is controlled by a few settings.

Greedy choice always takes the most likely token. The output is steady and repeatable but can be dull and repetitive. Sampling picks randomly in proportion to the probabilities, so a token with 20 percent chance is chosen about one time in five. This gives variety.

• Temperature reshapes the probabilities before sampling. The scores are divided by the temperature. A low temperature, such as 0.2, makes the favourite token dominate, so answers are focused and consistent. A high temperature, such as 1.5, flattens the probabilities, so unlikely tokens get a real chance and answers become more varied and sometimes strange. Temperature 0 means always the top token.
• Top-k keeps only the k most likely tokens and drops the rest. With top-k = 3 the model chooses among the three best candidates.
• Top-p, also called nucleus sampling, keeps the smallest group of top tokens whose probabilities add up to p. With top-p = 0.9 it keeps just enough tokens to cover 90 percent of the probability. The group is small when the model is sure and large when it is not.

After top-k or top-p removes some tokens, the remaining probabilities are scaled up to add to 1 again, and one token is sampled. For facts, extraction and code, use a low temperature. For brainstorming and creative writing, a medium or higher temperature helps. Not every model or service lets you set all of these, so read the documentation of the one you use.

Even at a low temperature, the same prompt may not give exactly the same answer every time, because of small differences in how the calculations run. If you need exact repeatability, test your prompt many times and check the results, rather than trusting one run.

Why the model sounds so sure

The model produces a fluent answer whether or not it knows the facts, because fluency is exactly what it was trained for. Its internal probabilities are not a reliable measure of truth. So treat confident wording as style, not as proof. You can reduce the risk by giving the model the source text to work from, asking it to say when it does not know, and checking claims that matter.

Your task: see how the next word is chosen

You will build a page with a toy model: a list of candidate next words with raw scores. Sliders for temperature, top-k and top-p show the resulting probabilities, and a column shows how often each word was chosen in 1,000 draws. A second card estimates tokens and shows how much of a context window your text uses.

What you will build

• A probabilities function that applies temperature and softmax to a list of scores.
• A filterProbs function that applies top-k and top-p and scales the rest back to 1.
• A sampleMany function that draws many samples with a seeded random generator and counts the results.
• An estimateTokens function that guesses the token count from words and from characters.
• A page with sliders, a table of probabilities and counts, and a text box with a context window check.

Steps

1. Create lib/sampling.ts and define a Candidate type with a token and a logit (raw score). Add a toy list of 8 candidates for the sentence “Thank you for your ...”.
2. Write probabilities(candidates, temperature): divide each score by the temperature (never below a small minimum, to avoid dividing by zero), then use softmax.
3. Write filterProbs(probs, topK, topP): sort the probabilities from high to low, keep tokens until you have topK of them or the total reaches topP, set the others to 0 and divide the rest by their sum.
4. Write sampleMany(probs, times, seed): for each draw, take a random number and subtract probabilities from a random number until it drops to zero or below to pick an index. Count how often each index is chosen.
5. Write estimateTokens(text): words × 1.3, and characters ÷ 4. Show both, since they are only guesses.
6. Build the page with the three sliders, bars, a column that shows the result of 1,000 draws, and a context window check with a size you can change.

Hints

• At temperature 0.1 the top word should have nearly all the probability. At 2 the bars should look almost flat.
• Top-k = 1 makes the model greedy. Compare it with top-k = 8.
• Lower top-p until only one or two words remain, and see how the others vanish.
• Check that the filtered probabilities always add up to 1.

You know it works when the bars change smoothly as you move the sliders, the 1,000 draws roughly match the probabilities, words removed by top-k or top-p never appear in the draws, and the token estimate rises as you paste more text.

Solution

The sampling logic, with temperature, top-k, top-p and the token estimate:

import { makeRng } from './rng';

// A language model does not pick "the" word. For every possible next token it produces a score (a logit).
// Software turns the scores into probabilities and then picks one at random according to them.

export type Candidate = { token: string; logit: number };

// Toy scores for the text "I am applying for the" (invented numbers, only to show the mechanics).
export const TOY: Candidate[] = [
  { token: ' job', logit: 3.2 },
  { token: ' position', logit: 2.9 },
  { token: ' role', logit: 2.4 },
  { token: ' post', logit: 1.6 },
  { token: ' scholarship', logit: 0.4 },
  { token: ' banana', logit: -1.5 },
  { token: ' moon', logit: -2.5 },
];

// Temperature divides the scores before softmax. Low = the favourite dominates. High = flatter, more surprising.
export function probabilities(cands: Candidate[], temperature: number): number[] {
  const t = Math.max(0.05, temperature);
  const scaled = cands.map((c) => c.logit / t);
  const max = Math.max(...scaled);
  const exps = scaled.map((s) => Math.exp(s - max));
  const sum = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / sum);
}

// Top-k keeps only the k most likely tokens. Top-p keeps the smallest group whose probabilities add up to p.
// Both then renormalise so the remaining probabilities add up to 1 again.
export function filterProbs(probs: number[], topK: number, topP: number): number[] {
  const order = probs.map((p, i) => ({ p, i })).sort((a, b) => b.p - a.p);
  const keep = new Set<number>();
  let cumulative = 0;
  for (const [rank, { p, i }] of order.entries()) {
    if (rank >= topK) break;
    keep.add(i);
    cumulative += p;
    if (cumulative >= topP) break;
  }
  const kept = probs.map((p, i) => (keep.has(i) ? p : 0));
  const sum = kept.reduce((a, b) => a + b, 0) || 1;
  return kept.map((p) => p / sum);
}

export function sampleMany(probs: number[], times: number, seed: number): number[] {
  const rng = makeRng(seed);
  const counts = probs.map(() => 0);
  for (let n = 0; n < times; n++) {
    let r = rng();
    for (let i = 0; i < probs.length; i++) {
      r -= probs[i]!;
      if (r <= 0 || i === probs.length - 1) { counts[i]!++; break; }
    }
  }
  return counts;
}

// A rough guess of the number of tokens. Real tokenizers differ by model and by language.
export function estimateTokens(text: string) {
  const words = text.trim() ? text.trim().split(/\s+/).length : 0;
  return { byWords: Math.round(words * 1.3), byChars: Math.round(text.length / 4) };
}

And the page:

import { Component, computed, signal } from '@angular/core';
import { FormsModule } from '@angular/forms';
import { TOY, estimateTokens, filterProbs, probabilities, sampleMany } from '../lib/sampling';

@Component({
  selector: 'app-sampling',
  imports: [FormsModule],
  template: `
    <h1>How a model picks the next word</h1>
    <p class="hint">Text so far: <strong>"I am applying for the"</strong>. The model scores every possible next token. Temperature, top-k and top-p decide how it chooses among them. (The scores are invented to show the mechanics.)</p>

    <div class="card">
      <div class="grid2">
        <div><label for="t">Temperature: {{ temperature() }}</label><input id="t" type="range" min="0.1" max="2" step="0.1" [ngModel]="temperature()" (ngModelChange)="temperature.set(+$event)" /></div>
        <div><label for="k">Top-k: {{ topK() }}</label><input id="k" type="range" min="1" max="7" step="1" [ngModel]="topK()" (ngModelChange)="topK.set(+$event)" /></div>
      </div>
      <label for="p">Top-p: {{ topP() }}</label><input id="p" type="range" min="0.1" max="1" step="0.05" [ngModel]="topP()" (ngModelChange)="topP.set(+$event)" />
      <table>
        <thead><tr><th>Next token</th><th>Probability</th><th>Chosen in 1000 tries</th></tr></thead>
        <tbody>
          @for (c of cands; track c.token) {
            <tr>
              <td>"{{ c.token }}"</td>
              <td><div class="bar"><span [style.width.%]="final()[$index]! * 100"></span></div>{{ (final()[$index]! * 100).toFixed(1) }}%</td>
              <td>{{ counts()[$index] }}</td>
            </tr>
          }
        </tbody>
      </table>
      <p class="muted">Low temperature (near 0.1) always picks the favourite: safe and repeatable. High temperature spreads the choice and lets odd words in. Top-k and top-p cut off the unlikely tail so nonsense like "banana" cannot appear.</p>
    </div>

    <div class="card">
      <h2>Tokens and the context window</h2>
      <label for="x">Paste some text</label>
      <textarea id="x" [ngModel]="text()" (ngModelChange)="text.set($event)" style="min-height:90px"></textarea>
      <label for="w">Context window size (tokens): {{ windowSize() }}</label>
      <input id="w" type="range" min="1000" max="200000" step="1000" [ngModel]="windowSize()" (ngModelChange)="windowSize.set(+$event)" />
      <p>About <strong>{{ tokens().byWords }}</strong> tokens (by words) or <strong>{{ tokens().byChars }}</strong> (by characters). These are rough guesses; each model has its own tokenizer.</p>
      <div class="bar"><span [style.width.%]="used()"></span></div>
      <p class="muted">{{ used().toFixed(2) }}% of the context window. The prompt, the conversation history and the model's own answer all share this space.</p>
    </div>
  `,
  styles: [`.bar { height:8px; background:#e2e8f0; border-radius:99px; margin:2px 0 4px; } .bar span { display:block; height:100%; background:#4f46e5; border-radius:99px; max-width:100%; }`],
})
export class SamplingPage {
  cands = TOY;
  temperature = signal(1);
  topK = signal(7);
  topP = signal(1);
  text = signal('We are looking for a customer support executive who can answer calls and solve billing problems in Zendesk.');
  windowSize = signal(8000);

  base = computed(() => probabilities(this.cands, this.temperature()));
  final = computed(() => filterProbs(this.base(), this.topK(), this.topP()));
  counts = computed(() => sampleMany(this.final(), 1000, 11));
  tokens = computed(() => estimateTokens(this.text()));
  used = computed(() => Math.min(100, (this.tokens().byWords / this.windowSize()) * 100));
}

What you learned

• A model reads tokens, about four characters or three quarters of a word each in English, and tokens decide cost and limits.
• The context window is everything the model can see at once. Chat memory is the app re-sending earlier messages.
• Training has three stages: pre-training, instruction tuning and preference training. The weights do not change while you chat.
• Temperature, top-k and top-p control how the next token is picked: low is steady, high is varied.

In the next lesson you will look at the architecture and the practical side of running models: memory, quantization, speed and cost.