LLM Architecture and Optimization

In lesson 4 you met the transformer. In this lesson you will look at its parts in a little more detail, and then at a very practical question: what does it take to run a model? A model with billions of parameters needs memory, and it needs to read that memory for every token it writes. Understanding this explains why some models run on a phone and others need a data centre, why long chats cost more, and why people talk about quantization, caching and small models. At the end you will build a calculator that estimates whether a model fits on a device and how fast it can run.

The parts of a transformer

A modern language model is a stack of identical blocks. The number of blocks, often between 20 and 100 or more, is the model’s depth. Each block has two main parts.

• Self-attention. Every token looks at the other tokens before it and collects the information it needs, weighted by importance. Many attention “heads” run side by side, and each can focus on a different kind of relation: grammar, names, topic, position.
• A feed-forward network. A small neural network applied to each token on its own. Much of the model’s stored knowledge is thought to sit here.

Around those parts are a few helpers. The first step turns each token into a vector, the embedding, and adds information about its position, because attention alone does not know which word came first. Normalisation layers keep the numbers in a healthy range. Residual connections add each block’s input back to its output, which lets very deep networks train without losing the signal. At the top, a final layer turns the last vector into one score for every token in the vocabulary, and softmax produces the probabilities you saw in lesson 6.

What are parameters?

The parameters of a model are its weights: all the numbers adjusted during training. When you read “a 7 billion parameter model”, it means 7,000,000,000 numbers. More parameters usually give more capacity to store knowledge and skill, but they also need more memory and more calculation. Size is not everything: a smaller model trained on better data, for longer, can beat a bigger one.

Parameters are not the same as the context window or the amount of training text. Parameters are the model’s size. The context window is how much it can read at once. The training data is what it learned from.

How much memory does a model need?

Each parameter is a number that must be stored, and the space it takes depends on its precision. The common choices are:

• 32-bit floating point (fp32) takes 4 bytes per parameter. It is precise and was used for training in the past.
• 16-bit floating point (fp16 or bf16) takes 2 bytes. It is the usual choice for serving models.
• 8-bit integer (int8) takes 1 byte.
• 4-bit integer (int4) takes half a byte.

The memory for the weights is simply the number of parameters multiplied by the bytes per parameter. A 7 billion parameter model in 16-bit takes 7 billion × 2 bytes = 14 gigabytes. In 4-bit it takes about 3.5 gigabytes. You also need extra space for working data. A rough rule is to add about 20 percent, and more for long conversations.

Quantization: smaller numbers, smaller model

Quantization means storing the weights with fewer bits. It is like saving a photo at lower quality: the file is much smaller, and if you do it with care you hardly see the difference. Going from 16-bit to 8-bit halves the memory, and to 4-bit cuts it to a quarter. The model also runs faster, as we will see next. The cost is a small loss of quality, which grows as you use fewer bits. Many everyday tasks are fine at 8-bit or even 4-bit, but you should test your own task, especially for maths, code and non-English languages.

Why speed depends on memory

When a model writes text it produces one token at a time. For each token it needs to use all of its weights, so the chip must read the whole model from memory again and again. For this reason, the writing speed is usually limited by memory bandwidth, how many gigabytes per second the chip can read, and not by how fast it can multiply.

A rough upper limit is: tokens per second ≈ memory bandwidth ÷ size of the weights. With a model of 14 GB and a chip that reads 900 GB per second, the limit is about 64 tokens per second. Quantize the same model to 4-bit (3.5 GB) and the limit rises to about 257. These are only estimates. Real speed depends on the software, how many requests are handled together, and the hardware. But the estimate shows an important truth: smaller weights mean both less memory and more speed.

The cache: why long chats get expensive

When a model has already processed your earlier tokens, it does not want to redo the work for each new token. It keeps the intermediate results of attention for all earlier tokens, called the KV cache (keys and values). This saves a lot of calculation but uses memory that grows with the length of the conversation and with the number of users served at once. That is one reason why very long contexts are slow and costly, and why services may charge differently for cached and uncached input.

More ways to make models cheaper and faster

• Batching. Serve many users at the same time, so that one read of the weights helps many requests.
• Smaller models and distillation. Train a small “student” model to copy the behaviour of a big “teacher” model. You get much of the quality at a fraction of the cost.
• Mixture of experts (MoE). The model contains many specialist sub-networks, and for each token only a few are used. The model has many parameters in total but uses only part of them per token, which saves calculation.
• Efficient attention methods. Software tricks that compute attention faster and with less memory.
• Caching prompts and answers. If many requests start with the same long instructions, or ask the same question, reuse the earlier work or answer.

Choosing a model is a trade-off between quality, speed, cost and privacy. A small model on your own device is private and free per use but less capable. A large hosted model is more capable but costs money per token and sends your data to a provider. A good engineer picks the smallest model that does the job well, and tests that claim.

Your task: a model size calculator

Build a page that takes the size of a model and a precision, and tells you how much memory the weights need, which example devices can hold it, and an upper limit for the speed on each.

What you will build

• A table of bytes per parameter for fp32, fp16, int8 and int4.
• Functions for the memory of the weights, the memory with 20 percent working space, the speed limit, and the saving compared with 16-bit.
• A list of example hardware with memory and bandwidth numbers. Label them clearly as examples.
• A page with a slider for model size from 1 to 200 billion parameters, a precision selector and a table with a Fits column.

Steps

1. Create lib/memory.ts. Define the Precision type and a BYTES table: fp32 4, fp16 2, int8 1, int4 0.5.
2. Write weightsGB(paramsBillions, precision): parameters in billions times bytes gives gigabytes directly.
3. Write neededGB with the overhead factor 1.2.
4. Write maxTokensPerSecond(paramsBillions, precision, bandwidthGBs) as bandwidth divided by the weights size.
5. Write savingVsFp16 so the page can say how much memory a lower precision saves.
6. Define four example devices, such as a phone with 8 GB and a laptop with 16 GB, and build the page with a table that shows yes or no for each.

Hints

• Check your numbers by hand: 7 billion parameters in fp16 should show 14 GB, and about 16.8 GB with working space.
• Notice that a 7 billion model needs 4-bit precision to fit comfortably on the 8 GB phone example.
• Remember the speed limit is an upper bound. Real systems are usually slower.
• Do not claim the example hardware is a real product. The numbers are only round examples.

You know it works when moving the slider updates the table at once, lowering the precision makes more rows say yes and raises the speed limits, and your hand calculation matches the page.

Solution

The memory and speed calculations:

// Back-of-the-envelope maths for running a language model. These are rough estimates, not benchmarks.

export type Precision = 'fp32' | 'fp16' | 'int8' | 'int4';

// Bytes needed to store one number (one "weight") at each precision.
export const BYTES: Record<Precision, number> = { fp32: 4, fp16: 2, int8: 1, int4: 0.5 };

export const PRECISIONS: { id: Precision; label: string }[] = [
  { id: 'fp32', label: '32-bit float (full precision)' },
  { id: 'fp16', label: '16-bit float (common for serving)' },
  { id: 'int8', label: '8-bit integer (quantised)' },
  { id: 'int4', label: '4-bit integer (heavily quantised)' },
];

// Example round numbers for different kinds of hardware. They are NOT specifications of real products.
export const HARDWARE = [
  { id: 'phone', label: 'Phone (example)', memoryGB: 8, bandwidthGBs: 50 },
  { id: 'laptop', label: 'Laptop (example)', memoryGB: 16, bandwidthGBs: 100 },
  { id: 'gpu24', label: 'Gaming graphics card (example)', memoryGB: 24, bandwidthGBs: 900 },
  { id: 'gpu80', label: 'Data-centre accelerator (example)', memoryGB: 80, bandwidthGBs: 2000 },
];

// Memory for the weights alone: parameters (in billions) times bytes each. 7 billion x 2 bytes = 14 GB.
export const weightsGB = (paramsBillions: number, p: Precision) => paramsBillions * BYTES[p];

// Extra memory for the working space (activations and the conversation cache). 20% is a rough rule of thumb;
// long conversations need more.
export const OVERHEAD = 1.2;
export const neededGB = (paramsBillions: number, p: Precision) => weightsGB(paramsBillions, p) * OVERHEAD;

// When a model writes one token at a time, the chip must read (almost) every weight from memory for each token.
// So the speed limit is roughly: memory bandwidth divided by the size of the weights.
export const maxTokensPerSecond = (paramsBillions: number, p: Precision, bandwidthGBs: number) => bandwidthGBs / weightsGB(paramsBillions, p);

// Quantisation saves memory but can lose some quality. This reports how much memory it saves versus fp16.
export const savingVsFp16 = (p: Precision) => 1 - BYTES[p] / BYTES.fp16;

And the page:

import { Component, computed, signal } from '@angular/core';
import { FormsModule } from '@angular/forms';
import { HARDWARE, PRECISIONS, maxTokensPerSecond, neededGB, savingVsFp16, weightsGB, type Precision } from '../lib/memory';

@Component({
  selector: 'app-memory',
  imports: [FormsModule],
  template: `
    <h1>Will this model fit, and how fast will it run?</h1>
    <p class="hint">Rough estimates for planning. Real speed depends on the software, the batch size and the hardware. The hardware numbers are examples, not product specifications.</p>

    <div class="card">
      <label for="p">Model size: {{ params() }} billion parameters</label>
      <input id="p" type="range" min="1" max="200" step="1" [ngModel]="params()" (ngModelChange)="params.set(+$event)" />
      <label for="pr">Precision</label>
      <select id="pr" [ngModel]="precision()" (ngModelChange)="precision.set($event)">
        @for (p of precisions; track p.id) { <option [value]="p.id">{{ p.label }}</option> }
      </select>
    </div>

    <div class="card">
      <h2>Weights: {{ weights().toFixed(1) }} GB &nbsp; <span class="muted">(about {{ need().toFixed(1) }} GB with working space)</span></h2>
      <p class="muted">Quantising to this precision saves {{ (saving() * 100).toFixed(0) }}% of the memory compared with 16-bit, and may cost a little quality.</p>
      <table>
        <thead><tr><th>Hardware</th><th>Memory</th><th>Fits?</th><th>Speed limit</th></tr></thead>
        <tbody>
          @for (h of rows(); track h.id) {
            <tr>
              <td>{{ h.label }}</td>
              <td>{{ h.memoryGB }} GB</td>
              <td [class]="h.fits ? 'ok' : 'bad'">{{ h.fits ? 'yes' : 'no' }}</td>
              <td>{{ h.fits ? '~' + h.tps.toFixed(0) + ' tokens/second' : '-' }}</td>
            </tr>
          }
        </tbody>
      </table>
      <p class="muted">The speed limit is memory bandwidth divided by the size of the weights, because each new token needs the chip to read the model once. A smaller (quantised) model is both smaller and faster.</p>
    </div>
  `,
})
export class MemoryPage {
  precisions = PRECISIONS;
  params = signal(7);
  precision = signal<Precision>('fp16');

  weights = computed(() => weightsGB(this.params(), this.precision()));
  need = computed(() => neededGB(this.params(), this.precision()));
  saving = computed(() => savingVsFp16(this.precision()));
  rows = computed(() =>
    HARDWARE.map((h) => ({ ...h, fits: this.need() <= h.memoryGB, tps: maxTokensPerSecond(this.params(), this.precision(), h.bandwidthGBs) })),
  );
}

What you learned

• A transformer is a stack of blocks, each with attention and a feed-forward network.
• Parameters are the weights. Memory for the weights is parameters times bytes per parameter.
• Quantization stores weights with fewer bits, saving memory and increasing speed, at a small cost in quality.
• Text generation is usually limited by memory bandwidth, and the cache grows with the conversation.
• Batching, distillation, mixture of experts and caching reduce cost.

From the next lesson on we move from understanding models to using them well. First: how to write prompts that give reliable answers.