AI Glossary

The share of a mixture-of-experts model actually used for one token. Active parameters set the speed and the memory bill; total parameters set what the model knows. Quoting only one of the two numbers hides half the product.

See also Mixture of experts

A model given tools and a goal, allowed to decide its own next step and to loop until it is done. The difference from a chatbot is not intelligence but authority: an agent takes actions with consequences outside the chat.

See also Tool use · ReAct loop · Human in the loop

The work of making a model behave as intended — helpful, honest, and unwilling to do the things its operator has ruled out. It is a training-time property that a determined prompt can still work around at run time.

See also Reinforcement learning from human feedback · Jailbreak · Guardrail

The mechanism that decides, for each token, which other tokens matter to it. Its cost grows roughly with the square of the context length, which is why long inputs get expensive faster than they get long.

See also Context window · KV cache

Producing one token at a time, each conditioned on everything written so far. It explains two things at once: why output speed is quoted per token, and why a model cannot go back and revise a sentence it has already emitted.

See also Token · Streaming · Time to first token

The raw pre-trained model, before it was taught to follow instructions. It continues text rather than answering questions; the “instruct” or “chat” variant of the same weights is the one you actually want in an application.

See also Instruction tuning

Serving several requests through the GPU together. It multiplies total throughput without making any single answer faster, which is why a hosted API can be cheap per token and still feel slow.

See also Tokens per second · Rate limit

A public test set used to rank models. It answers “which model is better in general”, which is rarely the question you have; a model can top a leaderboard and lose on your task.

From our own stackOn our own extraction work a 753-billion-parameter model and a 321-billion-parameter one agreed on all twelve fields that matter, and the smaller was five to seven times faster. Measure on your own task before paying for the bigger model.

See also Eval · Benchmark contamination · Ground truth

A public test set has leaked into a model's training data, so its score measures recall of the answers rather than skill. It is one of the reasons a leaderboard result rarely survives contact with your own data.

See also Benchmark · Eval

The volume at which owning hardware becomes cheaper than renting tokens. It exists only against a specific hosted price: against the cheapest open-weight endpoints the electricity alone can cost more than the tokens.

From our own stackOn our own numbers one mid-range card overtakes a mid-tier hosted API at roughly 200,000 output tokens a day, overtakes a frontier model almost immediately, and never overtakes the cheapest open-weight endpoints on price at all.

See also Self-hosting · Price per million tokens · GPU

The intermediate reasoning a model writes before answering. Some APIs return it in a separate field, some write it straight into the answer — and code that expects a clean answer breaks on the second kind without any error.

From our own stackTwo models we run handle the same “no thinking” flag in opposite directions: one ignores it and puts the reasoning in a separate field, leaving the answer clean; the other ignores it and writes the reasoning into the answer. Both are silent. The flag is worth a test, not an assumption.

See also Reasoning model · Structured output

Splitting documents into passages small enough to retrieve precisely and large enough to still make sense alone. It is the least glamorous part of a retrieval system and the part that most often decides whether it works.

See also Retrieval-augmented generation · Embedding

The first request after a model has been evicted from memory pays for loading tens of gigabytes from disk. Keeping a model resident costs VRAM continuously; letting it unload costs a slow first answer every time.

See also VRAM · Time to first token

A number the model reports about its own certainty. It is generated text like everything else, not a calibrated probability, and a wrong answer can carry a high one.

See also Hallucination · Eval

Deciding what goes into the limited context and what stays out — the discipline that replaced prompt wording as the main lever once models got good. Most quality problems in agent systems are context problems.

See also Context window · Prompt · Memory

How much text a model can hold in view at once, prompt and answer together. A large window is an upper bound, not a promise: the model sees exactly what your code sends, not what is in your files.

From our own stackOne of our extraction jobs returned an empty field on every document for weeks. The model had a million-token window; the prompt sent the first 12,000 characters, and the decisive sentence sat at character 224,283. When a field is persistently empty, check whether the text ever reached the model before rewriting the prompt.

See also Attention · Chunking · Context engineering

The usual measure of how close two embeddings are, from 1 (same direction) down to 0. The number is only meaningful relative to a corpus you have calibrated on — there is no universal threshold for “the same”.

See also Embedding · Semantic search

Where your prompts are processed and stored, and under whose law. For regulated or confidential material it decides the architecture before any quality comparison starts — a model that may not see the data has no score.

See also Self-hosting · Hosted API

The architecture behind most image and video generation. It starts from noise and removes it step by step toward the prompt, which is why generation time scales with the number of steps rather than the length of the output.

See also GPU · VRAM

Training a small model on the outputs of a large one. The result is far cheaper to run and keeps much of the behaviour on the tasks it was distilled for — and only on those.

See also Synthetic data · Quantization

Output quality changing over time without your code changing — a hosted model is updated, the input distribution shifts, a source site changes shape. Only a repeated eval catches it; nothing throws an error.

See also Eval · Fail-open / fail-closed

A list of numbers that positions a piece of text by meaning, so that similar meanings land close together. It is what makes semantic search, clustering and duplicate detection possible without keyword overlap.

From our own stackOur newsletter de-duplicates stories by embedding rather than by headline. A near-verbatim rewrite of an earlier story scored 0.982 cosine similarity — invisible to title matching, obvious to the vectors.

See also Semantic search · Vector database · Cosine similarity

A test set built from your own work, with your own definition of a good answer. It is the only instrument that tells you whether a prompt change, a model swap or a price cut actually helped.

See also Benchmark · Ground truth · LLM as a judge

What a component does when its input is missing or stale: carry on regardless (open) or stop (closed). Each is right somewhere, and choosing by accident is how a broken check turns into either silent bad output or a silent outage.

From our own stackThe policy file that can halt our publishing pipeline is deliberately fail-open: missing, unreadable or older than seven days all mean “carry on as before”. Only a fresh, explicit halt stops it — a monitoring outage must not silently stop the business.

See also Orchestration · Guardrail

Continuing training on your own examples to change how a model behaves — its format, tone or domain habits. It changes behaviour well and adds facts badly: for facts, retrieval is the cheaper and more current answer.

See also Low-rank adaptation · Retrieval-augmented generation · Instruction tuning

A large general-purpose model trained once at great expense and adapted afterwards for specific work. “Frontier” is the marketing term for whichever tier is currently largest and most capable.

See also Fine-tuning · Base model

The processor that makes inference practical, by doing thousands of multiplications at once. For serving a model the relevant figures are memory size and memory bandwidth far more often than raw compute.

See also VRAM · Self-hosting · Quantization

Cases with a known correct answer, labelled by a human. Building a few dozen of them is the cheapest quality investment in an AI project, and the one most often skipped.

From our own stackThe gate on our video pipeline is calibrated against a hand-labelled set of 70 clips. It is what turns “the judge seems better now” into a number, and what makes a threshold change reviewable.

See also Eval · Precision and recall

A check outside the model that constrains what it can cause: a schema it must satisfy, a spending limit, an allow-list of callable tools. Instructions in a prompt are a request; a guardrail is enforcement.

See also Prompt injection · Tool use · Human in the loop

A fluent, confident, false answer. It is not a bug to be patched but a property of predicting plausible text; the working countermeasures are retrieval, citations and verification, not sterner instructions.

See also Retrieval-augmented generation · Confidence score · Ground truth

Someone else runs the model and bills per token. No capital cost, instant access to the largest models, and three standing conditions: your data leaves the building, the model can change under you, and the price is theirs to set.

See also Self-hosting · Price per million tokens · Data residency

A required human decision at a named point in an otherwise automatic pipeline. Placed well, it costs one approval and removes the entire class of unrecoverable mistakes; placed everywhere, it removes the automation.

From our own stackOur video pipeline writes, renders, scores and captions on its own, and then stops: nothing reaches the public account until a person taps approve. That gate has never been automated away, deliberately.

See also Agent · Guardrail · Fail-open / fail-closed

Running a trained model to get an answer, as opposed to training one. Training is a one-off capital cost; inference is the bill that arrives every day, and it is the only one most companies ever pay.

See also Tokens per second · Price per million tokens

Output tokens usually cost several times what input tokens cost. A long document sent once is often cheaper than a chatty exchange about it, and a reasoning model bills its thinking at the output rate.

See also Price per million tokens · Reasoning model · Prompt caching

The training step that turns a text continuator into something that answers what it was asked. It is why a model responds to “summarise this” at all, and why the same weights behave very differently before and after.

See also Base model · Reinforcement learning from human feedback

A prompt that talks a model out of its own rules. It is a moving target rather than a solved problem, which is why enforcement that matters belongs in code around the model, not in the model's instructions.

See also Guardrail · Alignment · Red teaming

The date after which the model saw no training data. Anything later has to arrive in the prompt — through retrieval, a tool call or a web fetch — or the model will answer from an outdated world with full confidence.

See also Retrieval-augmented generation · Tool use · Hallucination

The intermediate state a model keeps for the tokens it has already read, so it does not recompute them for every new token. It is why the first token is slow and the rest are fast — and why long contexts eat VRAM even when the model is small.

See also Time to first token · VRAM · Prompt caching

A model trained on very large amounts of text to predict the next token. Everything it appears to do — answer, translate, summarise, write code — is that single operation applied over and over.

See also Token · Transformer · Parameter

Using one model to score another's output. It scales far past human review and carries the judge's own biases and instability, so it wants calibration against labelled cases before anything is gated on it.

From our own stackRe-running our quality judge over its own thirteen past rejections reproduced six of them: about 46 percent self-consistent. Any single verdict, and any “model A beats model B by three points” on that corpus, sits inside the noise.

See also Self-consistency · Eval · Vision-language model

Fine-tuning that trains a small add-on layer instead of the whole model. It fits on one consumer card, produces a file of megabytes rather than gigabytes, and can be swapped in and out at run time.

See also Fine-tuning

The ceiling on how much a model may write in one call. On a reasoning model the ceiling covers the thinking as well as the answer, so a budget that looks generous can be consumed entirely before the answer starts.

From our own stackA stricter scoring question pushed our quality judge straight through its 4,096-token ceiling: it returned an empty answer that surfaced as a parse error on twelve percent of clips. The finish reason, not the error message, is what identifies this.

See also Reasoning model · Structured output

Anything that survives between calls, because the model itself keeps nothing. In practice it is a store your code writes to and reads back into the prompt — which makes “what should be remembered” an engineering decision, not a model feature.

See also Prompt · Context engineering · Retrieval-augmented generation

An architecture that routes each token through a small slice of the model instead of all of it. A 321-billion-parameter model with 18 billion active runs at roughly the speed of an 18B model while carrying the knowledge of a much larger one.

See also Active parameters · Parameter

The degradation that follows when models are trained mostly on the output of earlier models. Rare cases disappear first, the distribution narrows, and the result is fluent, average and increasingly wrong at the edges.

See also Synthetic data

An open protocol for exposing tools and data to a model through one standard interface, so a capability written once is available to every client that speaks it — instead of being re-implemented per framework.

From our own stackOur own MCP server exposes web search, page fetch and a model call. The research and lead pipelines share it; adding a capability there adds it everywhere at once.

See also Tool use · Agent

Several models working on one task, each with a role, their outputs combined by a further step. It buys diversity of approach and pays for it in latency, cost and a much harder question: which one was wrong.

See also Orchestration · Agent

Open weights you may not use to make money — CC BY-NC and “research only” terms. The weights download exactly like any other, which is why this constraint is usually discovered late, in something already shipping.

From our own stackOne component of our own video pipeline carries CC BY-NC terms. It is tracked as a known constraint with a named replacement path, because a licence problem in a published pipeline is a product decision, not a legal footnote.

See also Permissive licence · Territorial licence restriction · Open weights

The same prompt can return different answers, even at temperature zero, because of batching and floating-point order on the serving side. Tests over model output therefore need tolerances, not equality assertions.

See also Temperature · Self-consistency

The trained parameters are published for download, so the model can run on your own hardware. This is not the same as open source: the training data and the training code usually stay closed, and the licence can still restrict what you may do with the output.

See also Permissive licence · Non-commercial licence · Self-hosting

The scheduling layer that decides which step runs where and when, retries what failed and keeps the state between steps. In any system that runs unattended it is the part that determines reliability — not the model.

See also Agent · Fail-open / fail-closed

One learned number inside a model; counts are quoted in billions (7B, 70B, 400B). The number indicates memory needed and, loosely, capability — it is a size, not a score.

See also Active parameters · Quantization

Apache 2.0 or MIT terms on model weights: commercial use, modification and redistribution allowed, with attribution. This is the clean case, and it is worth confirming per model rather than per vendor — families are often mixed.

See also Open weights · Non-commercial licence

The first and by far most expensive training phase, in which a model learns language from raw text. It happens once, costs millions, and is not something a company outside the model business ever repeats.

See also Fine-tuning · Base model

Precision is how much of what the system returned was right; recall is how much of what was right it returned. You trade one against the other, and which way you lean is a business decision, not a technical one.

From our own stackOur public knowledge graph is built for precision over recall: audited at 100, 98.3 and 96.6 percent precision on its three relation types. It would rather omit a link than draw a wrong one.

See also Eval · Ground truth

The standard unit of hosted inference pricing, quoted separately for input and output. Comparing two providers on it is only valid at the same context length and the same output length — and it says nothing about latency.

See also Input vs output tokens · Break-even · Self-hosting

Everything sent to the model for one call: instructions, examples, retrieved documents and the user question. It is the entire state the model has — there is no memory between calls that was not put back into the prompt.

See also System prompt · Context engineering · Memory

Reusing the computed state of an unchanged prompt prefix across calls. Where a long system prompt or document is sent repeatedly, it cuts both latency and the input bill — but only if the prefix is byte-identical each time.

See also KV cache · Input vs output tokens

An attack where instructions hidden in content the model reads — a web page, an email, a PDF — are followed as if they came from you. Any agent that both reads untrusted text and holds a capability worth abusing is exposed by construction.

See also Guardrail · Jailbreak · Tool use

Storing a model's weights at reduced precision — 4 bits or 3 instead of 16 — so a larger model fits in less memory. Quality falls off slowly at first and then sharply; the useful range is found by testing, not by reading the label.

From our own stackThe voice assistant in this house is a 12-billion-parameter model quantized to roughly three bits per weight. That is what lets it hold a single mid-range consumer card and answer in the room, with nothing leaving the building.

See also VRAM · Self-hosting · Parameter

The cap a provider puts on requests or tokens per minute. It is a capacity planning input, not an error condition: any pipeline that runs unattended needs to expect a 429 and back off rather than lose the work.

From our own stackOne hosted endpoint we measured allows 2,000 requests a minute and answers 429 above that, with a 30-second cooldown. Numbers like these are worth measuring rather than reading: the published figure and the enforced one differ.

See also Batching · Hosted API

The basic agent cycle: think, call a tool, read the result, decide again. Every production version needs a hard iteration cap and a fallback, because a loop that cannot end is the default failure mode.

See also Agent · Tool use

A model that writes a working-out pass before its answer, spending tokens to get a better result. The reasoning shares the output budget with the answer, so an unbounded thinking pass can consume the entire budget and return nothing.

From our own stackMeasured here: with no bound set, one model was still reasoning at 63,000 characters and returned an empty answer on four of five calls. Bounding the effort level fixed it; raising the token cap would not have.

See also Chain of thought · Max tokens

Deliberately attacking your own system to find what makes it misbehave, before someone else does it in public. For anything customer-facing it belongs in the release process, not in a one-off audit.

See also Jailbreak · Prompt injection

Training a model against human preference rankings rather than against a correct answer. It is what makes assistants helpful and polite, and also what makes them agreeable — models trained this way lean toward telling you what you appear to want.

See also Alignment

Looking up relevant documents first and putting them in the prompt, so the model answers from your material instead of from memory. It is the standard answer to “the model does not know our data”, and it is cheaper and more current than fine-tuning.

See also Chunking · Vector database · Semantic search · Fine-tuning

How often a model gives the same verdict on the same input twice. Measuring it first is what tells you how large a difference between two models has to be before it means anything.

See also LLM as a judge · Non-determinism

Running open-weight models on hardware you control. It converts a per-token bill into a fixed cost plus electricity, and it is the only arrangement in which no prompt and no document ever leaves your network.

From our own stackMeasured here: about 1.47 kWh per million output tokens, taken from whole-machine draw at the socket rather than from the card's own reading. The energy is the part most self-hosting calculations leave out.

See also Break-even · Open weights · GPU · Data residency

Searching by meaning rather than by shared words, so “how do I cancel” finds a page titled “ending your subscription”. It complements keyword search rather than replacing it: exact identifiers still need exact matching.

See also Embedding · Cosine similarity · Retrieval-augmented generation

Sending tokens to the client as they are produced instead of waiting for the finished answer. It does not make generation faster; it makes the wait visible, which is usually the difference that matters.

See also Time to first token · Autoregressive generation

Forcing the model to answer in a fixed shape — JSON against a schema — so a program can consume it. Anything that parses model output in production needs this plus a retry: free-text answers drift in ways that break parsers.

From our own stackOn two of six otherwise identical calls, one model dropped the opening brace of its JSON — the split between its reasoning field and its answer landed mid-token. A retry loop absorbs it; a parser without one would have lost the record.

See also Tool use · Chain of thought

Training data generated by a model rather than collected from the world. It solves scarcity and privacy problems, and it inherits every bias and blind spot of the model that produced it.

See also Distillation · Ground truth

The standing instruction placed above the conversation: role, rules, output format, what to refuse. It is the cheapest lever in the whole stack and usually the first thing to fix when output is wrong.

From our own stackWhen a bigger model beats a smaller one on a task with a hard constraint, check first whether the prompt ever stated the constraint. In one of our jobs a 753B model passed a validator eight times out of eight where a 321B model passed four; adding the missing rule to the prompt took the smaller model to eight as well.

See also Prompt · Structured output

The knob that controls how much randomness goes into picking each next token. Near zero the model repeats its most likely continuation; higher values buy variety at the cost of precision. Extraction wants it low, copywriting does not.

See also Top-p sampling · Non-determinism

A model licence that simply does not apply in your jurisdiction. Several strong open-weight releases exclude the EU, the UK or South Korea outright, which disqualifies them for a European operator regardless of quality.

From our own stackWe have twice had to drop a benchmark-leading model for this reason alone. Reading the territorial clause before the benchmark table saves the evaluation entirely.

See also Non-commercial licence · Open weights

How long the model takes to start answering, as distinct from how fast it writes once started. Users judge responsiveness almost entirely by this number, and it grows with the length of the prompt, not the answer.

See also Tokens per second · KV cache · Streaming

The unit a model reads and writes — roughly three quarters of an English word, and fewer characters than that for German compounds, code or JSON. Context limits, prices and speeds are all counted in tokens, never in characters.

From our own stackGerman text costs noticeably more tokens than the same text in English, because the tokenizers were fitted mostly to English. A bilingual site pays that difference on every page it sends to a model.

See also Tokenizer · Tokens per second · Price per million tokens

The component that cuts text into tokens and reassembles it. Each model family has its own, so the same paragraph is a different number of tokens — and a different price — depending on who you send it to.

See also Token

The rate at which a model emits output. Below about 10 tok/s reading feels like waiting; above roughly 30 it outruns a comfortable reading pace, and further speed only matters to machines.

From our own stackMeasured on our own machine on 2026-08-19: 38.7 tokens per second sustained from a 12B model on one RTX 3060 Ti, with under one percent spread across runs. A hosted frontier model swung by a factor of eleven on the same day.

See also Time to first token · Batching · Inference

Giving a model a set of functions it may call — search, fetch, query a database, place an order. The model does not run them: it emits a request, your code decides whether to honour it, and that boundary is where safety lives.

See also Agent · Model Context Protocol · Guardrail

A second randomness control: consider only the most likely tokens that together make up p of the probability mass. Changing it and temperature at the same time makes the effect of either impossible to attribute.

See also Temperature

The neural-network architecture behind almost every current language model. Its defining part is attention, which lets every token in the input influence how every other token is read.

See also Attention

A store that indexes embeddings and returns the nearest ones to a query vector. For a few thousand documents an ordinary database with a vector column is enough; the specialist product earns its place at a different scale.

See also Embedding · Semantic search · Retrieval-augmented generation

A model that accepts images as well as text. Vision is a discrete capability: a text-only model handed an image does not degrade gracefully, it refuses the request.

From our own stackThe quality gate on our video pipeline is a vision model that scores frames before anything is published. When we evaluated a larger model as a replacement, the deciding fact was not its score: it has no vision at all, which rules it out of that job whatever else it can do.

See also LLM as a judge · Diffusion model

Memory on the graphics card. It is the hard constraint in self-hosting: weights plus KV cache must fit, or the model spills into system memory and slows down by an order of magnitude.

See also Quantization · KV cache · GPU