How Large Language Models Actually Work (Explained for Lawyers)

Some time before ChatGPT existed, I received a three-word text message from my son: “try this out”. Attached was a link to an obscure startup called OpenAI. In that moment I thought: this is not normal software. I have since told that story on a TEDx stage, and I still think it is the right place to begin an explanation of how LLMs work for lawyers, because the instinct was correct and the explanation was missing.

Here is the explanation. It needs no mathematics, fits in one sitting, and will change the way you prompt.

The image I use is a parrot in your living room. Imagine it has listened to every conversation ever held in the house and has an uncanny gift for producing the sentence most likely to come next. It does not know what a lease is; it knows what people say after the word “lease”. Most of the time the difference is invisible. Occasionally the next sentence is a Court of Appeal citation that does not exist, delivered in exactly the same tone of voice.

The model predicts the next token, nothing more

A large language model does one thing: given the text so far, it produces a probability for every possible next fragment of text, picks one, appends it, and repeats. That is the whole machine. ChatGPT, Claude, Gemini and Copilot are all this loop with different training and different wrapping.

Training teaches the loop by example: hide the next word, make the model guess, penalise a bad guess. In Sebastian Raschka’s walk-through, “assigning little probability to the observed continuation is therefore penalized strongly.” Repeat across trillions of words and you get a system that is extraordinarily good at plausible continuation. Andrej Karpathy’s summary is the one to remember: the base model is “just an expensive autocomplete” (that phrase and the framing in the next sections come from a widely shared digest of his deep-dive lecture).

The legal consequence follows immediately. When you ask for a case holding that a limitation period is tolled by bankruptcy, the model is not searching a database. It is producing the tokens that most plausibly follow your question, and those tokens will look like a case name, a reporter citation and a holding, because that is what follows such questions in the training data. Steven Schwartz, the Mata v. Avianca lawyer, put the misunderstanding in one sentence: “It just never occurred to me that it would be making up cases.” Once you see the mechanism, the surprise runs the other way: it is remarkable how often the autocomplete produces a real one.

Tokens: why the model miscounts letters and why German costs more

The “fragment of text” above has a name: a token. Models do not read letters or words. Text is split into sub-word pieces by byte-pair encoding, which starts from characters and repeatedly merges the most frequent pairs until the vocabulary reaches a target size. Common words survive whole; rare ones are chopped (“dehumidifier” becomes “de”, “hum”, “id”, “ifier”). GPT-4’s tokeniser has roughly 100,277 tokens.

Two quirks that matter in practice follow from this.

Thomson Reuters’ own prompting guidance lists “math, counting, and sorting” among the things LLMs do badly. That is tokenisation, not a bug the next release will fix.

Pretraining and post-training: a lossy zip file of internet knowledge

Training happens in two phases, and the split explains a great deal of what you see.

Pretraining feeds the model filtered internet text at enormous scale (the FineWeb dataset alone is “over 1.2 billion web pages”) and produces the base model: the expensive autocomplete. Post-training turns it into an assistant: humans write example conversations, the model is fine-tuned on them, then refined with reinforcement learning from human feedback, where people rate answers and the model is nudged towards the ones they preferred. “Pre-training takes months, but post-training is much faster. It can take as little as a few hours.”

What sits inside the model afterwards is, in Karpathy’s phrase, “a lossy zip file of internet knowledge”. Lossy is the operative word: facts that appeared often in the training data are stored well; facts that appeared once are stored badly or not at all.

For legal knowledge this has a measurable shape. Stanford’s Large Legal Fictions study (Dahl, Magesh, Suzgun and Ho, more than 800,000 queries) found that GPT-4 misidentified the court for 83.1% of district court cases but almost never for Supreme Court cases, and that hallucinations were “most common among the Supreme Court’s oldest and newest cases, and least common among its post-war Warren Court cases (1953-1969)”. Frequency in the training data is reliability. Your 2024 intermediate appellate decision from a mid-sized state appears in the zip file once, if at all.

The zip file was also sealed on a date. After its training cut-off the model knows nothing unless the context window tells it. Stanford’s follow-up study caught Lexis+ AI applying the Casey undue-burden standard after Dobbs had overruled it; even a retrieval tool can surface stale law when the retrieved text and the model’s memory disagree.

Parameters vs the context window: vague recollection vs working memory

This is the single most useful distinction in the whole subject, and it is Karpathy’s: “Model parameters store vague recollections” while “context tokens function as working memory”.

Anthropic’s documentation defines the second half precisely: “The ‘context window’ refers to all the text a language model can reference when generating a response, including the response itself. This is different from the large corpus of data the language model was trained on, and instead represents a ‘working memory’ for the model.” Everything you send counts against it: the system prompt, every earlier message, uploaded documents, tool results and the model’s own reasoning tokens.

The practical rule: put the source in working memory rather than relying on recollection. The benchmarks agree: in the Vals Legal AI Report, Harvey scored 94.8% on Document Q&A against 70.1% for the lawyer baseline, while on open-web EDGAR research the lawyers won. Document-grounded tasks are where AI beats lawyers. That is the whole argument for retrieval-augmented generation, which has its own guide.

Working memory has a catch. Anthropic’s docs now say “more context isn’t automatically better. As token count grows, accuracy and recall degrade, a phenomenon known as context rot.” Chroma’s context-rot study of 18 models found that even one distracting passage hurt, and that “models perform worse when the haystack preserves a logical flow of ideas”: shuffled text was retrieved better than coherent text. A million-token window is a capacity, not a recommendation; the context window guide covers how much to upload and in what order.

Below is a [services agreement] between [PARTY_A] and [PARTY_B], governed by [English law].
Task: identify every provision that allocates risk away from [PARTY_A].
Work in two steps. First, find the passages relevant to the task and reproduce them verbatim inside <quotes> tags, each with its clause number. Second, using only those quotes, list the risk-shifting provisions, one line each: clause number, the operative words, why it shifts risk.
If a provision you expect to find is absent, write "NOT IN DOCUMENT" rather than assuming it.
Do not rely on your general knowledge of what such agreements usually say.

<document>
[paste]
</document>

Why it bluffs: the exam-taker with no credit for “I don’t know”

Hallucination is not a glitch layered on top of a truthful system. It is built into both training phases, and two 2025 papers explain how.

The first is from OpenAI (Kalai, Nachum, Vempala and Zhang, September 2025; a journal version appeared in Nature in April 2026). Its argument is simple: “If 20% of birthday facts appear exactly once in the pretraining data, then one expects base models to hallucinate on at least 20% of birthday facts.” Rare, arbitrary facts cannot be compressed into the zip file, so the model guesses; DeepSeek-V3 gave three different confident wrong birthdays for co-author Adam Kalai.

Post-training makes it worse, because “language models are primarily evaluated using exams that penalize uncertainty”. A right answer scores one, a wrong answer zero, and “I don’t know” also zero, so “the majority of mainstream evaluations reward hallucinatory behavior.” Anyone who sat a multiple-choice bar exam recognises the incentive: you never leave a blank. Karpathy puts it from the practitioner’s side: “During post-training, models learn that they must always give an answer.”

The second paper is Anthropic’s interpretability work from March 2025, which traced circuits inside Claude 3.5 Haiku. “Refusal to answer is the default behavior.” A separate “known entity” feature fires when the model recognises a name and switches the refusal off. The researchers could “cause the model to hallucinate (quite consistently!) that Michael Batkin plays chess” by activating that feature for a person who does not exist. The natural failure is the same: something sounds familiar, the refusal circuit stands down, and the model fills in facts it does not have.

A plausible-sounding case name is Michael Batkin. So is a neutral citation in the right format: in Ayinde v Haringey one of the five non-existent cases carried a real citation, [2020] EWHC 2435 (Admin), that belongs to an unrelated business-rates case.

The OpenAI paper proposes a fix at the level of evaluation: tell the model the scoring rule (answer only if you are more than t confident, since mistakes are penalised t/(1−t) points and “I don’t know” scores zero). You can borrow it as a prompt. I have not seen its effect benchmarked on legal tasks, so treat it as a nudge, not a guarantee.

Answer the question below only where you are more than 90% confident. A wrong answer is penalised 9 points, a correct answer scores 1 point, and "I don't know" scores 0. For each point in your answer, state your confidence as High, Medium or Low, and say what you would need to see to raise it.
Do not cite any case, statute or regulation unless you are certain it exists and is current; where you are not certain, write [VERIFY] after it.
Question: [your question, with jurisdiction and date]

Temperature, sampling and why the same prompt gives different answers

Ask the same question twice and you get two answers. Lawyers find this unsettling; it is the sampling step described at the start. At each position the model draws from a probability distribution over the next token rather than always taking the most likely option.

Temperature is the knob that shapes the draw. Anthropic’s API reference describes it as “Amount of randomness injected into the response. Defaults to 1.0. Ranges from 0.0 to 1.0. Use temperature closer to 0.0 for analytical / multiple choice, and closer to 1.0 for creative and generative tasks.” Then comes the sentence to underline: “Note that even with temperature of 0.0, the results will not be fully deterministic.”

Two consequences. First, one run proves nothing. The Singapore Academy of Law’s prompting guide tells lawyers to “run a prompt repeatedly to verify results”, and Judge Newsom of the Eleventh Circuit did exactly that when he asked ChatGPT, Claude and Gemini ten times each whether a cashier held at gunpoint had been “physically restrained”, looking for the common core. Second, wording moves the distribution: Grimmelmann, Sobel and Stein showed that rephrasing “the ___ filing” as “the date of the ___ filing” flipped GPT-3.5’s ordinary-meaning answer from 2:1 one way to 5:1 the other. If your conclusion depends on which way the model leaned once, you do not have a conclusion.

Reasoning models, tool use and agents: what changed in 2025-2026

Three additions to the loop account for most of what has changed since 2024. None changes what the loop is.

Reasoning models (OpenAI’s o-series, and the reasoning modes built into later models such as GPT-5.1 and Claude Opus 5) are trained to spend tokens working through a problem before answering. DeepSeek’s public R1 research showed the recipe: with reinforcement learning, “as training progresses, the model is able to use more tokens to get better at reasoning.” Those thinking tokens sit in the context window and are billed as output. On multi-step tasks the gains are real; OpenAI’s guidance names “legal contracts, financial statements, and insurance claims” as documents these models suit.

But reasoning is not checking. OpenAI’s April 2025 system card found o3 hallucinated on 33% of PersonQA questions and o4-mini on 48%, against 16% for the older o1, because the newer models “make more claims overall”; Transluce even caught o3 claiming to have run code “on a 2021 MacBook Pro outside of ChatGPT”. The prompting advice flipped too: OpenAI’s guidance for its o-series says that “prompting them to ‘think step by step’ or ‘explain your reasoning’ is unnecessary”, and Anthropic says Claude Opus 5 “verifies its own work without being told to”, so legacy “double-check your answer” lines now cause over-verification.

Tool use is the model calling something outside itself: a web search, a calculator, a document store. Karpathy’s cure for hallucination is teaching the model that “if they don’t know something, they should look it up instead of making things up”. It works up to a point: GPT-4o with web search reached 90% on OpenAI’s SimpleQA benchmark. It does not work all the way. Stanford’s pre-registered test of retrieval-based legal research tools found Lexis+ AI and Ask Practical Law AI hallucinating on more than 17% of queries and Westlaw AI-Assisted Research on more than 34%, and 61% of Westlaw’s errors were reasoning errors despite the right documents having been retrieved. Retrieval puts text in working memory; it does not know that a Nebraska decision cannot reverse the Supreme Court.

Agents chain tool calls into multi-step work. Legora’s definition is the cleanest: “A chatbot receives a question and provides an answer. It’s assistive technology. It helps you work. An agent receives a task, creates a plan of action, executes the plan, and returns a result. It’s execution technology. It does the work, end-to-end.” Harvey’s governance line is the one to keep: “Agents do not sign documents. Lawyers do.” Agents have a separate guide.

Why the model’s explanation of itself is not evidence

When the output looks wrong, the instinct is to ask the model how it got there, or whether it is sure. The same Anthropic research shows why that misleads.

Asked to add 36 and 59, Claude computes the answer along two parallel internal paths: one approximates the magnitude, the other nails the last digit. Asked how it did it, the model “describes the standard algorithm involving carrying the 1”. That is not what happened inside; it is what a person would write. Given a hint about the expected answer, the model “works backwards, finding intermediate steps that would lead to that target”. The explanation comes from the same next-token machine as the answer, shaped by what you seem to want.

The verification protocol on this site is built on that principle: existence, quotation, holding, status, jurisdiction and documentation, each checked by a human in a primary database.

What this means for the way you should prompt and verify

Ethan Mollick’s description is the best short brief for a new user: “Treat AI like an infinitely patient new coworker who forgets everything you tell them each new conversation, one that comes highly recommended but whose actual abilities are not that clear.” Each clause maps onto a mechanism above, and each mechanism onto a habit.

One more habit follows from Stanford’s finding that models “often uncritically accept users’ incorrect legal assumptions”: check your own question before the model answers it. In the false-premise tests, GPT-4 hallucinated on 53.1-69.1% of questions whose premise was wrong by construction, such as why Justice Ginsburg dissented in Obergefell (she did not). The tendency to agree with you has its own guide; the prompt below is the cheapest defence.

Before answering, examine the question itself. Here it is: "[your question]".
Identify every factual or legal premise embedded in the question. For each premise, say whether it is (a) established in the materials I have given you, (b) a common but contestable assumption, or (c) something you cannot verify.
If any premise is wrong or doubtful, say so plainly and restate the question in a form you can answer. Only then answer, applying [jurisdiction] law as at [date], and tag every authority [VERIFY].

The frameworks for turning these habits into prompts (role, context, task, constraints, format) are covered in the prompt engineering guide, and the prompt library has 130 ready to paste. If you would rather build the habits with someone watching, module one of AI Lab for Lawyers is this page done live: browser tools only, no coding, and this mental model applied to your own documents before we touch prompting, confidentiality or agents.

Where to go next: the fundamentals hub collects the cluster this page anchors. The natural continuations are why AI makes up fake cases, which takes the bluffing mechanism through the sanctions record, and the AI glossary for lawyers, which defines the forty terms you will meet in every vendor meeting. If you want the model rehearsed rather than read, that is what the Lab is for.