How LLMs Work, Where They Fail, and How to Verify Them

I wrote down how an LLM produces an answer and where it goes wrong, along with what I missed on the check questions. This post covers how it behaves, where it fails, and how to actually verify its output. It does not cover the internals (transformers, attention).

1. An LLM predicts the next token

An LLM computes a probability distribution over the next token after the text it was given, picks one, and repeats. A token is a piece smaller than a word. With many tokenizers, Korean takes more tokens than English for the same content, which shows up in cost and length limits.

Training happens in two stages. The model first learns to predict the next token over a huge body of text (pretraining), and is then trained further to follow instructions and give the answers people prefer (instruction tuning, RLHF and similar).

What the model is trained to do is pick a plausible next word, not to state what is true. Most of the limits below come from this.

2. Temperature — why the same question gets different answers

The question was what is wrong with "I set temperature to 0, ran it once, got the right answer, so the prompt is verified." I picked two things: setting temperature to 0, and calling it verified after a single run — and the first one was the wrong answer.

Temperature controls how often lower-probability tokens get picked. Raise it and answers become more varied; lower it and the model sticks to the most likely tokens, giving consistent, conservative answers. Setting it to 0 to reduce variance is common and fine in itself.

The mistake is believing that 0 means the same answer every time. Anthropic's documentation states that even at temperature 0 the results are not fully deterministic, and identical inputs can produce different outputs across calls (Anthropic Glossary).

The bigger mistake is the other one. Running the same input many times measures variance; it does not verify a prompt. If one input came out right, the prompt is right for that one input. Verification needs different cases — ordinary inputs, edge values, odd inputs (section 5).

3. Hallucination — a detailed answer is not an accurate one

The setup was asking about the structure of the S6F11 message in an equipment communication protocol (SECS/GEM) and getting a very specific list of field names and descriptions. Asked what I would do before using it in code, I said I would check it against real data, and gave as the reason that "it explained so much that its memory might have gotten corrupted." The action was right; the reason was not.

Hallucination is confidently producing content that sounds right but is not true. It has nothing to do with how long the answer is. It shows up in particular places:

• Information that is rare or absent in the training data — industry-specific protocols, internal company facts, exact figures, paper citations, URLs, API names
• Questions built on a false premise — ask about "the third argument of function X" and the model tends to invent one even if X takes two
• Long, precise calculations

S6F11 is the first case. The narrower and more specific the fact, the higher the risk, and a detailed, confident tone is not evidence of accuracy.

There is one more thing to check in this question. The outer shape of S6F11 (Event Report Send) is a DATAID, a CEID, and a list of reports, each an RPTID with its values. But which variables go into each report is defined per equipment, with S2F33 (Define Report) and S2F35 (Link Event Report) (secsgem docs). A generic field layout dropped straight into code can be wrong for that specific tool.

So the two actions are: check the primary source (the SEMI standard or the equipment vendor's GEM manual), and compare against real log samples.

Ways to reduce hallucination:

• Supply the source material and say "if it is not in the material, say you don't know"
• Ask for sources, then open them yourself — sources can be invented too
• Have calculations run as code

4. The context window — why it forgets what you said earlier

At the end of a long conversation, a condition set at the start ("timestamps are in KST") had been ignored. I said the cause was the context getting too long, and that I would have a script check the timestamps. The cause was right; the fix was half of it.

The context window is the maximum number of tokens the model can see at once. Conversation history, pasted documents, system instructions and the model's own output all count against it. When it overflows, early parts get cut or summarized, and early constraints disappear.

A large window is not a guarantee either. Research has found that models do best when the relevant information sits at the beginning or end of the input, and much worse when it sits in the middle (Lost in the Middle, Liu et al. 2023).

A checking script is a good move, but it catches the problem after the fact. Prevention belongs next to it:

• Repeat important constraints with each request
• Put them in fixed instructions (a system prompt, a spec document)
• When a task runs long, summarize and move to a fresh session

5. So how do you verify — before and after on the same evaluation set

Asked how to tell whether a changed prompt is actually better, I could not answer. I only remembered that there was a method.

The method is comparing before and after on an evaluation set — the same idea as regression testing in software.

1. Fix the evaluation set first. Build 10 to 30 cases pairing an input with its expected result. Mix ordinary inputs, edge values, odd inputs, and inputs that failed before. Picking cases after seeing results means picking them to fit the conclusion, so build them before the change.
2. Fix the scoring. Where there is a clear right answer, score automatically. Where there is not, write a rubric and have a person or an LLM score it (LLM-as-judge) — LLM scoring wobbles too, so a person checks part of it.
3. Run the old and new versions on the same set. Change one thing at a time. Outputs vary, so run each case several times and look at the pass rate.
4. Look at regressions separately. Even if the total goes up, check whether cases that used to pass now fail.

The record left behind looks like this:

8 cases. v1 passed 5/8; the 3 failures were date-format problems.
v2, with a format example added: 8/8, previously passing cases kept.

"I changed it and it seems to work" is not verification.

For what to score, I vaguely remembered "four perspectives, something like consistency and accuracy." I never found the exact source.

The best-known named framework is the four dimensions used in summarization evaluation — coherence, consistency (does it agree with the source), fluency, and relevance (SummEval, Fabbri et al.). In practice people often group accuracy, consistency (similar results for the same input), relevance and safety, but that grouping does not seem to come from any single document. Being able to say what you compare matters more than remembering the names.

Wrapping up

• An LLM is a probability model trained to pick a plausible next word, so the same input can produce different answers.
• The rarer a fact is in the training data, the more likely it is to be made up — sources included.
• As the context grows, conditions given early on can get lost.
• When you change a prompt, compare before and after on an evaluation set you fixed in advance, and check regressions separately.