Thinking in probabilistic systems

Every software engineer trusts one axiom without thinking about it: call a function twice with the same input and you get the same output. Tests lean on it. assert add(2, 2) == 4 passes today, tomorrow, and on the CI box.

Now send this to an LLM API twice, with temperature set to 0, the most "deterministic" setting there is:

prompt = "Tell me about Richard Feynman"
# run 1 -> "...he grew up in Queens, New York..."
# run 2 -> "...he grew up in New York City..."

Same prompt. Same parameters. Same endpoint. Different answer. Researchers at Thinking Machines Lab ran exactly this 1,000 times at temperature 0 and got 80 distinct completions.^[1] No bug, no flaky network. That's just what the system does.

So here's the shift: an LLM call isn't a function call. It's a draw from a probability distribution. Once that sinks in, the rest of the discipline follows: why you can't assert exact strings, why there's no SLA on output correctness, why you design for failure up front.

Why temperature 0 still isn't deterministic#

Most engineers reach for the same wrong explanation: "GPUs are random." They're not. Run the same matrix multiply on the same GPU with the same inputs a thousand times and you get bitwise-identical results every time.^[1:1] The randomness lives somewhere less obvious.

Temperature 0 does one thing: it replaces random sampling with argmax. The model picks the single highest-scoring token instead of rolling dice. That removes the sampling randomness completely. But it does nothing to the numbers feeding argmax, the logits, and that's where the drift starts.

The cause is a property called batch invariance, or rather the lack of it. Your request doesn't run alone; the server batches it with whatever other requests arrive at the same moment, and that batch size changes constantly with traffic:

• At a small batch size, the math library splits the work one way across the GPU.
• At a large batch size, it splits the work a different way to stay busy.
• Different splits sum the same numbers in a different order.

Floating-point addition isn't associative: (a + b) + c doesn't always equal a + (b + c) down to the last bit. So the reduction order shifts a logit by a tiny amount. Usually that doesn't matter. But at one token position, two candidates sit almost tied, the tiny shift flips which one wins argmax, and every token after that diverges.

Batch size changes the order the GPU sums the logits; one near-tie flips, and everything after that token forks.

In the Feynman run, all 1,000 completions were identical for the first 102 tokens. The fork happened at token 103: "Queens, New York" in some runs, "New York City" in others.^[1:2] It isn't unique to GPUs, either; the same effect shows up on CPUs and TPUs.^[1:3] The deeper mechanics of sampling, temperature, and top-p come later in the book. For now the takeaway is enough.

There's a second, slower source of drift you can't see at all: the model behind the endpoint changes. Providers ship new weights under the same name. The gpt-4o you called in January may not be the gpt-4o you call in June, even though your code never changed. OpenAI exposes a system_fingerprint field as a canary; log it on every call, and when your outputs start drifting for no visible reason, that log tells you whether the model changed underneath you. The official guidance is blunt: determinism is "best effort", not a guarantee.^[2]

The five failure modes#

Non-determinism is the first surprise, but it's not the dangerous one. The dangerous failures produce a confident, well-formatted, completely wrong answer with no exception thrown. Treat these five as the normal weather of building with models, not rare edge cases. They overlap the OWASP LLM Top 10 for 2025 but run broader; OWASP is security-focused, and two of these are plain reliability problems.^[3]

The schema violation is the one you'll hit most. The hallucination is the one that erodes trust fastest, because a wrong answer looks exactly like a right one in the output string.

Graceful degradation is the default posture#

Because every call can fail in one of those five ways, you don't bolt error handling on at the end. You assume degraded output is normal and design the happy path around it. Each failure mode maps to a concrete control:

The pattern underneath most of these is validate, then retry, then fall back. Call the model. Check the output against the minimum bar. If it fails, retry a couple of times with backoff. If retries run out, drop to a cheaper or different model and flag the response as degraded so the caller knows to trust it less.

# needs an API key to run; illustrative wrapper
import time

def call_with_fallback(primary_fn, fallback_fn, validate_fn, max_retries=3):
    for attempt in range(max_retries):
        try:
            result = primary_fn()
            if validate_fn(result):
                return result, False  # (output, used_fallback)
        except Exception:
            pass
        time.sleep(1.5 ** attempt)  # exponential backoff

    result = fallback_fn()           # primary exhausted
    return result, True              # flag: came from the fallback path

That used_fallback flag matters. A degraded answer that announces itself is recoverable; a silent one is a future incident.

The same logic generalizes. Cache common requests, open a circuit breaker when the provider error rate climbs past about 20% in a rolling window, and escalate to a human for anything irreversible or money- and health-related.^[3:2] The detailed versions of these controls come in the API, eval, and observability parts. The posture is what you adopt now.

References#