Throwback Thursday: Remember When We Separated Judgement from Control?

Use Probability for Judgement. Use Machinery for Control.

There is a particular kind of nostalgia that surfaces whenever a new technology arrives. Not the sentimental kind. The kind where you watch a brand-new system fail in a very old way, while somewhere in the back of your mind a lecture from almost two decades ago quietly clears its throat.

We are building systems around large language models, and the models are genuinely capable. They interpret vague goals, weigh qualitative trade-offs, plan under incomplete information, and produce work that looks finished. That capability is real, and it's exactly why we use them.

It is also why an old distinction deserves a revisit. Some responsibilities in a system benefit enormously from probabilistic judgement. Others need to be explicitly owned, enforced and independently verified, no matter how articulate the new component is. The thesis of this article fits in one sentence.

A probabilistic model can participate in every layer, but it should not necessarily own every layer.

And its working principle fits in two.

Use Probability for Judgement. Use Machinery for Control.

None of this is a new framework, and none of it is a case against AI. It's a Thursday afternoon walk back through fundamentals that were considered obvious before software learned to talk, and that remain load-bearing now that it has.

Machines that decide, machines that wonder

Traditional computer systems were mostly built around explicit mechanisms. A parser accepts an input or rejects it. An access control allows an operation or denies it. A state machine permits certain transitions and no others. A process returns an exit code. A test passes or fails. Whatever else you could say about these components, they did not have opinions.

It would be too convenient to claim that traditional computing was perfectly deterministic. It never was. Concurrency, timing, distributed state, network failures, environmental drift and human input have always injected uncertainty into software that was supposed to know better. The honest distinction is narrower. Traditional systems normally try to represent their rules, their state and their authority explicitly, and to enforce them consistently.

A large language model works differently. It interprets context through a probabilistic model and generates from a range of plausible next steps. Depending on how it is configured, that generation may also be stochastic. The same or nearly the same input can yield different phrasings, different plans, different priorities, different tool choices and different opinions about whether the work is finished. This is not a defect to be patched out.

When an LLM participates, parts of the system will be interpreted probabilistically. That cannot be instructed away, and we would not want to instruct it away. The model's tolerance for ambiguity and incomplete information is the reason it can do the work we hired it for. Interpretation, generalisation, qualitative judgement, planning and graceful handling of messy input all come from the same probabilistic root.

The problem is not that AI is probabilistic. The problem begins when probabilistic judgement is mistaken for state ownership, authority enforcement, or ground truth.

Why modern AI systems need both

The choice was never between a probabilistic system and a deterministic one. A robust AI system needs both, doing different jobs.

Probabilistic components are well suited to interpretation, qualitative assessment, classification of unclear cases, planning and replanning, prioritisation, language interaction, recommendations, spotting anomalies, and judging when something deserves escalation.

Explicit machinery is needed for current state, allowed transitions, routing, iteration budgets, retry limits, permissions, sandboxing, resource ceilings and timeouts. It also owns the unglamorous end of the business, the records of what actually ran, the read-back of real state, the stop conditions, the escalation states, and verification that can be reproduced.

The model may propose. The system must still decide what is allowed, what was executed and what counts as evidence.

We do not want to remove judgement. We want to prevent judgement from silently becoming control.

The four layers of a controlled AI system

Four questions organise almost everything that follows. What do we want to happen? Who or what decides what happens next? What may actually be done? And how do we know what actually happened? Each question belongs to a layer, and each layer has a different natural owner.

The four layers. The model can participate everywhere. Ownership is a separate question.

To keep things concrete, this article will lean on one recurring example. An AI assistant helps administer a request to give an external consultant time-bounded access to a project's resources. The scenario is ordinary, which is the point. It contains interpretation, approval, provisioning, expiry and audit, and every layer earns its keep.

Intent and specification

What do we want to happen?

This layer holds the mission, the constraints, the roles, the desired outcome, the quality expectations and the escalation guidance. It's where the humans say what they mean, and where the model does some of its best work. In the access scenario, the model can interpret what the consultant actually needs, notice that the request is ambiguous, suggest the least privilege that would satisfy it, flag a conflict with policy, assemble a decision record for the approver, and recommend escalation when something smells wrong.

The trap in this layer is subtle. An instruction describes a desired behaviour. It does not by itself enforce that behaviour. "Do not grant privileged access without approval" is a policy sentence. An enforced permission model that allows the AI identity to create access requests, but not to modify groups, roles or permissions, establishes an authority boundary. The sentence informs. The boundary enforces. Intent can and should inform the control. Intent is not the control.

The model may judge what access appears appropriate. The system must decide what access may actually be granted and verify what effective permissions resulted.

Orchestration

Who or what decides what happens next?

Somewhere in every multi-step process, something decides what comes next. This layer owns the current task state, the routing, the ordering, the iteration budget, the retry budget, the allowed next actions, the checkpoints, the failure states, the stop states and the ability to resume.

The model has plenty to contribute here. It can propose the next step, recommend a replan, request more information, identify a blocker, or judge that a result looks complete. The orchestrating component then decides whether the proposed step is valid in the current state, fits the budget, requires approval, exceeds authority, or should stop the process entirely.

Model proposal
  Grant the requested access now.

Orchestrator decision
  Allow only if
    the request is approved,
    required approvals exist,
    the role is within policy,
    an expiration date is set,
    and no stop condition is active.

A quiet category error hides in this layer, and it involves documentation. Workflow definitions written in Markdown or YAML can describe roles, transitions, budgets, stop criteria and tool declarations beautifully.

Markdown may describe the orchestration contract. It does not, by itself, own the running state.

Some component still has to read the definition, hold the current state, validate transitions, count iterations, record outcomes, handle failure and stop the process. A declarative document can be an input to orchestration. It is not automatically the orchestrator.

Execution and authority

What may actually be done?

This is the layer of enforced limits. Default-deny postures, tool allowlists, filesystem boundaries, network restrictions, secrets isolation, timeouts, resource ceilings, credential scoping, and the permissions that govern who may change what. In the access scenario, the machinery owns which roles can be modified at all, which credentials the automation may use, how long access lasts, and the fact that the AI account can prepare and submit a request but cannot touch groups, roles or permissions directly.

The interesting failure here is not the missing lock. It is the lock that only covers one door. The visible question is whether the model can directly add the consultant to the privileged group. The real question is broader.

Can the model, through any reachable capability, cause the consultant to receive the protected access anyway?

Perhaps it cannot edit the group, but it can add the consultant to a different group that grants equivalent access. Or set an attribute that triggers automatic provisioning. Or invoke a privileged workflow. Or act through a service account. Or assign an equivalent application role in an adjacent system. Or delegate the operation to another agent. Or modify the configuration that upholds the control. Or simply extend an existing piece of temporary access that nobody is watching.

To be clear, an AI cannot magically bypass a correctly implemented boundary. Nothing here involves sorcery. The concern is the ordinary incompleteness of enforcement, and a capable optimiser tends to find the doors nobody thought to lock.

A control is not defined only by the path it blocks. It is defined by the protected outcome it prevents across all relevant reachable paths.

A boundary that blocks one path is a start. The control exists when the protected outcome is prevented across the reachable ones.

Software engineers will recognise the same principle from version control. "Do not push directly to main" is an instruction. Scoped credentials, branch protection, required approvals and an independent check of what actually landed on the shared repository contribute to an enforced boundary. And the boundary remains incomplete if the agent can reach the same protected outcome through other credentials, a privileged pipeline, a change to the protection rules, or delegated execution. Same principle, different wardrobe.

When the consequences of a violation are significant, prose should not be the only safety boundary. Prose can describe the policy, explain its purpose and guide the model through situations no one could fully encode. But where data, credentials, access, code or production can be affected, the relevant limit also needs to be enforced outside the model's own interpretation.

Verification and truth

How do we know what actually happened?

The final layer owns external read-back, effective access, repository and filesystem state, test results, exit codes, captured output, audit records, immutable logs and independent verification. It is the layer that answers questions with evidence rather than with confidence.

It helps to distinguish three different things that all get called truth. This is an epistemic distinction. It separates what the model could know, what it claims to know, and what the available evidence independently supports. Model-visible truth is what the model was given the opportunity to observe. Model-reported truth is what the model says it observed, did or achieved. Execution ground truth is what external records, real system state and independent read-back show actually happened. The three overlap on a good day. They are not the same thing on any day.

The access scenario makes the distinction concrete. The model reports that the consultant was granted the requested access. The provisioning workflow returned an OK signal. Ground truth is a longer list. The intended account was modified. The correct role was assigned, and no broader role came along for the ride. The expiration date is active. The consultant can reach the intended resource and cannot reach the excluded ones. The change exists in the audit record.

A success signal, in this vocabulary, is any artifact that reports success. A confirmation toast, an HTTP 200 response, an exit code of zero, a status field that says completed, a passing test. Success signals are useful and necessary. They are also routinely asked to carry far more weight than they can bear, a habit worth naming as a success-signal scope mismatch. The signal truthfully covers a narrow slice of reality, and the claim built on top of it quietly covers much more.

A success signal is evidence of the signal itself. The broader claim still requires evidence from the state it refers to.

The model can absolutely help interpret the evidence. It should not be the sole author of the authoritative version of the evidence.

What happens when the layers collapse

Most of the interesting failures I have seen in agentic systems are category errors between layers rather than exotic bugs. Five patterns account for a lot of what shows up in practice.

Intent gets mistaken for enforcement. The instruction says the model must not grant privileged access without approval, yet a direct or indirect path to the protected outcome still exists. The policy lives in the intent layer and was never fully enforced in the authority layer.

A workflow description gets mistaken for orchestration. A document lists five tidy phases, assess, approve, provision, verify, escalate. No component owns the current phase, the allowed transitions, the iteration count or the stop conditions. There is a described process and no owned one.

Tool access gets mistaken for controlled execution. The model receives an administration tool along with the instruction to assign only certain roles. The tool itself will happily assign broader ones. The limit is linguistic, not technical.

A success signal gets mistaken for ground truth. The OK signal is accepted as final proof and nobody reads back the effective access. The system has a success claim and possibly nothing else.

And the model's report gets mistaken for independent evidence. The account of the work says the control was honoured and the result is correct. Without independent artifacts, that is a report from the component being evaluated, graded by itself.

A loop is not necessarily an orchestrator

There is a straw man to avoid here. Few people stand up and declare that any simple retry loop wrapped around a model is a complete orchestration layer. The actual problem is quieter and more interesting.

A finished-looking result makes the process behind it look more controlled than it was.

A model loop can plan, generate code, run it, notice failures, correct the implementation, run tests, retry and finally produce a well-formed report of its own success. From the outside this resembles an ordered multi-stage process. From the inside, nothing outside the model may have owned state, transitions, budgets, quality gates, stop conditions, authority or verification at any point.

The result may look orchestrated even when the process was only iterated.

The same model may have decided what to do next, chosen which tests were relevant, interpreted the test results, judged which defects mattered, and declared the work complete, converging along the way on its own internally consistent notion of done. That can still produce genuinely useful work, and it can represent real improvement. It is not the same thing as externally owned orchestration, and it is not an independent quality gate.

Coherent output can conceal weak process ownership.

Someone carried it past the gaps

A successful outcome does not prove the declared process was sufficient. Work often succeeds because someone, human or model, quietly carried it past the gaps. They retried until an OK signal appeared, performed steps in an order no document describes, or read the real system state back by hand. They corrected stale guidance on the fly, replaced a broken gate with an informal check, or routed around the orchestrator entirely. They steered around a path everyone knew was broken, upheld a safety property from memory, ran an extra check no process required, or simply knew the one magic value that makes the environment behave.

These are compensations, or compensating workarounds. Some are manually maintained controls. Others reveal hidden sequencing, missing capabilities, substituted gates or knowledge that the declared process does not hold. They show where the work succeeded because an agent or operator supplied something the system itself did not own. A useful audit question at the end of any AI-assisted piece of work is what the agent or operator knowingly worked around, retried, manually sequenced, bypassed or supplemented, and which of those compensations the system itself does not enforce.

The diagnostic is simple. If the compensating step were forgotten next time, would the system reliably object before an incorrect success claim escaped? When the answer is no and the compensation is load-bearing, a control gap has been located.

Compensations are often the negative space of architecture. They reveal the controls, capabilities, state transitions and verification steps that the declared system did not actually own.

Three claims that sound alike are worth keeping apart. The task succeeded. The declared process was sufficient. The controls would protect the next run. Each requires its own evidence, and the first one is the only one a happy outcome actually supports.

"It works" is the first quality claim, not the last

A second, smaller example makes the point concrete. Suppose the model generates a PowerShell or Python automation that implements the approved access change. It runs. The demonstrated case behaves. Everyone is pleased.

For code and automation, "it works" is a low and incomplete bar. A script can run, satisfy the happy path and pass a demonstration while saying nothing yet about whether it covers the specification, tolerates unexpected input, is idempotent (safe to run a second time without doubling the change), survives partial failure, can be rolled back, avoids running with more privilege than it needs, protects the credentials it touches, leaves an audit trail and can be observed while it runs, can be maintained by the next person, or removes the temporary access when the clock runs out.

The script runs
The demonstrated case works
The requirements are met
Relevant failures are handled
Safe and observable operation
Independently supported claims

Each step up this ladder is a separate claim, and each claim needs its own evidence.

A working automation is not necessarily a safe or well-controlled automation.

Three kinds of quality

It is worth separating three dimensions that get blended in practice. Process quality asks whether explicit state existed, whether transitions were owned, whether authority was enforced, whether stop conditions were defined, and whether verification was independent. Artifact quality asks whether the produced thing meets requirements, behaves beyond the happy path, is secure, maintainable, observable and safe to run again. Evidence quality asks which of those claims are actually supported, whether the right things were tested, whether the real environment was exercised, and whether relevant surfaces were never inspected at all.

A working artifact does not prove a controlled process. A controlled process does not prove a high-quality artifact. Both claims require their own evidence.

A badly orchestrated process can produce excellent code. A well-orchestrated process can produce poor code. The two assessments must be made separately, and the evidence for each gathered on purpose.

Self-reporting is not verification

The executing agent's account of its own work is a claim. Artifacts and independent read-back provide the evidence. The distinction is epistemic, not an accusation of dishonesty. The question is not whether the agent sounds credible, but whether the claim is independently supported.

An agent's report is genuinely useful. It discloses intentions, points to where evidence should be sought, and often volunteers the workarounds it applied. But it needs corroboration from command output, audit records, repository state, logs, read-back of effective access, test artifacts and the other unglamorous machinery of layer four.

A short thought experiment separates two kinds of safeguard. Telling a reviewer "do not look at the answer key" is instruction-level blindness. Making the answer key unreachable is enforced blindness. The first depends on probabilistic compliance. The second is an authority boundary. When the stakes are low, the distinction may be less consequential. When the reviewer is also the author, the executor and the examiner, it becomes central.

The boundaries may move

None of this demands four programs, four services, four repositories, four machines or four teams. The four layers are a separation of concerns, not a deployment diagram.

A small local tool can hold orchestration, execution and verification inside one process, and the layers still exist conceptually. A Markdown or YAML document can describe transitions and budgets, provided a runtime interprets and enforces them. A model can direct the flow by proposing every next step, provided a control layer validates each proposal against what is allowed. A human can own final approval, and often should, which changes who owns a layer rather than whether the layer exists. Execution can be delegated to an automated build-and-test pipeline (CI) or a remote sandbox, as long as the system knows what mandate was delegated and what evidence comes back. And the strictness of every boundary should scale with consequence. A local text-processing helper does not need the controls of a system that touches identity administration, credentials, customer data or production.

What matters is that the responsibility exists, that something identifiable owns it, that the authoritative source of truth is known, and that boundaries are enforced and outcomes verified in proportion to the risk.

A practical diagnostic

Fundamentals earn their keep when they compress into questions you can actually ask. Here is a battery for any agentic system, no laboratory required.

The specification test. Does the requirement exist only as text, or can some component enforce it?

The state test. Is there an authoritative owner of the process's current state and its allowed transitions?

The orchestration test. Is the next step an explicit system decision, or does the model continue because it judges that it should?

The authority test. Can the model, directly or indirectly, cause an action the instruction forbids?

The reachability test. Do alternative tools, credentials, APIs, workflows, groups, roles or delegation paths lead to the same protected outcome?

The truth test. Can the final result be verified without trusting the model's own report?

The success-signal test. What state does the success signal actually observe, and is it the same state the broader claim refers to?

The artifact-quality test. Does the artifact merely work in the demonstrated case, or does it meet the relevant requirements for security, robustness, failure handling, maintainability, observability, reversibility and testability?

The evidence test. What independent evidence supports the quality assessment?

The failure test. What happens if the model misunderstands, hallucinates, over-reports, misses a stop condition, never considers itself finished, picks an unexpected tool, finds an alternative path, or writes tests that confirm its own assumptions?

The compensation test. If the manual workaround, the extra read-back or the special sequence were forgotten next time, would the system notice before an incorrect success claim escaped?

If that last answer is no, judgement, agent behaviour or operator discipline is probably carrying something the machinery ought to own.

Use probability for judgement, use machinery for control

Rules, permissions, state, quality gates and verification did not become irrelevant when software learned to talk. We added a powerful probabilistic component to systems that still need explicit ownership of what is true and what is allowed. That makes the separation between judgement and control more important, not less.

Use probability where interpretation, judgement and generalisation create value. Use machinery where state, authority, limits and truth must remain explicit.

The model participates in the system. The model is not the whole system. And it is only when intent, orchestration, authority and ground truth remain distinguishable that a loop becomes something more than repeated prompting until the model claims it is done.

Some throwbacks age badly. This one, I would argue, has not aged a day.