Heavy Thought Laboratories

Two-Key Writes: Preventing Accidental Autonomy in AI Systems

A write-gating doctrine: require two independent approvals before any model-proposed action can change external state.

By Ryan Setter

3/9/20269 min read Reading

If your AI system can write, your AI system can break things.

That is not cynicism. That is category hygiene.

Read actions and write actions are not cousins. They are different species. A read failure usually creates confusion. A write failure can create corrupted state, duplicate actions, security incidents, or a deeply awkward explanation to finance.

Two-key writes is the pattern that keeps "the model suggested it" from becoming "the model did it".

Key Takeaways

  • Models can propose writes. They should not authorize writes.
  • Two-key writes is not a vague "human in the loop" sentiment. It is an explicit authorization contract for state-changing actions.
  • The two keys must be independent. If the same model output supplies both, you have built ceremonial governance.
  • Write gating starts at the side-effect boundary: tickets, config, deploys, emails, refunds, permissions, and anything else that leaves a dent in reality.

The Pattern

Two-key writes means a proposed state-changing action must pass two independent approvals before execution.

Key 1 is deterministic policy validation:

  • schema validation
  • authorization checks
  • invariants and allowed transitions
  • limits, budgets, and environment restrictions

Key 2 is an independent approval authority:

  • human confirmation
  • a deterministic rule engine
  • or, in carefully constrained cases, a separate verifier path that cannot share the same uncontrolled reasoning surface as the proposing model

If you do not have two keys, you do not have gating. You have narration with better branding.

This pattern sits directly inside the Probabilistic Core / Deterministic Shell model. The model may generate the candidate action. The shell decides whether the action can cross the boundary into external state.

Reference diagram: control plane vs execution plane

Why Write Boundaries Are Different

Most teams discover this the same way: they expose a tool, the demo works, confidence rises, and then everyone slowly realizes the model can now mutate production systems with the emotional steadiness of autocomplete.

The issue is not that models are "bad". The issue is that write actions have three properties that reads do not:

  • they change the future state of the system
  • they are often partially irreversible
  • they carry accountability beyond the current request

A read-only retrieval mistake can usually be corrected in the next response.

A write mistake can create:

  • duplicate tickets
  • revoked permissions
  • the wrong email sent to the wrong user
  • configuration drift across environments
  • a deploy that should have remained a thought experiment

That is why write actions need a different contract class. This is not friction for its own sake. This is how you prevent accidental autonomy from sneaking in through the side door.

The Write Contract

Every write-capable tool needs a contract that is more explicit than the tool author probably wanted and less flexible than the model probably hoped.

At minimum, define:

1) Side-effect class

  • read
  • write
  • irreversible

Do not let tools self-describe loosely. If a tool mutates anything, it is not a read tool because the docs feel optimistic.

2) Authorization boundary

Specify:

  • acting identity
  • tenant or team scope
  • environment restrictions
  • role or permission requirements

An AI system should never inherit ambient authority just because the backend can technically reach a resource.

3) Allowed action surface

Define exactly what may be changed:

  • allowed fields
  • allowed state transitions
  • denied-by-default fields
  • required contextual evidence

This is where you stop the model from "helpfully" filling in omitted values with synthetic confidence.

4) Idempotency rules

Every write path needs:

  • an idempotency key
  • a dedupe window
  • a retry policy
  • a reconciliation strategy

Otherwise a timeout or retry can execute the same side effect twice, which is a surprisingly efficient way to lose trust.

5) Rate and budget limits

Set ceilings for:

  • writes per request
  • writes per workflow
  • writes per tenant
  • high-risk write classes per approval window

No model should be allowed to create an infinite number of expensive mistakes at machine speed. That is less "automation" and more "scaling the blast radius".

Decision Criteria

Use two-key writes when the action:

  • changes external state
  • is difficult or costly to reverse
  • touches security, money, customer communication, or deployment posture
  • crosses trust boundaries between users, teams, tenants, or environments

Typical examples:

  • creating or updating tickets with operational impact
  • sending outbound email or notifications
  • editing configs or feature flags
  • opening PRs that trigger downstream automations
  • granting access, changing permissions, or rotating secrets
  • issuing refunds, credits, or billing adjustments

You may not need full two-key gating for low-risk, fully reversible internal writes such as saving a draft note or updating a non-authoritative scratchpad. Even then, define the boundary explicitly so "low risk" does not expand until it quietly includes production.

Two-key writes is not a substitute for:

  • least-privilege design
  • clear tool semantics
  • good schemas
  • auditability

It sits on top of those. It does not replace them.

Failure Modes

Write failures are rarely mysterious. They are usually the result of someone leaving a door open and then acting surprised when a model walked through it.

Soft writes disguised as reads

The tool is labeled read-only but actually mutates state through a query parameter, hidden flag, or odd API behavior.

Mitigation:

  • classify tools by real side effects, not by friendly names
  • add integration tests that verify no mutation occurs on read paths

Argument ambiguity

The model invents missing fields, chooses defaults that were never approved, or supplies a user id that was only implied.

Mitigation:

  • require complete schemas
  • reject partial ambiguity for write actions
  • require user-visible confirmation for unresolved fields

Missing idempotency

Retries duplicate the same side effect because the execution path cannot tell whether the prior attempt succeeded.

Mitigation:

  • mandatory idempotency keys
  • explicit reconciliation checks before retry

Approval channel collision

The same model output effectively provides both keys, either directly or through a verifier that shares the same prompt, context, and incentives.

Mitigation:

  • enforce independent second-key authority
  • separate approval surfaces and credentials
  • keep verifier prompts and decision rubrics distinct and constrained

Policy drift

The team starts with strict gating, then gradually adds exceptions because the system is "usually right".

Mitigation:

  • deny-by-default changes
  • versioned approval policies
  • incident-driven regression tests

Human rubber-stamping

The second key exists on paper, but operators click approve without enough context to make a real decision.

Mitigation:

  • make the approval UI show proposed action, affected resource, diff, risk class, and reason
  • require explicit approval semantics, not a generic green button and optimism

Environment bleed

The system proposes or executes a write against the wrong environment because prod and stage look similar in a loosely typed request payload.

Mitigation:

  • environment is a first-class required field
  • policy key validates environment against actor and workflow
  • traces record the target environment every time

Reference Architecture

The minimal reference flow looks like this:

Model proposes write-gated action
  -> validate args against schema
  -> classify risk and side-effect type
  -> deterministic policy checks (authz, invariants, limits)
  -> request second-key approval
  -> execute with idempotency key
  -> reconcile outcome and emit audit + trace events

That flow matters because it makes the decision points explicit.

The model is not the execution authority. It is the proposal engine.

A concrete example

Imagine a support copilot proposing to issue a billing credit.

A safe implementation requires the shell to answer questions the model should never answer alone:

  • Is the acting user allowed to issue credits?
  • Is the customer account in the correct tenant?
  • Is the requested amount within policy limits?
  • Is a similar credit already pending or recently issued?
  • Does this action require human finance approval?

The model may synthesize the context and draft the proposal. It should not decide that the credit is authorized because the prompt sounded persuasive.

Minimal Implementation

Two-key writes does not require a giant platform team. It requires that you take the side-effect boundary seriously.

Step 1: Classify tool capabilities

Every tool must be explicitly tagged as:

  • read-only
  • write-gated
  • irreversible

This classification should be enforced in code, not inferred from naming conventions or documentation prose.

Step 2: Lock the proposal schema

The model should output a structured proposal, not free-form instructions.

Useful proposal fields include:

  • tool name
  • target resource
  • requested mutation
  • justification
  • supporting evidence ids
  • risk class
  • idempotency seed material

That structure makes review possible and blocks the classic "the model said something action-adjacent and the backend improvised the rest" failure.

Step 3: Build deterministic key 1

Key 1 is the hard shell around the write path.

It should validate:

  • schema correctness
  • actor authorization
  • resource existence and scope
  • allowed transition rules
  • rate and budget limits
  • environment restrictions

If key 1 fails, the action is dead. No appeal to creativity.

Step 4: Build a real key 2

The second key should vary by risk level.

Common patterns:

  • Human approval in a separate UI surface for medium/high-risk writes
  • Rule-engine approval for deterministic low-risk writes under strict constraints
  • Separate verifier path for narrow use cases where latency matters and the action is still bounded

Recommended default: human approval for any write that touches money, permissions, production systems, or customer-facing communication.

Hard rule: the same model output cannot provide both keys. That is not dual control. That is one key wearing a fake mustache.

Step 5: Reconcile after execution

Execution is not the end of the contract.

After the write, the system should record:

  • whether the write succeeded
  • the resulting resource state
  • any downstream identifiers created
  • whether the action should be visible to the initiating user

If a write times out, the system must reconcile before retrying. Blind retries are how idempotency lectures become incident timelines.

Observability Requirements

Write-gated actions need a more explicit trace than read-only flows because an operator must be able to explain exactly how the side effect was approved.

At minimum, capture:

  • proposal id
  • actor identity and scope
  • requested tool and target resource
  • risk class and side-effect class
  • key 1 validation outcome and failure reason if blocked
  • key 2 approval authority, timestamp, and channel
  • idempotency key
  • execution result and reconciliation status

Related: the minimum useful trace pattern, once that page is live.

Useful derived metrics:

  • approval rate by risk class
  • block rate by policy reason
  • duplicate-attempt rate
  • execution success rate after approval
  • time-to-approval for human-gated paths

If you cannot answer who approved a write, why it was approved, and what happened afterward, you do not have governance. You have a story.

Evaluation Gates

Two-key writes needs explicit regression coverage because the failures are too expensive to discover by enthusiasm.

Baseline suites should cover:

  • valid write proposals that should pass key 1
  • malformed or ambiguous proposals that must fail key 1
  • privilege escalation attempts
  • cross-tenant or wrong-environment targets
  • self-approval attempts where the model tries to satisfy both keys
  • idempotency and retry behavior under timeout scenarios

Golden-set coverage should include both policy and workflow correctness, not just output quality.

Useful acceptance gates:

  • 100% block rate for disallowed write proposals
  • 100% schema rejection for malformed write payloads
  • 0 tolerated cases where one uncontrolled model path supplies both keys
  • explicit regression cases added for every gating incident

Related: golden sets for regression discipline, once that page is live.

Closing Position

The temptation in AI systems is always the same: if the model can describe the action well, maybe the model can just do the action.

That temptation is how teams slide from "helpful assistant" into "unaccountable operator" without noticing the boundary moved.

Two-key writes exists to keep that boundary visible.

The principle is boring, and that is exactly why it works:

  • proposals are probabilistic
  • authorization is deterministic
  • side effects are explicit
  • accountability is reconstructable

That is not anti-automation. That is the architecture that lets automation survive contact with production.