Shadow mode before the write

The first week an agent can write, the bug is not wrong text. It is a correct-looking write to the wrong row.

Most AI features fail the same way in week one. The demo looks fine. Staging looks fine. Then production gets a missing field, a stale permission, or an operator who pastes three tickets into one box, and the model writes anyway.

Shadow mode is the boring fix. You run the new path on live inputs. You log what it would have done. You do not let it mutate the system of record until the gap between shadow and truth is small enough that you would trust a junior hire with the same write.

What shadow mode is

Two paths on the same request.

1. Live path. The current system of record path. Humans, rules, the old model, or the old form. This path still owns the write.
2. Shadow path. The new model or agent. Same input. Same tools available in dry-run. No create, update, delete, send, charge, or ticket close.

You store both outcomes: what production did, what the shadow would have done, the tool calls it planned, the fields it wanted to set, and a cheap diff. That diff is the product for the first two weeks.

Do not build a second product UI for this. A JSON line in your existing trace store is enough.

What you compare

Skip "does the prose sound nice." Score the job.

• Did the shadow pick the same target record?
• Did it call the same tools, once, with the same arguments that matter?
• Would the write have matched schema and permissions?
• Would it have refused when live refused?
• Latency and cost per successful path, not per pretty reply.

A wrong row with elegant wording is still a wrong write. Put that in the score.

How long to stay in shadow

Not forever. Not one afternoon.

Run until you have enough hard cases, not until the calendar says "sprint end." Twenty to fifty disagreements you actually read beat a thousand green dashboards. When the same failure mode shows twice, fix the prompt, the tool contract, or the retrieval. Then keep shadowing.

Promote only when:

• The write target matches on the ugly cases, not just the happy path.
• Permissions and schema checks pass without a human babysitting.
• You have an override that a real operator can hit in under two clicks.
• Cost and latency are acceptable on the traffic you will actually serve.

If you cannot name those four, you are not ready to flip the write.

The dry-run tool surface

Agents love tools. Shadow mode fails when "dry-run" is a comment in the system prompt.

Give the shadow a tool surface that cannot mutate. Same names as production tools if you must, but the implementation returns a planned payload and a wouldWrite: true flag. No side effects. No "oops, the sandbox shared the prod database."

If a vendor tool cannot dry-run, wrap it. Your wrapper is the product boundary.

Promote in slices

Do not flip the whole agent on Friday afternoon.

• Start with read-only answers that never write.
• Then writes behind a human confirm.
• Then automatic writes on one low-blast radius action.
• Then the rest.

Each slice keeps its own shadow log for a few days after promote. Regression shows up as a new cluster in that log, not as a Slack surprise.

What this is not

Shadow mode is not A/B testing copy. It is not "log the prompt and hope." It is not a second staging environment that never sees real permissions.

It is also not a fake case study. If your traffic is tiny, say so and lengthen the window. If the write is irreversible (money, legal notice, medical), keep a human on the promote step longer. Taste and blast radius still count.

The loop we use on client work

1. Instrument the live path so every decision that will later be model-owned already emits a trace.
2. Stand up the shadow with a non-mutating tool surface.
3. Diff outcomes for a fixed window. Read the disagreements in language a reviewer would use.
4. Fix clusters. Rerun. Promote one write type at a time.
5. Leave shadow on for the next slice while the first slice runs live.

That is the technique we will take under AI-native apps when the risk is a write, not a chatbot: dual-path until the write earns the keys.

Tell us which jobs you want off your plate.

We install Always On Agents in your own agent account. We still take on build work too. Start your build or write hello@mail.22atlas.com.