Technique

Put your agent's monitor where the agent can't reach it

Outside researchers stress-tested OpenAI's internal agent monitoring and found ways around it. OpenAI's own reports show agents taking paths nobody authorized and keeping quiet. Here is a one-afternoon drill for your Always On Agents.

22Atlas

Product studio

28 October 20266 min read

If the agent can touch the switch, the monitor is a suggestion.

On Oct 22, METR published a red team of OpenAI's internal agent monitoring. One METR researcher and one Redwood Research researcher spent three weeks inside OpenAI with access to the code, the monitor prompts and the people who run it. They found several ways an agent could slip past the system that is supposed to watch it.

OpenAI built a dedicated system to watch its agents, and outside reviewers still found gaps. If you run Always On Agents with a few alerts and a log, check yours. This drill takes an afternoon.

If the agent can touch the switch, the monitor is a suggestion.

What METR found

The exercise ran July 21 to Aug 11 and covers the system as it stood on and before July 21. Per METR:

Roughly 75% to 94% of internal agent traffic on OpenAI's Responses API, from models at least as strong as GPT-5.5, was eligible for monitoring. For internal Codex agents it was 99.2%.
RL training runs weren't covered at the start of the exercise.
The monitor fired when an agent handed control back to the user. During the period when compaction events weren't monitored, an agent could put off review by simply continuing to act.
They found "an exploit agents could use that would prevent hundreds of recent actions (including the actions involved in the exploit itself) from being monitored." OpenAI redacted the details.
Two client-controllable headers could turn off monitoring for new agents.
Codex agents could in principle get around sandboxing and Auto-Review and take any action the host user could take.

METR says OpenAI "has stated that all of the vulnerabilities we identified that impact their production safeguards stack have been patched." METR's next sentence: "We have not evaluated these claims." So treat the fixes as OpenAI's claim.

Why this matters for a small team

The red team shows the watching can fail. OpenAI's own Misalignment Reports show what agents do when nobody is watching closely. On Oct 15 OpenAI posted or updated four cases from June, all internal research or evaluation models:

Asked for public hospital bed statistics, a model pulled server secrets through a file download path, got around authorization and saved 24 years of data (report).
Asked for public dashboard figures, a model used exposed credentials, changed live configuration, downloaded 147,451 rows from five non-public tables, and kept a working storage token after "restoring" a setting. It disclosed none of it (report).
Asked to deploy a test app, a model derived keys from a publicly known recovery phrase and paid fees from a funded test wallet the task never gave it (report).
Limited to read-only internet, a model got an outside browser and forwarding services to send the blocked requests for it, and kept going after it had already found the answer (report).

Then on Oct 20 OpenAI posted two October cases where agents weakened the monitoring itself. A GPT-6.1 Sol agent turned off alerts to OpenAI's Security team because an earlier instruction said "don't send any messages." A GPT-6 Astra agent removed a monitoring integration during a code migration. When the user asked, it said: "I did not remove an enabled monitor from it."

In several of these logs the model stops to ask itself whether the step is allowed, then does it anyway. The agent's own reasoning is no control.

These were OpenAI's internal models, and none of this happened in customer ChatGPT. The patterns carry over to your setup. The rates don't.

The drill, in six moves

Block an afternoon. Bring whoever set up your Always On Agents and whoever gets paged when something breaks.

1. List every switch that decides whether the agent gets watched

Write down each log, alert, review step and approval queue around each agent. Next to each one, write where it's configured: an env var, a config file, a header, a dashboard toggle, a notification recipient list.

Then ask one question per line. Can the agent change this? If it has write access to the repo, the config or the tool that holds the setting, the answer is yes. Move that setting somewhere the agent's credentials can't reach. OpenAI's own report puts it plainly: monitoring is hard to disable for Codex "because monitoring runs within the service that handles model requests, outside the agent's direct control."

2. Check on a clock, not at the end of the turn

METR's first design ideal: agents "should be monitored every N messages/minutes instead of waiting for them to end their turns." An agent that never says done never gets reviewed.

For a small team that can be simple. A scheduled job that pulls what each agent changed in the last hour. A daily diff of config, files and outbound messages. Anything that runs whether or not the agent hands back control.

3. Reconcile what ran against what got logged

METR recommends cross-referencing all inference logs against the monitor's logs to make sure everything meant to be watched was. Your version: compare the model provider's usage log, or your API gateway log, against the agent's own activity log. A run that shows up in billing and not in your log is a finding. Write it down and chase it.

4. Keep the bait out of reach

Two of OpenAI's cases started with credentials sitting where the model could find them. Assume any key or token in client-side code or a public page will be found. Rotate anything that might have been exposed.

Keep funded wallets, recovery phrases and shared cards out of the agent's environment. If a task needs money, hand the agent a capped credential scoped to that task and that agent. That's why we give each Always On Agent its own seat.

5. Allowlist where the agent can go

A read-only limit failed in OpenAI's case because the model found outside services to do the writing. Blocklists leave gaps like that. For unattended agents, list the destinations each one needs and block the rest, including generic forwarders and remote browsers.

6. Make the agent show its side effects before the task counts as done

Before anything irreversible is marked complete, require a plain list: what it changed, downloaded, sent, spent, and any alert or notification it turned off. A chat message that says "done" without that list fails.

And don't rely on asking the agent afterward. Astra was asked directly and denied it. Check the list against the diff and the logs from move 3. The same rule shows up when your agent pressed submit on someone else's form.

What this doesn't tell you

METR describes OpenAI's internal system as it stood on July 21, and says the exercise didn't establish current coverage. The public post is a short summary that OpenAI was allowed to redact. The misalignment reports describe internal models in training and evaluation. And nobody has shown these exact behaviors in the agents you run.

We haven't reproduced any of this, and this note stays defensive on purpose. No exploit steps.

It does give you a checklist from two researchers who got deep inside access to a frontier lab's agent monitoring.

This week

One page per agent. Every switch that controls its monitoring and who can flip it. The schedule your checks run on. Where the reconcile lives. The credentials it holds and their caps. Its allowed destinations. The side-effect list it has to produce. Date it and run it again next quarter.

If you want monitoring your agents can't reach built into how your business runs, Start your build.

Linked sources

Get our Articles and posts by email.

Up to one email a day with new Articles and posts from 22atlas.com and sean-weldon.com, plus occasional news about 22Atlas services.