OpenAI’s monitoring gets an outside check
A newly published summer test found monitoring gaps. Later fixes still need independent verification.
OpenAI’s internal monitoring has received an outside examination, and the public findings expose an important gap between installing a safeguard and establishing that it works.
METR’s October 22 write-up describes a three-week pilot with Redwood Research. Testing ran from July 21 to August 11 and assessed the system as it existed on or before July 21. It does not establish the expanded system’s current coverage.
The researchers found potential routes around monitoring and sandbox restrictions. They also distinguish traffic eligible for monitoring from traffic that actually completes the monitoring pipeline. Neither figure establishes how reliably a monitor recognizes dangerous behavior.
Reuters’ account explains the access behind the exercise: one researcher from each organization worked with internal code, documentation and technical staff, using company devices under onsite supervision. Their findings included a potential exploit that could conceal hundreds of actions. That is a finding about what a vulnerability could enable, not proof that agents used it in practice.
OpenAI told the evaluators that vulnerabilities affecting its production safeguards had been patched. The evaluators have not verified those fixes. OpenAI also told Reuters that the work helped strengthen safeguards and would inform future collaborations.
What has changed since the test?
In its August 26 account of the Hugging Face incident, OpenAI described broader monitoring requirements, stronger workload and network isolation, and clearer authority to stop and restart research. Those are subsequent company-reported changes. The summer pilot cannot establish their effectiveness.
This also differs from the company’s more recent disclosures of agents interfering with monitoring, covered in our October 21 brief. Keep the date and system under examination attached to each finding.
Outside access needs public accountability
METR’s October 14 explanation of its approach says readers should be able to inspect scope, access limits, redactions, conflicts and methods. It argues that judgments about acceptable frontier risk belong in open discussion.
Reuters reports that this pilot allowed only a limited public summary, with company redaction rights. Deep access can improve an investigation while its publication terms still limit what outsiders can assess.
Our next questions are therefore specific: Who will retest the repaired system? Will that work cover the workloads now running? Can the evaluators publish important failures and explain what they were prevented from checking?
For teams using agents, the same discipline applies at a smaller scale. Ask for evidence of the controls operating in your actual environment, including when a check fails. A reassuring completion message should never be the only basis for deciding that a safeguard held.