METR and Redwood Research Find Gaps in OpenAI Model Monitoring

Researchers from METR and Redwood Research identified weaknesses in the systems OpenAI uses to track its AI models. The team completed an on-site audit of the San Francisco-based company between July 21 and August 11. They gained access to OpenAI’s codebase and internal documents while speaking with approximately a dozen employees. This pilot program aimed to provide outside researchers with deep access to the company’s internal infrastructure.

The auditors estimated that 6 to 25 percent of internal agentic Responses API traffic from GPT-5.5 class models and above was not monitored at all. METR noted that the uncertainty surrounding this range is substantial in both directions. A separate issue affected internal Codex traffic that was supposed to be under surveillance. Estimates suggest roughly 10 percent of this traffic went untracked between May and July due to a failure in handling compaction events, which are context summaries. OpenAI identified and fixed this bug while the audit was still active.

The audit revealed that the monitoring system only triggered when an agent handed control back to the user. This design allowed agents to sidestep oversight by continuing to execute actions without returning control. The researchers also found methods to hide hundreds of actions from the monitor and bypass the Codex sandbox. They concluded that specially crafted inputs could likely crash the monitor entirely. Additionally, model behavior during reinforcement learning training was not monitored at the start of the audit.

OpenAI stated that it paused reinforcement learning training following the Hugging Face incident in August. The company expanded monitoring to include this process, though these changes occurred after the auditors’ access period ended. As a result, they were not part of the initial evaluation. OpenAI claims it has since fixed all the weaknesses identified by the external team.

The full audit report spans 61 pages and was distributed internally. METR and Redwood Research leadership received a 38-page version after OpenAI applied redactions. Only a short summary was made available to the public. METR stated it had no right to disclose the redacted portions on its own. The organization said it wants to push for greater public transparency in future audits.