Faster AI decisions still need a review path
The Decisions API enters beta. A new METR note shows why both automated choices and the tools used to review them need scrutiny.
OpenAI’s Decisions API entered public beta on October 6, according to its release notes. It gives developers a narrower way to use AI: answer a condition, choose from specified options, or assign a score, rather than compose a conversational response.
That could be useful for routing incoming requests or sorting a collection of documents. OpenAI claims substantially lower latency than its Responses API. Treat that as a company comparison, not an independent result for your application. The beta currently supports GPT-6 Luna and accepts text and images.
There is already a small practical experiment to inspect. Simon Willison released an early alpha plugin for his LLM command-line tool, including an image-classification example. It is an initial implementation, not evidence that a production workflow will be reliable. We have not tested it ourselves.
Leave room for uncertainty
A tidy output format does not eliminate a wrong decision. OpenAI’s guide recommends testing thresholds on labeled examples, accounting for the costs of false positives and false negatives, and providing an “other” option for inputs that do not fit the available choices. Its API reference also allows a refusal in place of an answer.
For a first project, our recommendation is a reversible sorting task. Keep the original input, inspect a sample of decisions, and send uncertain cases to a person. Decide in advance what happens when the model refuses or the request fails. Avoid treating a confidence value as a guarantee, or making consequential decisions about people solely from a model’s classification.
Review tools need protection too
A separate October 6 METR note explains how a vulnerability in an evaluation transcript viewer could conceal actions from a reviewer. METR says this was a proof of concept, not an exploit observed in its evaluations; the underlying records remained available, and the vulnerability was patched.
The lesson reaches beyond this new API. A review process depends on evidence a reviewer can trust, not just a polished screen. Keep records outside an agent’s control and test the systems that display them.
We welcome useful tools that reduce repetitive work. Faster classification can be one of them. But speed, structured answers and human oversight are separate properties: an application needs to establish each, and retain a workable route to challenge its decisions.