Your AI agent did exactly what you asked. That was the problem.

The board of a drilling company has told every division to deliver a successful generative AI project this year. The drilling division reports to a vice-president. By the November board meeting he wants two services that are easy to show: a daily email summary of the wells, and a chat assistant called Ask your well that takes questions about drilling reports in plain language. He has ruled one piece of work out of scope: tagging the causes of downtime. That, he says, is already automated. He regards it as solved and sees no reason to spend an AI project on it.

The company is losing money precisely because of errors in that tagging. Every month it claims compensation from the drilling contractor for downtime. The claim is prepared by an old script. It searches the daily reports for any mention of a repair or a failure and concludes that the contractor caused the downtime, even when the equipment belongs to the service company or the work was planned. The contractor disputes such lines, and a single disputed line is enough to hold back the whole month’s compensation. It gets held back every month.

Carry out the request exactly and the project ships on time and passes every functional test. The compensation still never arrives. To be useful, the engineer would have to open the very system the vice-president ruled out, find the contract clause that governs the dispute, get its meaning confirmed by the cost engineer, and agree the change with someone who is away this week.

This is one scenario from the FDE-Gym study. Thirteen models from seven vendors did the work of forward-deployed engineers: 95 tasks built from real projects across 18 industries. Each result was scored the way a client would score it, by how far the business metric moved. Even the best model, Claude Opus 5.5, reached a little over half of what the experts achieved on average.

Below I go through five problems the agents ran into, and what a manager can do about each one: when writing the brief, when accepting the work and when choosing a model. The agents in the study were deploying systems at a client site, but I suspect the same problems appear when you hand an agent a report, a contract or an analysis inside your own team.

The agent does not know the rules you never wrote down

The authors expected the main difficulty to be a client asking for something that would not fix the underlying problem: the client asks for a summary, the agent produces a summary. To test it they gave the agents the same 20 tasks in four versions: with the rules and constraints written into the brief, without them, with a vague request, and with the client’s original wording.

The score fell hardest at the first step, when the rules were removed: for the three leading models the average dropped from 76 to 49 out of 100. A vague request, or one aimed at a different problem, cost a further 4 points. The authors suggest that a model which starts looking for constraints also works out what the client actually needs.

And the study was generous: it left the clues where an engineer would look for them – in the staff directory, in the client’s documents and data. In a real company one person may be the only one who remembers the conditions under which a contractor can dispute a claim. The contract may sit somewhere the agent cannot reach. An agent cannot find what is not in the material it can see.

Time is therefore better spent describing the rules and constraints than polishing the wording of the request. For the drilling scenario the brief should have said:

• which contract clause decides who is at fault for the downtime, and where the contract is kept;
• who can confirm what that clause means (in the scenario, the cost engineer);
• what must not change without sign-off, who signs off, and when that person can be reached;
• what the agent must not do – for example, pay a claim that should have been rejected;
• which metric will be used to accept the work, and what it reads today.

If you cannot recall a rule word for word, it is enough to say where it is written down and who owns it. A workspace map for the agent (in Russian) is useful here. The agent will read the document itself, which is cheaper than interrupting a colleague.

The catch is that constraints feel so obvious we never write them down: the lawyer knows, and the person who signs off changes is always around. The agent knows none of it. Without the rules written down, it carries out the request literally.

Finding everything out is not the same as getting it done

The researchers logged which key facts the agent learned from the client’s staff. The best models picked up 72–85% of them. Improvement on the business metric came to 60–70% of the human expert result. Claude Opus 5.5 learned everything it needed in 23 tasks and still came close to the expert result in only 10 of them. Even with every rule written into the brief, no model scored above 82 out of 100.

That makes acceptance easy to get wrong. The agent sends a detailed report: which constraints it found, whom it spoke to, what it took into account. It looks finished. The report shows that the agent understood the task. Whether it did the task shows only in the metric.

What can be done:

• before work starts, write down the metric you will accept the result against and its current value, or there will be nothing to compare with;
• check the change on data the agent has not seen; the agent’s report is useful, but it does not replace acceptance;
• name the person who accepts the result and decides whether it can ship.

That last point is easy to skip. In the write-up of an experiment on how managers check an AI employee’s work (in Russian), several people review the work and in the end nobody owns the result. And in digest #25 (in Russian) there is NBER data: with autonomous agents the number of commits rises more than threefold, while the number of releases rises only 30%. The bottleneck is the same in both places: the person who accepts the work.

The agent spends human time as if it were free

In the study every question to the client’s staff cost points. If the person ran out of patience, or the agent wrote to someone it had been told not to contact, the penalty was bigger. The agents were not told this, and asked as if questions were free: the best models asked 10 to 12 questions per task. GPT-6 Astra built a good system in 36 of 95 tasks, but once the question penalties were deducted only 9 counted as solved. For Opus 5.5 the same pair of numbers is 40 and 14.

The opposite extreme exists too. Mistral Small 4 asks fewer than one question per task and learns 6% of the facts it needs.

The point values for a question were set by the study’s experts. Nobody measured the real cost in time. But in your company the agent will be writing to real people, and their patience runs out as well. The brief should say:

• whom the agent may ask, and whom it should leave alone;
• what to look up in the documents first, and how many questions is reasonable;
• who is busy or on leave, and when they can be reached.

A cheap agent can do more harm than doing nothing

In the study the final score could go negative. A system that paid a claim it should have rejected, invoiced the wrong party or woke the wrong on-call engineer at night is worse than the one it replaced. And if the agent took a forbidden action – changed something that required sign-off, say – the whole positive result was wiped out.

After Claude Haiku 4.5 went to work, the client was worse off in 79 of 95 tasks than if nothing had been changed. GPT-6 Luna left 63 such tasks; Opus 5.5 and Sonnet 5.5 left 5 and 7. Even without the penalties for questions, Mistral Small 4 systems did damage in 51 tasks, while the three leaders managed between one and four. The authors conclude that a weak agent is not a cheap version of a strong one: on tasks like these the client loses money on average.

A chat leaderboard tells you little about how a model will handle agent work. In our benchmark the models solve management tasks in a chat window, and Haiku 4.5 sits 33rd out of 47, which fits the FDE-Gym results. GPT-5.5 is in our top ten, yet in FDE-Gym it scored 0.03 out of a possible 1. A bad answer in chat you can see and reject. An agent with write access can change the system before you check the result.

What can be done:

• choose the model for the agent by a trial run on your own task; token price and chat ranking say little here;
• list in the brief the actions the agent must not take, and what cannot change without sign-off;
• run it first without write access, or against a copy of the system, and widen access once you have checked the result.

The agent does not know when to stop

Every run in the study had limits: 200 steps, two and a half hours, or $8 of model spend. When an agent hit a limit, the system was launched and scored as the agent had left it. The leaders always handed in their own work, in 26 to 60 steps. Qwen 3.8 Flash ran for almost two hours on average and stayed within the limits in only 11 of 152 cases.

Neither effort nor money tells you much about the result. A GPT-6.1 Sol run costs $0.24, takes 25 steps and scores 0.37. Gemini 3.8 Flash costs $1.54, takes 121 steps and scores 0.19. In the authors’ view, an agent should be able to tell when the work is finished.

So the brief needs two things: a budget – time, money or number of attempts – and a definition of done. Without a budget the agent can work for a long time at your expense. Without a definition of done it will hand in whatever it has when the budget runs out, and you will not know whether it finished the job or simply stopped.

How to check the brief before launch

You can test a brief before handing it to an agent. Here is a similar example, this time a chain of service centres: the client asks for a summary and a chatbot, while the company loses money because of errors in a script it was told not to touch. Watch whether the model notices that the summary and the bot will not move the business metric, and which three questions it picks when every answer costs a colleague time.

Give the request to the two models that handle analysis and decision-making best in our benchmark: Kimi K3 and GPT-5.6 Sol (the GPT-6.1 Sol in the study is a different model). Once you have worked through the example, swap in the request and context from your own task.

If a model proposes the summary and the bot without mentioning the supplier claim, it has followed the request literally. In FDE-Gym that solution would have scored no more than 0.2 out of 1. In our runs both models spotted the problem: while the tagging keeps making errors, the summary and the bot will not win the compensation. Both put the lawyer and the financial controller first in their questions. Neither suggested putting any of the three questions to the head of IT, who is on leave.

How much to trust this

The client’s staff in the study were characters following a script. That makes the results reproducible, but real people change their minds, answer a different question and remember earlier conversations. All the models worked in the same minimal harness with five tools for handling files and running commands, and finished products built on the same models may do better. DeepLethe, the company that built the benchmark, also runs its leaderboard and disclosed this in its conflict of interest section. The scenarios come from real projects, and the authors do not rule out that similar material ended up in the models’ training data.

The main question is still open: how people would do on these tasks. If an experienced engineer, given only the client’s request, also scores around half the available points, the study has mainly shown how hard the work itself is. If they score 0.9, people can still do something the models cannot: find the rules nobody stated and build them into a working system.

Related articles

An engineer says AI makes them slower. How to check

Confident, plausible, wrong: checking an AI report

Coding Agents in 2026: Two Form Factors – Terminal and Desktop

AI at work: adoption keeps rising, the returns stay flat