AI Can Only Automate 2.5% of Remote Work? Not So Fast

A new benchmark called the Remote Labor Index (RLI) made waves recently. Produced jointly by the Center for AI Safety (CAIS) and Scale AI, it tested six frontier AI agents on 240 real freelance projects sourced from platforms like Upwork. The headline finding: the best-performing agent, Manus, completed just 2.5% of projects at a level a client would accept. The rest scored even lower.

It is a bold claim. And it deserves a closer look.

What went well

Let’s start with what the RLI got right. This is one of the first benchmarks that uses real, paid freelance work as its test set. Not synthetic tasks, not academic puzzles, but actual projects that businesses paid for: 3D product renders, architectural plans, game development, video production, data dashboards. The total value of the work in the dataset exceeds $143,000. That grounds the evaluation in something tangible.

The evaluation process is also well done. Real people (not AI) reviewed and compared agent outputs against human deliverables. Different reviewers consistently agreed on their ratings, which adds credibility. The evaluation platform they built is open-sourced and supports dozens of file formats. That’s a solid contribution.

Now, About That 2.5%

Here’s where things get complicated.

The average human completion time for projects in the RLI was 28.9 hours, with a median of 11.5 hours. Some projects took over 100 hours. The agents were given a default session timeout of roughly one hour. That’s somewhere between 3% and 9% of the time humans had.

The agents also received a base prompt that literally said: “Do not ask any questions.” No iterative feedback. No “check your work and try again.” No task decomposition. No multi-step planning. Just: read the brief, produce the deliverable, zip it up, you’re done.

Think about how you work. Do you write a 30-page report in one pass without reviewing it? Do you deliver a client project without checking it against the brief? Of course not. You draft, you review, you adjust. You ask clarifying questions. You test your work. Yet the RLI expected AI agents to do what no professional would attempt: deliver complex, multi-hour projects in a single shot with zero feedback.

Andrew Ng highlighted this exact dynamic in his research on agentic workflows. He showed that GPT-3.5, a weaker model, scored 48.1% on a coding benchmark when run zero-shot but reached 95.1% when wrapped in an iterative agentic loop. A weaker model with good orchestration dramatically outperformed a stronger model running single-pass. The pattern holds across multiple studies: the Self-Refine framework (NeurIPS 2023) showed average improvements of roughly 20% across seven tasks. Reflexion (also NeurIPS 2023) boosted coding pass rates from 80% to 91%.

The RLI tested none of these patterns. And that matters.

The Wrapper Around the AI Is the Product

One of the most telling details in the paper is that Manus, which scored highest at 2.5%, runs on the same underlying Claude model available to other agents. It won not because of a smarter model but because of better engineering: a multi-layer planning pipeline, aggressive context management, integrated tools, and continuous self-monitoring. The Manus team reportedly rebuilt their agent framework four times to get the orchestration right.

This pattern shows up everywhere in the current agent landscape. On SWE-bench, GPT-4’s performance ranged from 2.7% to 28.3% depending on the tooling and orchestration around it, a 10x gap from engineering alone. Claude Opus 4.5 achieves 12% on one setup but only 2% on another. Same model, same weights, wildly different results.

What the RLI inadvertently demonstrated is that AI agent performance in 2025 is primarily an engineering and orchestration problem, not a model intelligence problem. The quality of prompts, tools, feedback loops, and task decomposition determines outcomes far more than which foundation model sits underneath.

Who’s Behind the Benchmark?

It is also worth noting the institutional context. CAIS is an AI safety nonprofit whose mission centers on highlighting AI risks. Scale AI is a $29 billion data-labeling company whose revenue depends on humans remaining essential to AI pipelines. Dan Hendrycks, the lead from CAIS, also advises Scale AI and co-authored a national security strategy paper with Scale’s CEO, Alexandr Wang.

None of this automatically invalidates the research. The methodology is documented and partially open-sourced. But when both organizations have business reasons to produce low automation scores, and the benchmark’s design choices systematically constrain agent performance, it is reasonable to ask whether different design choices might tell a different story.

For comparison, OpenAI’s GDPval benchmark tested individual tasks across 44 occupations and found GPT-5 approaching expert-level quality with a roughly 40% win rate against human professionals. The difference? GDPval evaluated discrete tasks with proper tooling. RLI evaluated entire end-to-end projects under severe constraints. Both findings can be true simultaneously: AI handles individual tasks well but struggles with complex workflows when given no support infrastructure.

What This Means for Teams Building with AI

The 2.5% figure is a valid floor measurement. It tells you what happens when you hand an AI agent a complex project with minimal tooling, no feedback, and almost no time. It does not tell you what’s possible with proper orchestration.

For engineering and product leaders evaluating AI adoption, there are a few practical takeaways from this research. The performance gap between naive and engineered AI deployment is enormous, and investing in agent orchestration, feedback loops, and tool integration is likely to matter more than waiting for the next model release. It might also be a good idea to treat the corrupted-files problem (17.6% of RLI outputs) as what it is: a tooling and infrastructure gap, not an intelligence limitation. And decomposing complex workflows into smaller, verifiable steps with human checkpoints is where real production value tends to emerge.

The RLI is a step in the right direction for AI benchmarking. We need more evaluations grounded in real economic work. But the 2.5% number reflects a very specific, very constrained test setup. Treating it as a general statement about what AI can or cannot do would be a mistake. The real question for teams building with AI right now is not “how smart is the model?” It’s “how well have we engineered the system around it?” That’s where the results actually change.

The Remote Labor Index paper is available at remotelabor.ai.

Note: the views expressed in this article are my own and do not represent the official positions of any past, present, or future employers, clients or stakeholders.