Experto

AI's Misunderstood Progress: New Report Updates Key Evaluation Metrics

Original Source: MIT Tech Review
•
Read time: 2 min read
•Published: November 20, 2026
Share:
Source: MIT Tech Review

Executive Summary

A new report from AI evaluation nonprofit METR reveals continued exponential improvement in some AI capabilities, particularly in verifiable tasks like coding. However, the updated analysis indicates that models still lag in tasks requiring human judgment. The report updates METR's widely discussed "time horizon plot," offering deeper insights into AI's progress and limitations.

## AI's Misunderstood Progress: New Report Updates Key Evaluation Metrics A new report from METR, an AI evaluation nonprofit, sheds light on the nuanced trajectory of artificial intelligence capabilities, revealing both rapid advancements and persistent challenges. The study indicates that while some AI functionalities continue to improve exponentially, particularly in "verifiable" tasks, models still struggle with assignments demanding human judgment. The report serves as a significant update to METR's widely shared, yet often misinterpreted, "time horizon plot." This plot, known for its complexity, previously illustrated that AI systems were doubling their task completion capabilities every four months. METR's evaluation methodology involves having AI models attempt a broad array of tasks, each initially completed by humans. The difficulty of each task is quantified by the time it took human testers to finish, with models then rated based on the length and complexity of tasks they could successfully complete. A key distinction highlighted in the updated plot is between "verifiable" tasks and those requiring subjective assessment. Verifiable tasks are those where an AI system's success or failure can be automatically checked, such as coding or mathematical problem-solving. AI systems have shown remarkable proficiency in these areas, largely due to the effectiveness of reinforcement learning. This technique allows researchers to train neural networks by repeatedly attempting tasks, strengthening pathways that lead to success and weakening those that result in failure. This direct feedback loop explains why AI has become exceptionally good at domains like coding and math, where the correctness of a solution is unequivocally determinable. However, applying reinforcement learning to tasks where success is less clearly defined or takes longer to ascertain—such as choosing a new research direction or making complex strategic decisions—proves far more challenging. Consequently, evaluating AI performance on these more open-ended, judgment-intensive tasks is inherently difficult. The original time horizon plot from METR did not include such tasks, leading to a potentially skewed perception of AI's overall progress. The updated plot now incorporates a wider range of tasks, including some estimated to take humans significantly longer to complete and others that are more open-ended, aiming to provide a more comprehensive and accurate picture of AI's evolving capabilities.
Confirm and follow the full story at the original source:MIT Tech Review
Found this interesting? Share it with your network:
Share: