How Well Do AI Agents Hold Up in the Real World?

A new benchmark has emerged as a significant reference point in the evaluation of artificial intelligence (AI) agents. VAKRA, introduced via the Hugging Face Blog, is a tool-grounded executable benchmark designed to simulate enterprise environments, revealing that most leading AI models struggle severely with multi-step workflow completion. Built on more than 8,000 locally hosted APIs backed by real databases spanning 62 domains, VAKRA goes beyond simple knowledge testing to measure agents' practical tool-use capabilities and compositional reasoning skills.

Why Existing Benchmarks Fall Short

Conventional AI evaluation methods have largely focused on isolated skills—solving math problems, writing code, summarizing text. But real enterprise environments demand something entirely different. Agents must call multiple APIs in sequence, retrieve information from unstructured documents, and make decisions under natural-language constraints.

VAKRA was designed to bridge this gap. It presents tasks requiring agents to execute 3–7 step reasoning chains that combine structured API calls with unstructured data retrieval. A key differentiator is that it analyzes full execution traces, assessing not just whether an agent reached the right answer but how it got there.

Four Core Capabilities and Task Structure

VAKRA organizes evaluation tasks around four capability axes.

Capability 1: API Chaining Using Business Intelligence APIs

This includes 2,077 test instances across 54 domains, using tools from the SLOT-BIRD and SEL-BIRD collections. Agents must execute 1–12 sequential tool calls to arrive at the final answer. For example, answering "Which football team has a build-up play speed of 31, dribbling of 53, and passing of 32?" requires a precise 5-step API chain: data initialization → three filtering steps → team name extraction. The correct answer was FC Barcelona.

The SLOT-BIRD collection provides 7 generic data manipulation tools (filtering, sorting), while SEL-BIRD extends this with more specialized functions, including flattening categorical arguments into separate functions. Each instance is provided with an average of 4 get functions.

MCP Server and Efficient Data Transfer Architecture

The MCP (Model Context Protocol) servers supporting this task include a special tool: get_data(tool_universe_id=id). Agents must call this first to initialize the data source. The tool returns only a lightweight preview of the data, storing the full dataset server-side to avoid large data transfers over the MCP protocol—a deliberate design choice to prevent inefficiency.

Model Performance: Why Are Scores So Low?

According to industry reports, leading large language models (LLMs) are performing poorly on VAKRA overall. This represents a fundamentally different challenge from simple Q&A or code generation. Three major failure patterns have been identified.

First, initialization failure: agents attempt to use data manipulation tools without first calling get_data, or pass an incorrect tool_universe_id.

Second, error accumulation across steps: errors in early stages of a multi-step chain propagate forward, resulting in a wrong final answer. Agents frequently fail to verify whether intermediate results are anomalous.

Third, tool selection confusion: the coexistence of similarly named tools across SLOT-BIRD and SEL-BIRD leads agents to frequently select the wrong tool for a given context.

Historical Thread: The Evolution of AI Agent Evaluation

The paradigm for evaluating AI agents has shifted dramatically over the past several years.

• 2020–2022: Knowledge-based benchmarks like MMLU and HumanEval dominated. Single question, single answer.
• 2023: The rise of AutoGPT and LangChain sparked interest in tool-using agents. Early tool-use benchmarks like ToolBench and APIBench emerged.
• 2024: Multi-step agent benchmarks like AgentBench and τ-bench proliferated—but most remained confined to simulated environments.
• 2025–2026: Evaluation evolved toward executable benchmarks using real APIs and databases. VAKRA sits at the forefront of this wave.

This shift reflects a structural transition in how AI is measured—from "does it know?" to "can it do?"

Comparison with Competing Benchmarks

[Expert Analysis] Outlook and Implications

The structural vulnerabilities in agents exposed by VAKRA are unlikely to be resolved in the short term. Current LLM architectures are optimized to generate responses in a single forward pass, making them inherently weak at "reflective execution"—the ability to verify intermediate results and adjust strategy on the fly.

Several directions are likely to gain prominence. First, reinforcement learning (RL)-based agent training is expected to expand. Using tool-use success or failure as a reward signal is being actively explored by multiple research institutions. Second, executable benchmarks are likely to become the new standard. Evaluation approaches using real APIs, like VAKRA, are expected to be adopted by more companies and research organizations. Third, agent orchestration layers are expected to rise in importance. Architectures where multiple agents collaborate or a separate validation module checks intermediate steps could become mainstream.

The VAKRA leaderboard is currently open, and researchers and companies can submit their models for evaluation. The competition over AI agents' real-world capabilities has entered a new phase.