A new evaluation framework published today on arXiv reveals that the benchmarks used to judge AI agents that operate smartphones have been measuring only the easiest part of the job — and that today’s most capable large language models fall apart when asked to do the rest.

The framework, called PhoneHarness, evaluates phone agents across three distinct action modalities that any real mobile workflow touches: graphical user interface interactions (tapping, swiping, and scrolling), shell commands executed directly on the device’s operating system, and programmatic API calls that bypass the screen entirely. Previous benchmarks — including AndroidWorld, AndroidLab, MAS-Bench, and AmbiBench — have evaluated agents primarily on GUI performance, leaving two of the three pillars of real phone automation unmeasured. PhoneHarness covers 14 task categories running on real Android environments, making it one of the most comprehensive mobile agent evaluations to date.

The timing is significant. Every major AI lab is now racing to ship phone-use agents commercially, and OpenAI is reportedly developing a smartphone designed to center the user experience entirely on AI agents. The commercial pitch from these labs has leaned heavily on benchmark performance. If those benchmarks were testing only the most visually intuitive third of what a real mobile agent needs to do, the field’s self-assessment has been wrong in a commercially consequential way.

GUI Scores Do Not Predict Shell or API Performance

The way current phone agents work is worth understanding precisely, because the gap PhoneHarness exposes is an architectural one. A GUI agent receives a screenshot of the device’s screen — or an XML accessibility tree representing the same UI elements — and uses an LLM or multimodal model to reason about what to do next. It then outputs a tap action tied to pixel coordinates or a specific element identifier, which the agent executes via the Android Debug Bridge (ADB) or an Android accessibility API. The model essentially learns to read a screen and point at things.

A shell command requires something completely different. Rather than pointing at a visual element, the model must generate syntactically correct command-line code that operates on the device’s underlying operating system: creating or moving files, querying system state, managing processes. Research on related tasks — including AI agents that fix Android build errors — has found that models often understand what needs to be done at a high level but fail to transfer that knowledge to effective low-level execution in shell environments. The problem is not reasoning; it is that shell syntax is a different output format from UI actions, and the training data for mobile shell interactions is sparse compared to the vast number of screenshot-action pairs that populate GUI training sets.

API calls present a third, distinct failure mode. A single API call can replace several GUI steps — booking a flight via an app’s API rather than tapping through the reservation interface step by step. But research published in Mobile-Bench found that LLMs produce unreliable results on API calls for mobile tasks, generating function calls that hallucinate parameter names, incorrect argument types, or nonexistent endpoints. The failure pattern is different from GUI grounding errors or shell-syntax gaps; it requires different training data and a different evaluation infrastructure to catch.

PhoneHarness provides that infrastructure. By running all three modalities inside a single evaluation harness against real Android environments, it produces the first apples-to-apples comparison of how much a model’s capability drops when the task shifts from screen-pointing to shell-commanding or API-calling. The baseline results, run across multiple leading LLMs, show a significant performance decline once CLI and tool-call actions enter the picture.

Why Lower Scores Mean the Benchmark Is Working

The mobile agent evaluation landscape has moved quickly over the past two years. AndroidWorld, published in 2024 and presented at ICLR 2025, became the dominant standard because it offered a reproducible emulator environment and deterministic evaluation. But agents have now reached over 90 percent success rates on its 116-task suite, effectively exhausting its ability to distinguish good models from great ones. MobileWorld, presented at ACL 2026 by researchers at Alibaba’s Tongyi lab, responded to that saturation by introducing cross-application workflows and Model Context Protocol-augmented tasks; the best-performing model on MobileWorld achieves only 51.7 percent success, compared to 90-plus percent on AndroidWorld. AndroidDaily, submitted to arXiv in May 2026, pushed further into real-world closed-source applications; the strongest model reaches 62 percent there.

PhoneHarness takes a structurally different approach from all of these. Rather than making GUI tasks harder — longer chains, more apps, more ambiguous instructions — it adds action modalities. The question it asks is not “can the agent navigate a more complex GUI?” but “can the agent use the phone the way a power user actually would?” That distinction matters because the commercial use cases that justify deploying a phone agent — scripting system behavior, integrating with app APIs for data retrieval, automating multi-step workflows that combine visual navigation with programmatic calls — routinely blend all three modalities. A developer automating a travel-booking workflow needs an agent that can tap through a booking interface, pull data via an API, and store results with a shell command. Evaluating only the first step of that workflow and declaring the agent ready is a measurement error.

This pattern has repeated itself across AI research. Text benchmarks gave way to instruction-following evaluations; coding benchmarks evolved from syntax correctness toward full repository-level tasks; math benchmarks moved from grade-school arithmetic toward PhD-level problems. PhoneHarness represents the same maturation arriving for mobile AI.

Why Scores Drop: Training Data, Not Just Test Design

The performance decline PhoneHarness reveals is not simply a matter of harder tasks. It reflects a structural mismatch between what frontier LLMs were trained on and what the new modalities demand.

The GUI grounding pipeline — screenshot in, tap coordinate out — has been the dominant paradigm for mobile agent research since the earliest demonstrations. This means that publicly available training datasets are heavily weighted toward screenshot-action pairs: human demonstrations of tap sequences, scrolls, and text entries captured on Android devices. The Android in the Wild dataset, one of the largest, contains 715,000 human demonstration episodes across hundreds of apps, almost all of them GUI-action pairs. Shell command traces and mobile API interaction logs for phones are far scarcer and almost never paired with a specific mobile task context.

A new benchmark can reveal that gap. Improving it requires fixing the data imbalance — collecting shell-interaction traces and mobile API call logs at scale, and building training sets that teach models to operate in all three modalities, not just the visually grounded one. Until that happens, a model that scores well on a GUI-only leaderboard has demonstrated exactly one of the three skills a deployable phone agent actually needs.

What PhoneHarness Changes for Developers

For developers building phone automation pipelines on top of today’s frontier models, the practical implication is direct: if a workflow ever requires a shell command or an API call — and most non-trivial mobile automations do — GUI benchmark scores predict nothing about how the system will actually perform on those steps. A model that scores 90 percent on AndroidWorld may fail on the first API call a real workflow requires.

PhoneHarness gives the research community a concrete target: match multi-modal GUI performance on shell and API modalities, not just on screen-based interaction. Closing that gap will likely require changes at both the model level — better tool-call grounding, more reliable shell syntax generation — and at the training-data level. The benchmark also provides baseline results publicly, so labs can measure progress over time against a consistent standard.

The broader benchmark trajectory in mobile AI is now clear. AndroidWorld established that agents can navigate GUI tasks reliably. MobileWorld showed that cross-application complexity and MCP tool usage expose large capability gaps. AndroidDaily revealed that real-world, closed-source apps present a harder challenge than open emulators. PhoneHarness adds the third dimension: action modality. Together, they describe a field that has been consistently overestimating how close it is to deploying agents that can actually handle a phone the way its users do.

Frequently Asked Questions

What does PhoneHarness test that other phone agent benchmarks do not?

PhoneHarness evaluates agents across three action modalities in a single harness: GUI interactions (tapping, swiping, and typing), CLI shell commands executed on the device’s operating system, and API calls that bypass the screen entirely. Prior benchmarks such as AndroidWorld and MobileWorld measure primarily GUI performance, which leaves two of the three major ways a phone can be automated untested and inflates apparent agent capability.

Why do AI models perform worse on shell commands and API calls than on GUI tasks?

The gap reflects a training-data mismatch. LLMs and multimodal models have been trained on large datasets of screenshot-action pairs demonstrating GUI interactions. Shell command traces and mobile API call logs for phones are far less common in training data, so models that understand conceptually what a shell command should accomplish frequently generate syntactically incorrect commands. API calls present a separate problem: models often hallucinate parameter names or incorrect argument types when generating function calls for mobile application interfaces.

What can AI agents actually do on a phone right now?

Current frontier models complete roughly 60 to 90 percent of GUI-only mobile tasks in controlled benchmark environments, depending on the benchmark’s difficulty. Cross-application workflows, tasks requiring MCP tool calls, and tasks on real-world closed-source apps push success rates substantially lower — the best models achieve 51 to 62 percent on harder evaluations. Shell command execution and API calling, tested in isolation, represent documented capability gaps that PhoneHarness now provides infrastructure to measure systematically.

What do developers need to know before deploying an AI phone agent?

GUI benchmark performance is not a reliable predictor of real-world phone agent capability if the deployment workflow involves shell commands or API calls. Developers should demand evaluation results that cover all three action modalities before trusting a model for non-trivial mobile automation. PhoneHarness provides an open framework and baseline results that enable exactly this kind of multi-modal assessment.