Co-authored by Yanquan Zhang and Michael Hogan.

As AI technologies become increasingly embedded in everyday work and learning environments, one fundamental shift is quietly taking place: AI is no longer just a passive tool but an active partner. In future classrooms and workplaces, AI systems may not only assist with tasks but also make decisions, suggest plans, and even negotiate how humans should behave. These developments highlight significant opportunities—but also deep challenges about the nature of human-AI teamwork.

If AI agents are to be considered teammates rather than tools, fundamental questions emerge about their interaction design. Should they always defer to humans? Should they take initiative? And critically, under what circumstances should they switch between these modes? These questions touch on core issues in human-AI teamwork design.

These are precisely the questions explored by Salikutluk and colleagues in their CHI 2024 paper, “An Evaluation of Situational Autonomy for Human-AI Collaboration in a Shared Workspace Setting.” The authors present a well-designed empirical study that investigates how AI agents manage their autonomy in dynamic, cooperative scenarios. Their key hypothesis is that situational autonomy adaptation, where the AI adjusts its level of initiative based on contextual factors, may be more effective than fixed levels of autonomy.

The Experimental Framework

To test this hypothesis, the researchers created a sophisticated cooperative simulation environment where human participants collaborate with an AI agent to process deliveries in a virtual office. The task—reminiscent of the popular game Overcooked—requires coordination between the human and the AI to sort documents, shelve books, and accept incoming deliveries. The researchers defined four levels of autonomy for the AI, ranging from “no autonomy” (agent does nothing unless commanded) to “high autonomy” (agent initiates actions independently), plus a “situational autonomy” condition in which the AI dynamically adjusted its behavior based on five criteria: self-confidence in decision-making, assessment of task priority and urgency, theory of mind modeling of human partner intentions, comparative competence evaluation, and requirements for human behavioral input.

Fifty participants were randomly assigned to one of these conditions and completed three trials of eight minutes each. The researchers collected both objective performance data (e.g., number of completed deliveries) and subjective ratings (e.g., perceived intelligence and cooperation of the AI teammate).

Key Empirical Findings

The results reveal several significant patterns that may challenge intuitive assumptions about optimal AI behavior. Most strikingly, the adaptive autonomy condition achieved the highest team performance scores compared to fixed autonomy levels. When it came to subjective evaluations, participants also rated the adaptive autonomy agent as significantly more intelligent than all fixed-level alternatives, including the highly autonomous agent. Participants appreciated the agent’s ability to take initiative when needed, such as when the human could not see that a delivery was arriving. They also valued its restraint in situations of uncertainty—preferring that it ask for confirmation rather than making potentially disruptive decisions on its own.

Paradoxically, participants in high-autonomy conditions, although experiencing some important reductions in their workload (i.e., in terms of command-giving and overall interaction), sometimes reported feeling that the AI imposed its own workflow preferences, highlighting tensions between work ‘efficiency’ and human agency that merit deeper investigation. This aligns with one of the paper’s most important points: more autonomy is not always better. Instead, success in human-AI teamwork seems to depend on context-sensitive initiative. The most effective AI is one that knows when to act—and when not to.

Critical Reflections

Despite its methodological strengths, the study raises several critical questions about the generalisability and implications of situational autonomy adaptation.

1. Can rule-based adaptation truly capture real-world complexity? Salikutluk and colleagues implemented autonomy switching using heuristic rules derived from pilot studies and literature analysis. But real-world work environments are messier, more ambiguous, and less predictable than any simulation. Will pre-coded rules designed for simulations generalize to real-world, high-stakes, dynamic and complex teamwork settings?

2. What happens when AI predictions about humans are wrong? Central to autonomy switching in Salikutluk and colleagues’ study was the AI’s “theory of mind”—its internal model of the human’s intentions and future actions. Cultivating a “theory of mind” and building a sound shared mental model is central to effective teamwork. In human-AI teams, errors in such models can lead to awkward or even harmful behavior. Salikutluk and colleagues noted cases where study participants were frustrated when the AI predicted the wrong label or asked them to take unnecessary actions. An AI agent may report confidence in its model and associated predictions, but this is not the same as accuracy, and designing systems that sustain a level of accuracy necessary for effective human-AI teamwork is a serious challenge.

3. Are humans truly empowered—or subtly overridden? In high-autonomy conditions, participants received fewer messages and gave fewer commands. On the surface, this might seem desirable. But some participants felt that the AI’s initiative disrupted their plans or imposed a workflow. In other words, autonomy can edge into coercion if not calibrated carefully.

Conclusion

In workplaces of the future, the expectation is that human-AI teamwork will become the norm, with humans and AI collaborating autonomously as interdependent agents. What will be needed are situationally aware, socially sensitive AI teammates that can reason not just about the task, but about their human partner. Technical competence is not enough. Building on the work of Salikutluk and colleagues, AI agents rated “most intelligent” in the future are likely to be those that can calibrate their initiative appropriately.

As AI systems continue to evolve, we will need more research like this—not only about what AI can do, but how it should do it in relation to human users. This involves hard design decisions about transparency, deference, communication styles, and control dynamics.

We are entering a new era of teaming—not just with people, but with machines. How we design human-AI teams and systems will influence fundamental dynamics related to trust, cohesion, productivity, and ethical collaboration. The work of Salikutluk et al. represents an important empirical contribution supporting design thinking—but also a reminder that adaptivity, like intelligence, is not a static property but a delicate social negotiation.

***

Connect with Yanquan Zhang and Michael Hogan