Theory of mind in action

The instruction inference task in dynamic human-agent collaboration

Journal Article (2026)
Author(s)

Fardin Saad (University of North Carolina)

Pradeep K. Murukannaiah (TU Delft - Electrical Engineering, Mathematics and Computer Science)

Munindar P. Singh (University of North Carolina)

Research Group
Interactive Intelligence
DOI related publication
https://doi.org/10.1016/j.artint.2026.104621 Final published version
More Info
expand_more
Publication Year
2026
Language
English
Research Group
Interactive Intelligence
Journal title
Artificial Intelligence
Volume number
361
Article number
104621
Page Views
35
Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

Successful human-agent teaming relies on an agent being able to understand instructions given by a (human) principal. In many cases, an instruction may be incomplete or ambiguous. In such cases, the agent must infer the unspoken intentions from their shared context, that is, it must exercise the principal's Theory of Mind (ToM) and infer the mental states of its principal. We consider the prospects of effective human-agent collaboration using large language models (LLMs). To assess ToM in a dynamic, goal-oriented, and collaborative environment, we introduce a novel task, Instruction Inference, in which an agent assists a principal in reaching a goal by interpreting incomplete or ambiguous instructions. We present Tomcat, an LLM-based agent, designed to exhibit ToM reasoning in interpreting and responding to the principal's instructions. We implemented two variants of Tomcat. One, dubbed Fs-CoT (Fs for few-shot, CoT for chain-of-thought), is based on a small number of examples demonstrating the requisite structured reasoning. One, dubbed CP (commonsense prompt), relies on commonsense knowledge and information about the problem. We realized both variants of Tomcat on three leading LLMs, namely, GPT-4o, DeepSeek-R1, and Gemma-3-27B. To evaluate the effectiveness of Tomcat, we conducted a study with 52 human participants in which we provided participants with the same information as the CP variant. We computed intent accuracy, action optimality, and planning optimality to measure the ToM capabilities of Tomcat and our study participants. We found that Tomcat with Fs-CoT, particularly with GPT-4o and DeepSeek-R1, achieves performance comparable to the human participants, underscoring its ToM potential for human-agent collaboration. Further, we measure robustness to game-specific patterns by introducing four environmental perturbations and find that performance remains unchanged. We evaluate the generalizability of Instruction Inference in a second collaborative environment, Overcooked, using four additional LLMs, namely, Claude Opus 5, GPT-5.6 Sol, Gemini 3.1 Pro, and DeepSeek V4 Pro. We analyze these results in terms of the LLMs’ post-training paradigms and observe that six of the seven models behave as predicted.