AGIBOT Releases Physical Reasoning Dataset to Advance Embodied AI

Image Courtesy: Public Domain

AGIBOT announced the open-source release of the “Physical Reasoning” dataset, the fourth release of AGIBOT WORLD 2026. For the first time in the AGIBOT WORLD 2026 initiative, the focus of task design shifts from manipulation difficulty to decision-making complexity, helping robots choose and execute actions as real-world conditions change.

AGIBOT WORLD 2026 is the first open-source dataset to cover the full range of embodied artificial intelligence (AI) research. It is organized around five research themes in embodied AI, each with dedicated data-collection methods and a fine-grained annotation system to support researchers in different subfields. Grounded in complete, real-world task requirements, Release 4 captures decision-making and execution processes for the same goal across different states and constraints. This approach is intended to help robots progress from isolated manipulations toward autonomously completing complex real-world tasks.

To respond, an agent first calls a tool to check the next day’s weather forecast. Based on the forecast, it determines which items are needed, and a robot then picks and places them. If sunny weather is forecast, the robot selects sunscreen; if rain is expected, it includes an umbrella.

Release 4 connects goal comprehension, task-state tracking, and action execution in real-world tasks. Covering over 10 categories of logical reasoning scenarios, the dataset deliberately includes multiple real-world states and execution branches for the same task. By providing high-level Q&A and low-level action trajectories, it supports coordinated training of high-level decision-making models and low-level action policies, linking task decision-making to physical robot execution. The aim is to help robots both “reason clearly” and “execute successfully.”

Real-World Tasks: The Right Action Depends on the Situation
Real-world tasks cannot be solved by simply repeating a fixed sequence of actions.

For example, to restock a hotel-room minibar to three full bottles, a robot must distinguish empty bottles from full ones and count the bottles already there. It must then decide how many bottles to remove and how many new ones to add. The required action sequence changes with the initial state.

Searching for toiletries across multiple drawers presents a different problem. Even when the camera view is unchanged, the next decision depends on which drawers have already been checked. Task-state assessment, dependencies between steps, and external conditions all matter in everyday activities such as housekeeping, cooking, and getting ready to go out.

For robots to complete these tasks autonomously, training data must capture more than “how to execute actions.” It must also capture “how to make decisions based on goals and conditions.” Release 4 is designed to address this need.

Same Task, Multiple States: Nine Categories of Reasoning Capabilities
Driven by real-world task goals, the dataset collects decision-making and execution processes for the same task under different states and constraints. It focuses on four key factors: object quantity, occlusion relationships, spatial occupancy, and task progress.

Consider retrieving a key from behind a box. If there is enough room to reach it, the robot can grasp the key directly. If the key is blocked, the robot must first move the box. The goal remains the same, but the action path changes with the scene.

The dataset addresses nine core reasoning abilities: attribute-constrained selection, context-conditioned selection, manipulation feasibility assessment, action precondition reasoning, goal-directed search, state and quantity reasoning, long-horizon planning, memory and state maintenance, and subtask ordering reasoning.

In practice, these capabilities often overlap. Minibar restocking, for example, requires product discrimination, quantity judgment, and state maintenance, while finding an item involves search strategy, search history, and spatial conditions.

Tasks are organized into three difficulty levels—Easy, Medium, and Hard. Easy tasks test individual judgments, such as attribute-constrained selection. Medium tasks introduce quantity requirements and action preconditions. Hard tasks combine multi-subgoal planning, long-horizon task-state tracking, and multiple contextual constraints. This structure allows researchers to isolate specific abilities and evaluate their compositional performance.

Execution outcomes are also annotated, including whether objects were correctly selected or placed and whether the required quantity was met. Successful and erroneous samples are given equal weight, providing material for research into how systems can adjust after deviations during execution.

One Data Pipeline Linking “Decision – Process – Action”
Supporting varied task situations requires annotations that connect the basis for each decision to the execution process. In Release 4, the high-level instruction describes the complete goal, while the task frame, subtask segments, and action steps capture the task structure. Together, these annotations help researchers map task progress to the current subgoal and study task breakdown, stage recognition, and next-subgoal prediction.

At the action level, the Dense Prompt (detailed action description) specifies the target object, manipulation region, bimanual role assignment, motion process, gripper state, and expected outcome. These descriptions are aligned with real-robot actions. Spatial annotations, including bounding-box, point, and trajectory-line annotations, further support visual-language grounding by linking language instructions to specific visual locations and motion paths.

Additional annotations, including “success segments” and “adversarial segments,” provide information for assessing task completion, distinguishing targets from distractors, and checking execution results. In combination, the annotations link the task goal, current stage, manipulation requirements, and execution outcomes in a single pipeline, connecting high-level decision-making research with action-policy training.

From Task Training to Real-World Applications
Researchers can use this release for joint or end-to-end training aimed at improving models’ situational understanding, process memory, and task reasoning. The dataset also supports decision–execution co-training of embodied agents: an agent selects subgoals based on the current situation and execution history; the action policy performs the manipulation; and the task state is updated so subsequent plans can be adjusted according to the results.

Grounded in real-world tasks, the dataset provides a training foundation for extending the capabilities of large models into the physical world. By connecting task decision-making, process memory, and real-robot execution, it is designed to help foundation models and embodied agents translate an understanding of user goals into physical actions.

In the longer term, this approach could help integrate robots into complete workflows in home services, hospitality, and commercial operations, where tasks require continuous judgment and execution. It could also lay the groundwork for the large-scale deployment of embodied AI in the real economy.

 

The “Physical Reasoning” dataset has already been used to support AGIBOT’s upcoming agent framework and to train its next-generation foundation model. Further progress will be announced in due course.