Zhejiang University and Differential Robotics

Where are the limits of GPT-6 Astra in general navigation and collaboration?

Enabling robots to navigate autonomously through complex, unknown environments as humans do has long been a major challenge in embodied intelligence. Human navigation depends on the interplay of multiple forms of intelligence: common sense and semantic understanding tell us that a nonstick pan is more likely to be found in a kitchen; logical reasoning helps us infer that a small, blurry red dot in the distance is a red flag; spatial perception and reasoning allow us to relate a two-dimensional map to the real three-dimensional environment and determine our position even without GPS; and long-, medium-, and short-term memory help us remember routes we have traveled, understand the progress of the current task, and use external tools to obtain information and support decisions. Beyond individual intelligence, humans also have strong capabilities for group collaboration: through communication, we break down complex tasks, divide work according to each member’s strengths, and share information, coordinate actions, and dynamically adjust plans during execution. Together, these capabilities—from individual autonomous decision-making to collective intelligent collaboration—constitute the central challenges facing robots on the path toward general intelligence.

In recent years, the rapid development of multimodal large models and agent technologies, particularly the emergence of GPT-6 Astra, has brought increasingly strong capabilities in visual understanding, logical reasoning, spatial cognition, and tool use. But when these capabilities are applied to robot navigation, how far can GPT-6 Astra actually go? How far is it from human-level general-purpose autonomous navigation? To investigate, we explored three representative and challenging navigation and collaboration tasks:

(1) Long-distance navigation in an open world with weak priors. When people are given a satellite image, a hand-drawn map, or even a simple topological map, can they reach a destination using only these imprecise maps, without GPS, prior mapping, or any external localization? This task focuses on whether an agent can connect an abstract two-dimensional map to a real three-dimensional environment and autonomously navigate hundreds of meters or even kilometers through a city without precise global localization.

(2) Long-horizon autonomous target search in a completely unknown environment. When people enter an unfamiliar building for the first time, they can use common sense, environmental cues, and spatial reasoning to find destinations such as kitchens or restrooms, even without any prior information. For robots, this requires not only understanding the current environment but also continuously exploring unknown spaces, maintaining historical memory, and dynamically adjusting search strategies. We aim to test whether a large model can truly achieve autonomous exploration and long-horizon navigation when given only a goal, with no specified route.

(3) Heterogeneous robot team collaboration on complex, open-ended tasks. How do the challenges grow when a single robot becomes a team of robots with different capabilities? We introduce four types of heterogeneous robots—a flying robot, an aerial manipulator, a mobile manipulator, and a humanoid robot—and ask them to jointly understand complex tasks, negotiate roles, share information, and work together in dynamic environments to complete open-ended tasks involving search, manipulation, repair, and delivery.

We observed several interesting phenomena in our experiments. In the first task category, GPT-6 Astra + Codex already demonstrated strong map understanding and spatial reasoning, connecting two-dimensional maps to three-dimensional environments and guiding long-distance navigation. In the second category, involving completely unknown environments, directly calling the GPT-6 Astra model API was prone to issues such as repeated exploration and unstable decisions due to insufficient historical memory and difficulties in context management. In the third category, involving multi-robot collaboration, the system also had to handle information synchronization, task decomposition, capability allocation, and changes in a dynamic environment; relying solely on direct decisions from GPT-6 Astra was often insufficient for stable, efficient team collaboration.

This raises a deeper question: is the bottleneck in general-purpose robot intelligence the capability of the foundation model itself, or the way we build a reliable agent system—an Agent Harness—around it? Furthermore, can we use the model’s own reasoning and programming abilities to let agents continually identify problems, optimize their architecture, and achieve Recursive Self-Improvement (RSI)?

1. Long-distance navigation with weak priors

CityWalk Like a Human

When people arrive at a zoo and receive a map, they can work out how to reach the aquarium. Maps are an important, high-value prior for navigation tasks [1]. However, obtaining high-precision maps that provide accurate coordinates and semantic information is often difficult. Abstract maps, such as the one in Figure 1, are much more common in everyday life. We call this problem long-distance autonomous navigation in an open world with weak priors: given only an abstract map, an agent must autonomously complete a long-distance urban navigation task, traversing intersections, streets, walkways, and open areas without GPS, prior mapping, or any external localization.

Figure 1. Park map for the navigation task. The red arrow marks the initial position; the destination is the basketball court.
Figure 1. Park map for the navigation task. The red arrow marks the initial position; the destination is the basketball court.
Video 1. PathPainter extracts roads from an abstract map and plans a global path.

Our earlier work, PathPainter [2], focused on deriving global path plans from maps. It combines powerful image generation models such as Nano Banana / GPT Image with the conventional A* path planning algorithm to perform global path planning. During execution, however, it still relies on global localization and lacks an understanding of the surrounding environment.

Two Paths to Intelligence: Emergence or Design?

As GPT-6 Astra advances across a wide range of fields, we are curious: although Astra has shown enormous potential in some tasks, can it continue to operate reliably over truly long-horizon urban navigation? Is a more powerful foundation model the only answer? We used GPT-6 Astra Ultra + Codex (hereafter “GPT”) to run the task shown in Figure 1 fully autonomously in simulation.

Figure 2. System architecture used for the GPT-6 Astra Ultra + Codex tests. The atomic actions are up, down, left, right, forward, and backward.
Figure 2. System architecture used for the GPT-6 Astra Ultra + Codex tests. The atomic actions are up, down, left, right, forward, and backward.

The results showed that GPT completed the same task with a success rate of nearly 100%, displaying remarkable autonomy and intelligence. Using a monocular RGB camera, it chose to retreat when it reached impassable areas, observe and explore when the map differed from reality, and seek landmarks to localize itself when its position was unclear. It still faced problems: its trajectories were not always appropriate, and it often moved onto obstacles, while collisions in the real world would carry substantial costs in hardware damage and safety risks. Yet these problems seemed almost minor in the face of the intelligence it displayed.

Video 2. GPT-6 Astra Ultra + Codex navigates to the basketball court in Figure 1: trial 1 of three, each following a different route.
Video 3. GPT-6 Astra Ultra + Codex navigates to the basketball court in Figure 1: trial 2 of three, each following a different route.
Video 4. GPT-6 Astra Ultra + Codex navigates to the basketball court in Figure 1: trial 3 of three, each following a different route.

At the same time, we built a navigation agent based on a Qwen model that likewise needs no global localization and can move freely through a city as a human would. We use four cameras and a ranging sensor. The continuous input from four camera views places substantial pressure on the context window and requires the agent to accurately track task progress, promptly understand environmental changes, and adjust its actions. As shown in Figure 3, the system has a hierarchical architecture inspired by fast and slow systems. The upper level plans a global path using an improved version of PathPainter [2]. At the lower level, a Subgoal Agent executes subgoals, while a high-frequency Monitor Agent checks in real time whether the current subgoal has been completed. The Subgoal Agent calls NavDreamer [3] as its World Action Model.

Figure 3. Hierarchical Qwen navigation architecture, combining global path planning, a Subgoal Agent, and a Monitor Agent. All agents use QWEN 3.8 Next Flash fp8.
Figure 3. Hierarchical Qwen navigation architecture, combining global path planning, a Subgoal Agent, and a Monitor Agent. All agents use QWEN 3.8 Next Flash fp8.

On the task shown in Figure 1, this system reliably follows a 570 m route from one corner of the map to the basketball court entrance at the opposite corner, passing through roads, walkways, plazas, and stairs. Compared with GPT, its trajectories are smoother, steadier, safer, and more consistent with social norms.

Video 5. A Qwen-based navigation harness completes the 570 m route to the basketball court.

The Convergence of Intelligence

The hierarchical architecture is controllable and interpretable, but depends on extensive manual design and tuning; its capability boundaries are explicitly defined by its designers. Using only monocular RGB input, GPT completed the same task in simulation with a success rate close to 100%, demonstrating retreat, exploration, and autonomous localization. However, its decision-making process is not interpretable, and the stability of its behavior remains difficult to guarantee. This 570 m simulation test suggests that foundation models and harness designs from leading AI companies are rapidly encroaching on the capability boundaries of manually engineered architectures.

Success in simulation must still be tested against real environments, inference latency, API costs, and long-tail failures. Next, we will deploy both systems on physical robots for comparison and explore their integration: a foundation model serving as the “brain,” with a World Action Model acting as a controllable, high-frequency “cerebellum,” so that general intelligence and interpretable architectures can complement each other. We believe that the real, noisy, and ever-changing world may be the ultimate test.

Research authors: Yijin Wang, Tianyang Chen, Shaokai Jin

An arXiv preprint is coming soon.

2. Long-horizon navigation in unknown environments

Specify “What to Find,” Without Specifying “How to Get There”

We make this task setting more challenging. Imagine entering an unfamiliar shopping mall for the first time: even without knowing the exact location of the restroom, a person can search using common sense, the spatial layout, and signage [4]. This reflects an important capability: autonomously finding a target through common sense and visual observation when the goal is clear but the environment is unknown. In recent years, some vision-and-language navigation work has continued to focus primarily on following language instructions that include route hints. For example, NaVILA’s real-robot demonstration of navigating to a restroom provides route information in advance, including going around stairs, entering a corridor, and turning right [5]. In contrast, we want a robot in a completely unknown environment to receive only an instruction such as “find the restroom” and independently decide where to explore using common sense and observations on site, without a human describing the route to the destination.

In this setting, we first tried calling GPT-6-Astra directly to make navigation decisions. In the experiment, the model’s environmental observations came only from images captured by four cameras, which it combined with the task instruction to decide its next action. Although GPT-6-Astra has strong visual understanding and reasoning capabilities, a lack of historical context can still cause repeated exploration. Simply calling the model and executing its output is therefore insufficient to guarantee successful long-horizon navigation.

Video 6. Finding the kitchen by directly calling GPT-6 Astra for navigation decisions.

From Single-Step Intelligence to Long-Horizon Autonomy

To address this problem, we built a harness for GPT-6 Astra to perform long-horizon autonomous goal navigation in completely unknown environments. The framework focuses on supplying historical context for sequential decisions, allowing the model to build on previous exploration and reduce repeated visits to the same areas. First, the model decomposes the natural-language goal into staged subtasks, and the system saves and maintains task progress. Before each decision round, the system obtains RGB images from the front, right, rear, and left cameras, together with odometry, and uses MoGe-2 to estimate a depth map for each view [6]. The model combines the current images, task progress, historical memory, and feedback from the previous execution to decide whether to move, turn, observe again, or revise the plan. The model directly selects a pixel location in an image as the movement target, which is then transformed into world coordinates using depth and pose. After the drone reaches the target point, the system records target cues, the actual start and end positions of the movement, and the execution result, updates memory and task state, and proceeds to the next decision round. This creates a closed loop of “planning—observation—decision—execution—feedback.”

Figure 4. Navigation harness for unknown environments. Task planning, four-view observation, memory, execution, and feedback form a continuous decision loop.
Figure 4. Navigation harness for unknown environments. Task planning, four-view observation, memory, execution, and feedback form a continuous decision loop.

Taking Autonomous Exploration Further

We conducted experiments in a simulated three-story villa with a total floor area exceeding 300 square meters. The environment is large, contains many rooms, and has a complex layout requiring route choices between rooms and movement across floors. In this unknown environment, we set navigation tasks such as “find the kitchen on the second floor” and “find the room with pink seat cushions,” requiring the system to explore autonomously and reach the target based on natural-language instructions. Across 12 experiments, the system reached the target successfully in 10, achieving a target-reaching rate of 83.3%. In one “find the kitchen on the second floor” task (Video 9), the system autonomously generated six staged subtasks. During execution, the drone started in a first-floor room, found the stairs, and climbed to the second floor. After discovering that a room on one side was a bedroom, it returned and changed its exploration direction. It eventually identified the kitchen using visual cues such as a sink, worktops, and cabinets, flew into the room, and hovered. The entire process involved 36 calls to GPT-6-Astra, completing navigation from exploration of an unknown environment to target confirmation and entry. Notably, when we replaced the core model with Qwen3.8-Flash-Next while keeping the same harness unchanged, the system could still complete some navigation tasks successfully, although its success rate was lower than with GPT-6-Astra.

In the future, we plan to introduce RSI to continually optimize the navigation harness, addressing issues such as occasional inaccuracies in determining subtask completion and reduced performance when transferring to other models, such as Qwen3.8-Flash-Next. Based on execution feedback and evaluation results, the system will automatically propose, test, and retain effective improvements to task decomposition and completion assessment, and validate them across different models to improve cross-model generalization. We also plan to deploy the system on physical robot platforms for navigation tests in real environments.

Video 7. Finding the kitchen on the current floor using the navigation harness.
Video 8. Finding the room with pink seat cushions using the navigation harness.
Video 9. Finding the kitchen on the second floor, including staircase traversal and target confirmation.
Video 10. Finding the meeting room with the white lattice dome in a large-scale laboratory environment.

Research authors: Wenjun Li, Jinhan Li, Mo Zhu

An arXiv preprint is coming soon.

3. Heterogeneous robots, working as a team

When Intelligence Extends Beyond the Individual

What happens when a single-robot task becomes a multi-robot team task? Imagine a maintenance team entering a factory that has stopped production, with only the instruction to “repair the equipment, make the product, and deliver it to the delivery station.” The equipment and materials must still be located on site; team members exchange clues, divide work according to their expertise, and adjust their arrangements as circumstances change. We envision a heterogeneous robot team—for example, a reconnaissance drone, an aerial manipulator, a mobile manipulator, and a humanoid—collaborating in the same way: the drone finds a repair kit, the mobile manipulator retrieves and transports it, the humanoid takes over installation, and the aerial manipulator searches for raw materials at the same time. Only once the equipment is repaired and the materials are ready can the team connect power and begin production.

Our earlier D-VLC work [7] demonstrated robots sharing visual cues, collaborating to open a door, and continuing their search. Now, we want a wider variety of robots to sustain handoffs in unknown environments and complete tasks such as repair, manufacturing, and delivery. They must not only find objects but also arrange who acts first, who takes over, and which work can proceed in parallel.

Video 11. D-VLC collaboration while searching for a restroom in an unknown building: a drone requests help, and a mobile manipulator opens a door.

In this more challenging setting, we again first tried having GPT-6 Astra directly decide the next actions of all four robots. In each round, the model reads the overall task, the camera views from the four robots, the observed map, and execution feedback, then produces the robots’ next local actions. After the robots execute them, the results are fed back to the model. As shown in Video 12, despite GPT-6 Astra’s strong understanding and reasoning capabilities, choosing actions one round at a time can still make it difficult to maintain the task progress and division of labor needed for long-horizon collaboration.

Figure 5. The heterogeneous robot team: a drone, an aerial manipulator, a mobile manipulator, and a humanoid.
Figure 5. The heterogeneous robot team: a drone, an aerial manipulator, a mobile manipulator, and a humanoid.
Video 12. Four heterogeneous robots under direct GPT-6 Astra control carry out the factory task of producing and delivering one valve-and-pipe assembly.

Self-Evolution of the Multi-Agent Framework May Be the Answer

To address slow task progress and inefficient reasoning when directly generating actions, we built on D-VLC’s multi-robot agent architecture to organize division of labor and collaboration over long-horizon tasks. In this framework, the execution model decomposes the overall task into subtasks with ordering dependencies and records discovered objects, each robot’s current work, and completed steps. Before each execution, the system combines these records with the outcome of the previous operation to assign work that can proceed in parallel. When an operation fails or the task changes, it preserves the completed portions and adjusts the remaining plan. On this basis, we draw on RSI to let GPT-6 Astra continually improve the framework based on experimental results. After each experiment, it reads videos, action records, and waiting logs, identifies the causes of stalled progress or repeated operations, and modifies the relevant code. After modification, the code first passes targeted tests, then runs the full task to check whether execution time has decreased and whether other functions have been affected. Effective changes are retained for the next round. Researchers define the goals, constrain the scope of modifications, and inspect the results. In this way, problems exposed in each experiment become the basis for the next round of code changes. As shown in Video 14, the evolved system completed search, repair, production, and delivery in the factory, demonstrating the value of combining model capabilities with an iteratively improvable collaboration architecture.

Figure 6. GPT-6 Astra-assisted iteration of the robot system, linking task decomposition, collaborative execution, and feedback-based correction.
Figure 6. GPT-6 Astra-assisted iteration of the robot system, linking task decomposition, collaborative execution, and feedback-based correction.

Autonomous Collaboration Through Collective Intelligence

We completed two full simulation experiments—a four-robot factory task and a two-story police station task—using the same frozen version of the code. The factory’s navigation map covers approximately 102 × 82 meters. Of the seven initial target categories, only the location of the delivery station was provided in advance. The model generated 12 subtasks with dependencies: repair and the preparation of two types of material could proceed in parallel, followed by connecting power, production, delivery, and disconnecting power. The robots executed 150 successful exploration skills and 12 successful non-exploration skills, completing the task in 12 minutes and 50 seconds. In the police station, each floor’s navigation map is approximately 50.5 × 50.5 meters, and the task involves 10 categories of objects. The model initially decomposed it into 22 subtasks, later adjusting this to 24 based on on-site feedback. These included selecting materials according to the work order and diagnosis, isolating equipment for repair, coordinated startup by two robots, quality inspection, delivery, and disconnecting power. The four robots executed 78 successful exploration skills and 24 successful non-exploration skills, completing the task in 14 minutes and 48 seconds. We also tested a task change midway through execution: in the factory task, after repair and preparation of the first set of materials but before power was connected, we interrupted the order and changed it from one assembly to two. The system preserved completed operations, prepared only the additional materials for the second assembly, and ultimately delivered 2/2 assemblies before shutting down and disconnecting power.

These results demonstrate the possibility of robots completing complex work as a team: members search for clues individually, share progress, and take over retrieval, transport, and manipulation according to their capabilities. The same code version completed different workflows in both the factory and the police station, also showing that by maintaining task dependencies and execution feedback, the collaboration framework can organize model capabilities into team actions that make sustained progress across scenarios. This provides a direction for our continued use of GPT-6-assisted RSI: enabling the system to accumulate evidence for improvements from each experiment and gradually adapt to longer tasks with more changes. In the future, we will continue RSI-style iteration to improve task efficiency and adaptation across environments, support heterogeneous robots dynamically joining or leaving the team and more task changes, and conduct physical experiments.

Figure 7. The two-story police-station simulation environment used for the four-robot collaboration task.
Figure 7. The two-story police-station simulation environment used for the four-robot collaboration task.
Video 13. Four heterogeneous robots collaborate in the police station to repair a cutting machine, make keys according to a work order, inspect and deliver them, and disconnect power.
Figure 8. The factory simulation environment used for equipment repair, production, and delivery tasks.
Figure 8. The factory simulation environment used for equipment repair, production, and delivery tasks.
Video 14. Four heterogeneous robots complete the factory task using the evolved Agent RSI framework, covering search, repair, production, and delivery.
Video 15. The factory order changes from one valve-and-pipe assembly to two during execution. The team retains completed work, prepares the additional materials, and delivers both assemblies.

Research authors: Yuan Zhou, Ruitong Lin, Xingxing Li, Shen Wang, Mo Zhu

An arXiv preprint is coming soon.

Conclusion

The results from long-distance navigation with weak priors, reasoning-based navigation in completely unknown environments, and distributed heterogeneous multi-robot collaboration indicate that GPT-6 Astra already possesses the complex reasoning and spatial understanding capabilities needed for general-purpose navigation. As tasks become more complex—for example, when coordinating multiple heterogeneous robots—they place greater demands on both model capabilities and the completeness of the Agent Harness. The emergence of GPT-6 Astra makes workflows spanning data, training, and closed-loop execution more flexible. Visual understanding and high-level decision-making that once required specially trained VLAs and complex engineering modules may now be achievable with general-purpose models combined with the memory, task management, and execution feedback provided by a harness. At the same time, the models’ ability to understand complex systems, modify code, and run automated tests [8] makes RSI a research direction worth exploring [9] [10]: it can accelerate engineering iteration and may also allow systems to gradually develop the ability to improve themselves. What truly merits research may be the problems that cannot easily be displaced by increases in model scale and capability, together with broadly applicable methods for solving them. Moreover, robotics research must ultimately return to the real world. Simulation and offline metrics reveal only part of the picture; closed-loop validation on physical robots offers a stronger test of practical effectiveness, robustness, and generalization. More convincing than impressive experimental numbers is whether a system can reliably complete tasks in complex real environments.

Project leads: Yuze Wu, Fei Gao, Xin Zhou

References

[1] Li, Anqi, et al. "Urbanvla: A vision-language-action model for urban micromobility." arXiv preprint arXiv:2510.23576 ↗ (2025).

[2] Wang, Yijin, et al. "PathPainter: Transferring the Generalization Ability of Image Generation Models to Embodied Navigation." arXiv preprint arXiv:2605.07496 ↗ (2026).

[3] Huang, Xijie, et al. "Navdreamer: Video models as zero-shot 3d navigators." IEEE Robotics and Automation Letters (2026).

[4] Shah, D., et al. Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning. CoRL, 2023.

[5] Cheng, A., et al. NaVILA: Legged Robot Vision-Language-Action Model for Navigation. RSS, 2025.

[6] Wang, R., et al. MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details. NeurIPS, 2025.

[7] Zhou, Yuan, et al. "D-VLC: Decentralized Vision-Language Collaboration for Heterogeneous Embodied Multi-Robot Systems in Unknown Environments." arXiv preprint arXiv:2607.29009 ↗ (2026).

[8] Yang, John, et al. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." Advances in Neural Information Processing Systems 37 (2024): 50528-50652.

[9] Zhang, Jenny, et al. "Hyperagents." arXiv preprint arXiv:2603.19461 ↗ (2026).

[10] Jenny Zhang, et al. "Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents." arXiv preprint arXiv:2505.22954 ↗ (2025).