A Hierarchical Onboard Vision-Language-Navigation Framework for Aerial Robots in Complex Environments

IEEE Transactions on Robotics · 2026

Yuze Wu1,2,*, Mo Zhu1,2,*, Xingxing Li2,*, Yuheng Du2, Zhaoqi Wang2, Yuxin Fan1,2, Wenjun Li1,2, Kenghou Hoi1,2, Zhichao Han1,2, Xin Zhou1,2,†, Fei Gao1,2,†

1 State Key Laboratory of Industrial Control Technology, Zhejiang University
2 Differential Robotics

* Equal contribution   ·   † Corresponding authors

Overview

VLN-AR enables aerial robots to follow natural-language instructions using onboard cameras and computation. A vision-language model identifies the next goal, a memory module tracks progress through multi-step instructions, and a lightweight planner converts each goal into a feasible flight path. We train the model in three stages using photorealistic scenes reconstructed with 3D Gaussian Splatting. We evaluate VLN-AR in simulation and on two aerial robots navigating indoor and outdoor environments.

Video Highlights

Flight highlights

Outdoor navigation (6×), indoor navigation (4×), a long-horizon task (8×), and target tracking (3×).

Indoor navigation

The robot follows instructions using object descriptions and spatial relations. Flights are shown at 4× and 8× speed.

Outdoor navigation and tracking

Landmark navigation at 6×, additional outdoor trials at 5×, and target tracking at 3×.

Long-horizon navigation and robustness

A multi-step navigation task at 8×, followed by robustness tests at 3×.

Project Overview

An overview of data collection, model training, and deployment on the aircraft.

Results

Real-world navigation
90.1%
91 of 101 flight trials completed across two aircraft.
Long-horizon navigation
88.0%
Multi-step task success with the navigation agent, up from 57.4% without it. Table VI
Onboard decoding
7.1–9.4×
Per-token speedup across 2B–7B models on Jetson Orin NX at 30 W. Table III

Real-world robustness

Successful flights out of 20 trials per condition, with no additional fine-tuning. Table IV · Figure 11

  • A square gate suspended between obstacles in the lab.
    Suspended obstacles18/20
  • The aerial robot rises above a whiteboard in a two-story building.
    Height changes >5 m19/20
  • The robot navigates an outdoor walkway at night.
    Low light at night17/20
  • A person crosses the robot’s path in a furnished room.
    Moving obstacles17/20
  • The robot approaches a chair in a previously unseen office.
    Unseen scenes18/20
Example flight in an unseen office
The aerial robot faces a group of office chairs. The robot reaches the chair with clothing draped over it. The robot flies away from the chair toward a cardboard box. The robot approaches the delivery box on a white pedestal.
“Go to the chair where the clothes are placed, and then approach the delivery box.” Figure 11

Closed-loop simulation

Evaluation setSuccess rateNavigation errorTrajectory time
AerialVLN78.15%22.12 m41.51 s
OpenFly76.88%23.22 m40.07 s
Held-out indoor scenes92.66%5.75 m16.18 s

Trajectory time is averaged over successful trials. Table II

Method

VLN-AR architecture showing image and instruction inputs, navigation predictions from the vision-language model, and the action expert that generates a flight trajectory.
The onboard navigation model and action expert (Figure 3 in the paper).

The vision-language model combines the instruction with current and past camera views to predict a target pose and decide when to replan or finish a task. For longer instructions, a navigation agent keeps a record of completed subtasks. The action expert uses the predicted goal and a depth image to plan a dynamically feasible trajectory.

Two supervised training stages teach visual grounding and navigation using a mixture of aerial navigation and vision-language data. Reinforcement learning in closed-loop simulation then refines the model’s decisions. At deployment, instruction-guided visual token selection reduces the computation required on the aircraft.

Citation

BibTeX
Download .bib
@ARTICLE{11803153,
  author={Wu, Yuze and Zhu, Mo and Li, Xingxing and Du, Yuheng and Wang, Zhaoqi and Fan, Yuxin and Li, Wenjun and Hoi, Kenghou and Han, Zhichao and Zhou, Xin and Gao, Fei},
  journal={IEEE Transactions on Robotics}, 
  title={A Hierarchical Onboard Vision-Language-Navigation Framework for Aerial Robots in Complex Environments}, 
  year={2026},
  volume={},
  number={},
  pages={1-20},
  keywords={Modeling;Navigation;Autonomous aerial vehicles;Cognition;Cognitive systems;Optimization;Training;Educational institutions;Trajectory;Argon;Aerial system;Autonomous navigation;Foundation model;Onboard Deployment},
  doi={10.1109/TRO.2026.3754505}}