NaVILA
Cheng 2024 · Paper
One-line summary — NaVILA extends the vision-language-action paradigm from tabletop manipulation to legged-robot navigation, pairing a vision-language model that emits mid-level actions in natural language with an RL locomotion policy that executes them, so a robot can follow language instructions through real 3D environments.
Problem
Vision-and-Language Navigation (VLN) with legged robots is attractive twice over: language is a flexible way for humans to command robots, and legs let the robot traverse challenging, cluttered scenes that defeat wheeled bases. But translating a human instruction all the way down to joint-level control is hard — manipulation VLAs quantize low-level commands into tokens, yet LLMs/VLMs were trained on natural language, and forcing them to emit precise non-verbal actions strains the very reasoning they were pretrained for. NaVILA asks whether language itself is the better action representation.
Method & architecture
- Two-level framework: a VILA-based VLM handles semantic instruction following and outputs mid-level actions expressed in language — e.g. “turn right 30 degrees”, “move forward 75 cm” — while a separate low-level visual locomotion policy converts them into joint movements in real time. The VLA runs at low frequency; the locomotion policy runs in real time, absorbing obstacle avoidance and balance.
- VLM (high level): VILA’s vision encoder → MLP projector → LLM, all unfrozen during SFT. A navigation prompt distinguishes the latest frame (“current observation”) from uniformly sampled earlier frames (“a video of historical observations” as a memory bank), using textual cues instead of new special tokens; 8-64 frames are used. A regex parser extracts the action type and its argument (distance/angle) from the LLM output — empirically every emitted action parsed successfully.
- SFT data blend: (1) simulated navigation from R2R-CE and RxR-CE via a shortest-path follower, with consecutive actions merged (up to three, e.g. two 25 cm steps → one 50 cm step) and stop-action rebalancing; (2) real human touring videos — 2K egocentric YouTube videos sampled into 20K trajectories, with actions recovered by MASt3R metric camera-pose estimation and instructions generated by VLM captioning + LLM rephrasing (first work to train navigation directly on human video in continuous settings); (3) auxiliary tasks — EnvDrop augmented instructions, trajectory summarization, ScanQA 3D question answering; (4) general VQA to retain world knowledge.
- Locomotion policy (low level): the discrete commands are cast to velocity commands executed for durations matching the VLA’s stated distance/angle. A single-stage PPO policy (no teacher-student distillation) outputs desired joint positions for the Unitree Go2’s leg motors. The critic sees privileged terrain height scans; the actor sees only real-world-available proprioception history plus a 2.5D height map built from LiDAR point clouds (Unitree L1, 360°×90° FoV, max-filtered over the last 5 scans) — LiDAR handles glass and strong sunlight where depth/RGB fail. Ray-casting in Isaac Lab yields >60K FPS training throughput on an RTX 4090.
- VLN-CE-Isaac benchmark: a new Isaac Sim benchmark with 1,077 traversable R2R Val-Unseen trajectories, evaluating full-physics joint-level execution rather than Habitat-style teleporting agents.
Results
- VLN-CE benchmarks (Val-Unseen): on R2R-CE NaVILA reaches SR 54.0 / SPL 49.0 / NE 5.22 using only single-view RGB — a ~17% SR improvement over prior single-view SoTA NaVid (SR 37.0) and comparable or superior to methods using panoramas, depth, odometry, or simulator-pretrained waypoint predictors; on RxR-CE it scores SR 49.3 / SPL 44.0 / nDTW 58.8.
- Cross-dataset zero-shot (trained on R2R only, tested on RxR Val-Unseen): SR 34.3 vs NaVid’s 23.8.
- Scene understanding: on ScanQA validation, NaVILA (64 frames) reaches CIDEr 102.7 vs the prior VLA SoTA NaviLLM’s 75.9, beating even 3D LMMs that require scans or posed RGB-D.
- Locomotion: vs distillation-based ROA, linear/angular velocity errors drop from 0.161/0.152 to 0.066/0.113 and collision rate from 3.09 to 0.81.
- VLN-CE-Isaac: the vision-based policy beats the blind (proprioception-only) policy by 14% SR on Go2 (50.2 vs 36.2) and 21% on the H1 humanoid (45.3 vs 24.4), approaching the oracle-execution upper bound (51.3).
- Real world: across 25 instructions × 3 trials in workspace, home, and outdoor scenes, NaVILA achieves an 88% success rate (75% on complex multi-room instructions), outperforming a GPT-4o baseline; the same VLA drives a Booster T1 humanoid without retraining, despite different camera height and view angle.
Why it matters for SLAM
NaVILA is a concrete example of the foundation-model challenge to classical SLAM-based navigation stacks: it replaces the explicit map-build-then-plan pipeline with an instruction-conditioned policy whose “map” is a bank of memory frames. At the same time, its design shows where geometry still fits — MASt3R pose estimation labels its human-video training data, a LiDAR height map keeps the robot safe, and metric localization and persistent maps are complementary to a VLA’s semantic instruction following; hybrid systems that give VLAs SLAM-grounded spatial memory are an active research direction.