Edge-BehAV — Offline Vision-Language Navigation with Geometric Goal Localization

0.775Macro-F1 offline, vs. 0.765 for the best cloud API

~9×faster than the cloud API it matches

0.20 mmean goal-distance error, down from 2.05 m

31/32closed-loop outdoor trials

Overview

Vision-language navigation lets a robot follow a natural-language instruction, find the goal it names, and respect the behavioral rules it states. Current outdoor systems reach that capability by calling cloud-hosted foundation models, which makes them unusable without connectivity, and they infer how far away the goal is from a 2D projection rather than measuring it.

Edge-BehAV moves every one of those stages onto the robot and replaces the distance heuristic with sensed geometry. Nothing leaves the platform at any point.

How it works

System architecture: offline instruction decomposition, behavioral cost map, goal estimation by LiDAR-visual fusion, and behavior-aware MPC. — Four modules. Language decomposition and goal estimation run offline; the cost map and planner are retained from the reference architecture and extended with a hybrid goal cost.

Supplementary video

Four closed-loop outdoor runs, each shown from the robot's onboard view and a third-person view. Silent and captioned, 2 min 42 s.

Contributions

A navigation-specific benchmark of edge language models

17 offline small language models from seven families, measured against 4 cloud APIs on 350 human-annotated navigation instructions, across an NVIDIA Jetson Orin NX, an NVIDIA Jetson Thor, and a laptop RTX 4070. The benchmark and every per-instruction model output are in this repository.

Hybrid semantic-geometric goal localization

Florence-2 open-vocabulary detection proposes the goal, Mobile-SAM segments it from that box, and the mask is projected into the LiDAR cloud to recover a metric position through an IQR-filtered centroid. Beyond reliable LiDAR range the system falls back to visual bearing guidance and latches a metric goal once the target comes into range.

A complete onboard system

Both modules were integrated on a Clearpath Husky A200 and validated in 32 closed-loop outdoor trials with no cloud services and no network connectivity.

Instruction decomposition

Macro-F1 is the mean set-level F1 over the four annotated fields. Latency is the median per call, measured on the laptop RTX 4070.

Model	Type	Macro-F1	Latency (s)
Qwen2.5-7B ★	offline LLM	0.775	1.13
GPT-5.5	cloud API	0.765	9.95
Qwen2.5-3B	offline LLM	0.718	0.53
Gemma4-E4B	offline LLM	0.716	2.77
GPT-5.4-mini	cloud API	0.700	2.19
Gemma3-12B	offline LLM	0.683	2.95
Qwen2.5-VL-7B	offline VLM	0.662	4.58
GPT-4	cloud API	0.512	4.05

The top offline model and the strongest cloud API have overlapping 95% confidence intervals, so the difference between them is not statistically meaningful, and the offline model gets there roughly nine times faster with no network. Full results for all 21 models are in results/.

Quality against latency for every evaluated model, with the offline Pareto front highlighted. — Quality against latency. Offline models occupy the fast, high-quality corner that cloud APIs cannot reach.

Where models fail

Per-category accuracy heatmap across models and the seven benchmark categories. — Per-category performance. Contextual and adversarial phrasing remains the hardest category, and the gap does not close with model size alone.

Goal detection

Landmark localization compared against ground truth for Florence-2, GPT-4o, and GPT-5.5. — Landmark localization on the nuImages subset: Florence-2 running on-board against cloud VLMs, compared to ground truth.

Florence-2 reaches an F-score of 0.974 at 307 ms per image, against 0.904 at 3198 ms for the FastSAM and GPT-4o pipeline used by the reference system, measured over 922 nuImages frames across seven landmark classes.

The evaluation scripts and per-image results are in detection/. The images themselves are not redistributed; nuImages must be obtained from its own source.

Across all capabilities

Radar chart comparing Edge-BehAV against BehAV and cloud state of the art across four capability groups.

The fully offline system matches or exceeds both the original online pipeline and the strongest cloud API across all four evaluated capabilities: language decomposition, goal detection, distance estimation, and closed-loop navigation.

Axes are normalised so that the outer ring is ideal.

Platform

Clearpath Husky A200 with an Ouster OS1-64 LiDAR and an Intel RealSense L515 depth camera.

Closed-loop experiments ran on a Clearpath Husky A200 carrying an Ouster OS1-64 3D LiDAR and an Intel RealSense L515 depth camera, with all computation on-board.

Offline language models were additionally profiled on an NVIDIA Jetson Orin NX and an NVIDIA Jetson Thor using identical INT4 checkpoints.

Using the benchmark

350 instructions across seven semantic categories, 50 each, annotated by hand with landmarks, navigation actions, behavioral actions, behavioral targets, and per-target behavioral costs.

git clone https://github.com/aliiisa1375/Edge_Behav.git
cd Edge_Behav
pip install openpyxl

python3 scripts/score_macro_f1.py --results results/laptop_rtx4070/qwen2.5_7b.csv

Field definitions and annotation conventions are documented in benchmark/SCHEMA.md.

License and citation

The benchmark and result records are released under CC BY 4.0; the code is released under the MIT License.

This page accompanies a manuscript under double-anonymous review. Author names, affiliations, and the full citation will be added once the review process concludes.