Three slots, every paper
File every paper into these three slots and the literature stops feeling like a zoo. When you read a new RS-VLM paper, ask only: what changed in each slot compared with LLaVA-1.5?
What encodes the pixels?
The vision encoder turns the image into a grid of feature vectors. Nearly everyone uses a CLIP/SigLIP ViT, which is exactly the field's weakness (CLIP-blindness: counting, relative position, small objects). The frontier move is swapping in an EO-native encoder.
What bridges to the LLM?
The bridge (projector) maps vision features into the LLM's token embedding space, so image patches become "words" the LLM can attend to. Simple MLPs won; query-based compressors trade fidelity for fewer tokens.
What grounds the output?
How does language connect back to space? This is where RS-VLMs genuinely innovate: from coordinates written as plain text, to special tokens that trigger a mask decoder, to agentic zoom loops.
Pick a model, watch what changes
Each model is drawn as a diff against the LLaVA-1.5 baseline. Shapes assume a 336×336 RGB input unless noted.
Inside each stage
Open the stage you're stuck on. The first one is open to start.
Slot APatchify & the vision encoder
24×24 = 576 patch tokens, each projected to a 1024-d vector, giving [B, 576, 1024]. Positional embeddings add location; ~24 transformer layers mix information globally.Why this matters for RS: a 10k×10k scene resized to 336px makes a car smaller than one patch — it simply ceases to exist for the model. Fixes: AnyRes tiling (encode crops separately, concat tokens), or agentic zooming. Also note CLIP was contrastively trained to match captions — it keeps what is in the image, discards where and how many. That bias is inherited by every downstream RS-VLM (measured by RSMMVP).
Slot BThe projector — the entire "alignment" trick
vision_feats [B, 576, 1024] → Linear(1024→4096) → GELU → Linear(4096→4096) → visual tokens [B, 576, 4096] # now "words"These are spliced into the prompt:
<sys> USER: <img₁…img₅₇₆> What is in the harbor? ASSISTANT:. Training Stage 1 freezes everything except this projector — hours, not days, of GPU time. That such a small module suffices tells you CLIP features and LLM word embeddings are already nearly linearly alignable — the deepest single fact in VLM land.
Slot CGrounding mechanisms — compared
<obb>{47, 112, 98, 163, 30°}</obb>. Zero extra architecture; grounding is "just" next-token prediction. Fragile — numbers drift after further fine-tuning, precision limited by tokenizer granularity.Special token + decoder (GeoPixel, LISA-style): the LLM emits a
<SEG> token; its final hidden state becomes the prompt embedding for a SAM-style mask decoder → true pixel mask. Precise, adds a decoder to train, needs mask-annotated dialogue data.Attention prompting (EarthMind): reweights LLM attention toward spatially relevant visual tokens; multi-granular (image→region→pixel) without coordinates in the text stream.
Agentic (GeoViS/GeoEyes): grounding becomes a policy — crop, look, decide whether to zoom again. Handles 10k×10k scenes; costs multiple forward passes per answer.
TrainingThe three training stages
Stage 2 — Instruction tuning: unfreeze LLM (or LoRA r=16–64), train on multi-task conversations (VQA, grounding, captioning mixed). This is where capabilities come from — and where RS models differentiate: GeoChat's 318k RS instructions, EarthDial's 11M+, MMRS-1M's multi-sensor set. Data construction is the real contribution of most RS-VLM papers.
Stage 3 (optional): mask-decoder training (GeoPixel), preference/DPO for robustness (RemoteShield), RL for search policies (GeoViS).
Gap G12Why multispectral input is hard for VLMs
Four questions to test the model
If you can answer these without scrolling up, the three-slot model has stuck.
Next: see all 61 RS-VLMs filed into the three slots, side by side.
Open the Comparison Matrix →