Beyond Text: The Era of Training Embedding Models on Images and Documents

Hugging Face's Python library Sentence Transformers has officially launched support for training and finetuning multimodal embedding and reranker models. In a blog post published on April 16, 2026, developer Tom Aarsen detailed the full pipeline for finetuning multimodal models — capable of handling text, images, audio, and video — on user-specific domain data. In his experiment, finetuning Qwen/Qwen3-VL-Embedding-2B for the Visual Document Retrieval (VDR) task boosted the NDCG@10 score from 0.888 to 0.947, surpassing all existing VDR models tested, including those up to four times larger.

Why Finetuning Matters

General-purpose multimodal embedding models perform reasonably well across diverse tasks — image-text matching, visual question answering, document understanding. But generality rarely translates to optimal performance in a specific domain. In VDR, for instance, a model must find the most relevant document page image from thousands given a text query like "What was the company's Q3 revenue?" — a task requiring deep understanding of tables, charts, and document layouts. Finetuning is the primary mechanism to teach models these domain-specific patterns.

Aarsen demonstrated this with concrete numbers. The finetuned model tomaarsen/Qwen3-VL-Embedding-2B-vdr recorded an NDCG@10 of 0.947 on the evaluation dataset, surpassing the base model (0.888) and every other VDR model tested — including those with four times the parameters.

What Changed

The core design principle of this update is full compatibility with the existing text-only training pipeline. Developers use the same SentenceTransformerTrainer; adding a new modality is as simple as including images in the dataset, with the model's processor handling preprocessing automatically. Training components remain identical: Model, Dataset, Loss Function, Training Arguments, Evaluator, and Trainer.

Supported loss functions include CachedMultipleNegativesRankingLoss for efficient large-batch training and MatryoshkaLoss for simultaneously optimizing multiple embedding dimensions. The Matryoshka approach allows flexible use of embeddings at various dimensions from a single model, enabling trade-off adjustments between storage and retrieval speed.

[Expert Analysis] Reshaping the Multimodal Retrieval Landscape

This update is likely to bring substantive change to the multimodal retrieval ecosystem beyond a simple library feature addition.

First, accessibility. Previously, finetuning multimodal embedding models required complex custom training code. Sentence Transformers' standardized pipeline dramatically lowers that barrier, enabling startups and small research teams to build domain-specialized models from their own data.

Second, the update challenges assumptions about model size versus performance. A 2B-parameter model outperforming 8B-scale models through domain finetuning validates the "specialization over scale" direction. For enterprises where multimodal document retrieval accuracy is a critical bottleneck in Retrieval-Augmented Generation (RAG) pipelines, finetuning strategies are likely to emerge as a viable alternative.

Third, VDR application expansion is expected. Industries handling documents rich with layout and visual information — financial reports, legal documents, medical imaging — are likely to accelerate adoption of VDR-based retrieval systems. High-value verticals such as enterprise document search, compliance review, and patent retrieval stand out as prime candidates.

That said, two challenges remain: multimodal model training demands substantial GPU memory and compute resources, and securing high-quality domain training data remains the pivotal variable for performance.