One API to Compare Text, Images, Audio, and Video

The Python embedding library Sentence Transformers released its v5.4 update on April 9, 2026, officially adding multimodal embedding and reranking capabilities. With this update, developers can use the same model.encode() API to map text, images, audio, and video into a shared embedding space. According to a post on the Hugging Face Blog, the additions directly target visual document retrieval, cross-modal semantic search, and multimodal Retrieval-Augmented Generation (RAG) pipelines.

What Are Multimodal Models?

Traditional embedding models convert text into fixed-size vectors. Multimodal embedding models extend this concept by projecting inputs from different modalities — text, images, audio, and video — into a single shared embedding space. This means a text query can be used to search image documents, or a description can retrieve matching video clips, all using the same cosine similarity functions developers already know.

The same applies to reranker (Cross Encoder) models. Previously limited to text-text pairs, these models can now score the relevance of image-text mixed pairs or documents that combine text and images.

Installation and Hardware Requirements

Multimodal features require additional dependencies per modality:

pip install -U "sentence-transformers[image]"
pip install -U "sentence-transformers[audio]"
pip install -U "sentence-transformers[video]"
pip install -U "sentence-transformers[image,video,train]"

Vision-Language Model (VLM)-based models like Qwen3-VL-2B require at least 8 GB of VRAM for the 2B variant, and approximately 20 GB for the 8B variants. CPU inference is extremely slow for these models; text-only or CLIP-based models are recommended for CPU environments. Cloud GPU services like Google Colab are suggested for those without a local GPU.

What Changed vs. Previous Versions

Model loading remains unchanged. Simply specifying a model name such as SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B") allows the library to auto-detect supported modalities. Fine-grained settings like image resolution and model precision can be controlled via Processor and Model kwargs.

Supported Pipelines

Three primary pipeline types are now unlocked by v5.4:

Cross-modal semantic search: Search image and video documents with a text query, or vice versa, using standard cosine similarity operations.

Multimodal RAG pipelines: Index visual documents such as image-rich PDFs, slides, and webpages into an embedding database, then retrieve and rerank them using text queries.

Mixed-modality reranking: When initial retrieval returns a mixed list of text and image documents, the reranker model produces a unified relevance score across all entries.

[Expert Analysis] Multimodal RAG Adoption Likely to Accelerate

This update significantly lowers the barrier to building multimodal search infrastructure. While multimodal RAG has attracted conceptual interest, its production adoption has been slow due to implementation complexity and the need for separate modality-specific pipelines.

Sentence Transformers already occupies a de facto standard position in the Python embedding ecosystem. By maintaining the same API interface while extending to multimodal, the update makes it highly likely that existing text-only RAG pipelines can incorporate image search capabilities with minimal code changes.

However, the GPU memory requirements of VLM-based models — at least 8 GB for Qwen3-VL-2B — may still pose a challenge for developers experimenting locally. As lighter multimodal embedding models emerge, adoption rates are expected to rise rapidly. Hugging Face's simultaneous publication of a training and fine-tuning guide for multimodal models suggests a clear intent to foster a broader custom multimodal model ecosystem beyond inference alone.