MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings

Multimodal embedding models, built upon causal Vision Language Models (VLMs), have shown promise in various tasks. However, current approaches face three key limitations: the use of causal attention in VLM backbones is suboptimal for embedding tasks; scalability issues due to reliance on high-quality labeled paired data for contrastive learning; and limited diversity in training objectives and data. To address these issues, we propose MoCa, a two-stage framework for transforming pre-trained VLMs into effective bidirectional multimodal embedding models. The first stage, Modality-aware Continual Pre-training, introduces a joint reconstruction objective that simultaneously denoises interleaved text and image inputs, enhancing bidirectional context-aware reasoning. The second stage, Heterogeneous Contrastive Fine-tuning, leverages diverse, semantically rich multimodal data beyond simple image-caption pairs to enhance generalization and alignment. Our method addresses the stated limitations by introducing bidirectional attention through continual pre-training, scaling effectively with massive unlabeled datasets via joint reconstruction objectives, and utilizing diverse multimodal data for enhanced representation robustness. Experiments demonstrate that MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results, and exhibits strong scalability with both model size and training data on MMEB.

Starting from pre-trained VLMs, our approach consists of two stages:

1. Modality-aware Continual Pre-training: We introduce a joint reconstruction objective that requires the model to simultaneously denoise interleaved text and image inputs, which encourages the model to jointly reason across modalities.
   1. For text: We apply masked language modeling (MLM), where masked tokens are predicted using the full multimodal context.
   2. For images: We adopt masked autoencoding (MAE): a subset of image patches are randomly masked and reconstructed by a lightweight decoder conditioned on image and text contexts.
2. Heterogeneous Contrastive Fine-tuning: As opposed to previous works which used mainly image-caption pairs, we add diverse heterogeneous data including:
   1. Long-form query-document pairs: Supporting document-level understanding and complex reasoning over extended context
   2. Curated multimodal pairs: Offering various visual and textual contexts beyond image captions of specific distributions
   3. Real-world text pairs: Enhancing linguistic representations across diverse domains

We present the general multimodal embedding performances on MMEB and the long-form document-level retrieval performances on ViDoRe-v2. MoCa surpasses all strong baselines on both benchmarks, demonstrating the effectiveness of our training pipeline in improving multimodal embedding quality.

To understand the contribution of each major design choice in our framework, we conduct ablation studies on both the Modality-aware Continual Pre-training and Heterogeneous Contrastive Fine-tuning stages. As shown in in the following table, removing any key component leads to a consistent performance drop on both benchmarks (MMEB and Vidore-v2), demonstrating the importance of each part.

To assess how the data scale of CPT affects the quality of the learned embeddings, we investigate the scaling effect of our framework. We conduct contrastive fine-tuning (CL) using checkpoints saved at different steps during a single CPT run for both 3B and 7B models.