Conan-embedding-v3: The Future of Omni-Modal Retrieval?

Omni-modal retrieval, the holy grail of embedding spaces, promises a unified platform for text, image, video, document, and audio inputs. But bringing this vision to life isn't easy. Each of these modalities comes with its own unique set of challenges, from data distribution to architectural needs. Enter Conan-embedding-v3, an ambitious attempt to crack this code.

Decoupled Specialist Fusion: A New Strategy

Conan-embedding-v3 introduces a novel approach called 'Decoupled Specialist Fusion.' Here, modalities are first trained independently as specialists. Once they're honed, their task vectors are fused into a single dense backbone. This isn't mere technical jargon, it's a strategic pivot that seeks to blend individual strengths into a cohesive whole.

But as promising as this sounds, the earnings call told a different story. When the backbone is fused, visual, video, and document retrieval capabilities shine. The issue? Audio. When an external encoder and projector are attached, the resulting 'Projector Drift' becomes a glaring problem. This drift leads to a significant regression in audio retrieval, despite all audio-specific modules being retained.

Projector Drift and Recovery

Let's talk about Projector Drift. It's a stumbling block in what could have been a perfect stride. Audio, which is increasingly vital in applications from AI assistants to media consumption, simply can't afford to lag behind.

To tackle this, Conan-embedding-v3 employs what they term 'Projector Recovery.' This involves a full-parameter fine-tuning of the projector while keeping the backbone itself untouched. It's followed by balanced multi-modal rehearsal, aiming to integrate the audio modality back into the fold without dragging down the overall system.

The Numbers Tell the Story

So, does it work? The results are mixed. On the MMEB benchmark, the model scores an impressive 74.9. Yet, on the 30-task MAEB audio suite, it clocks in at 55.61, revealing a performance gap that can't be ignored. The capex number is the real headline here, as it suggests significant investment for relatively uneven returns across modalities.

Are we witnessing the future of omni-modal retrieval, or is this just a step in a longer journey? The strategic bet is clearer than the street thinks. Conan-embedding-v3 may not be the perfect solution yet, but it's a key experiment in the quest for a unified digital world. Readers in AI development and tech fields should watch these developments closely. With each iteration, the dream of a smooth omni-modal experience inches closer to reality.