NVIDIA’s NV-Embed Model Achieves Top Spot on MTEB Leaderboard

NVIDIA's NV-Embed Model Achieves Top Spot on MTEB Leaderboard

NVIDIA’s latest embedding model, NV-Embed, has set a new record for embedding accuracy with a score of 69.32 on the Massive Text Embedding Benchmark (MTEB), which encompasses 56 diverse embedding tasks, according to NVIDIA Technical Blog.

Understanding the Metrics for Embedding Models

Embedding models are evaluated using several metrics, with Normalized Discounted Cumulative Gain (NDCG) and Recall being the most significant. NDCG is a rank-aware metric that measures the relevance and order of retrieved information, while Recall is a rank-agnostic metric that measures the percentage of relevant results retrieved. Most benchmarks report NDCG@10, but for enterprise-grade retrieval-augmented generation (RAG) pipelines, Recall@5 is often recommended.

What are MTEB and BEIR?

The MTEB benchmark covers 56 tasks, including retrieval, classification, re-ranking, clustering, and summarization. BEIR focuses on retrieval tasks and adds complexity with different question types and domains, such as fact-checking and biomedical questions. MTEB is largely a superset of BEIR, making it a comprehensive benchmark for evaluating embedding models.

NV-Embed’s performance on MTEB has been exceptional, achieving an NDCG@10 score of 69.32, the highest among all models tested. This performance is attributed to several key improvements in the model’s architecture and training process.

Key Improvements in NV-Embed

  • Latent Attention Layer: This new layer simplifies the process of combining the mathematical representation (embeddings) of a series of words, improving the model’s efficiency and accuracy.
  • Two-Stage Learning Process: The first stage uses in-batch negative and hard negative pairs for contrastive learning, while the second stage blends data from non-retrieval tasks for further training, enhancing the model’s robustness.

These advancements make NV-Embed a powerful tool for enterprise retrieval workloads, although its effectiveness depends on the nature and domain of the data being used.

Prototyping with NV-Embed

NV-Embed is available through NVIDIA’s API catalog, allowing organizations to integrate this high-performing model into their data processing pipelines. Additionally, the NVIDIA NeMo Retriever collection of microservices enables seamless connection of custom models to diverse business data, delivering highly accurate responses.

For further details, visit the NVIDIA Technical Blog.

Image source: Shutterstock

. . .


Source link

Related Articles

Do you run a company that want to build a new website and are looking for a web agency in Sweden that can do the job? At Partna you can get connected to experienced web agencies that are interested in helping you with your website development. Partna is an online service where you simply post your web development needs in order to get business offers from skilled web agencies in Sweden. Instead of reaching out to hundreds of agencies by yourself, let up to 5 web agencies come to you via Partna.
Back to top button