What is Milvus? The Open-Source Vector Database for AI

Milvus is a high-performance, open-source vector database built to manage and search unstructured data through dense and sparse embeddings.

When you are designing systems for billion-scale AI applications, the challenge is rarely just storage; it is maintaining millisecond-level retrieval across massive datasets. Milvus solves this with a specialized storage and retrieval layer that organizes embeddings generated by machine learning models for Retrieval-Augmented Generation (RAG), recommendation systems, and multi-modal search.

Milvus is written in Go and C++, a design choice that pairs high-level system orchestration with a low-level, hardware-optimized search engine. As a graduated project within the LF AI & Data Foundation, Milvus targets production environments requiring high availability and horizontal scalability. It connects unstructured data — images, text, or audio — to numbers your systems can act on.

If you are operating at the enterprise level, the performance bottleneck in vector search usually sits in the search engine's ability to use modern hardware. Milvus addresses this with SIMD instructions, GPU acceleration, and a cloud-native architecture that decouples storage from compute, letting you scale your query capacity independently of your data ingestion needs.

Unstructured data such as text, images and video turned into vectors and searched by similarity at billion scale

What is Milvus?

The Milvus project began at Zilliz in 2017 to solve the scaling limitations of existing approximate nearest neighbor (ANN) libraries. It joined the Linux Foundation's incubation program in January 2020 and graduated from the LF AI & Data Foundation in June 2021. The name "Milvus" refers to a genus of birds of prey, chosen to signal speed in flight, keen vision — search accuracy and recall — and adaptability across environments.

The search engine determines over 80% of a vector database's performance, and Milvus writes that component in C++. This core integrates hardware-aware optimizations such as assembly-level vectorization for AVX512 and SIMD instructions, alongside GPU indexing through NVIDIA's CAGRA. By treating vector search as a primary concern rather than an add-on, Milvus achieves significantly higher throughput than general-purpose databases that use secondary vector indices.

Milvus is a column-oriented database. That is a critical design choice for distributed systems because it reduces data access overhead — the system reads only the fields a query needs rather than whole rows. The layout also lets operations vectorize across columns, which is what keeps search latency low when your data sits on high-latency object storage.

The project's maturity shows in its adoption among contributors from the high-performance computing (HPC) community, including experts from NVIDIA, Intel, and AMD. This community-driven focus keeps Milvus compatible with the latest hardware advances, from NVMe SSD optimizations to the newest GPU architectures.

The four-layer architecture

Milvus uses a cloud-native architecture that completely disaggregates storage and compute. This modularity lets you scale query nodes (for read-heavy workloads) and data nodes (for write-heavy workloads) independently, preventing ingestion spikes from degrading your search p99 latencies.

The four layers of Milvus stacked top to bottom: Access Layer, Coordinator, Worker Nodes and Storage Layer

The Access Layer is the system's entry point, a set of stateless proxies. These proxies handle client request validation, load balancing, and result reduction. Since Milvus uses a massively parallel processing (MPP) architecture, the proxy aggregates intermediate results from across the cluster and performs the final reduction before returning the data to the client.

The Coordinator is the cluster's brain. At any point, exactly one coordinator is active, managing cluster topology, task scheduling, and cluster-level consistency. It handles Data Definition Language (DDL) and Data Control Language (DCL) tasks, such as creating collections or managing access control, and it is the Timestamp Oracle (TSO) that maintains temporal ordering across the distributed worker nodes.

Worker Nodes are the stateless executors of the system. Streaming Nodes manage the shard-level "mini-brain," handling growing data and ensuring consistency; Query Nodes manage historical data queries by loading segments from object storage; and Data Nodes perform offline background tasks like compaction and index building. This separation ensures that your indexing and data optimization tasks do not compete for resources with your live search queries.

The Storage Layer is the foundation of data persistence, divided into three parts: meta storage (using etcd) for collection schemas, object storage (S3, MinIO, or Azure Blob) for data and index files, and the Write-Ahead Log (WAL). Milvus uses Woodpecker for its WAL — a zero-disk, cloud-native log service that writes directly to object storage. Unlike traditional disk-based logs like Kafka or Pulsar, Woodpecker removes local disk dependencies, which simplifies cluster operations and recovery.

How Milvus indexes vectors

A Milvus vector index has three parts: the data structure, quantization, and the refiner. You choose between graph-based structures (HNSW), which offer high QPS and low latency, and Inverted File (IVF) variants, which fit high-throughput workloads or very large top-K requirements.

The three parts of a Milvus vector index — data structure, quantization and the Refiner — alongside a contrast of IVF and HNSW

Quantization methods like SQ8 and PQ compress embeddings to reduce the memory footprint. SQ8 reduces usage by 75% by compressing 32-bit floats into 8-bit integers. To compensate for the loss of precision during quantization, the Refiner (using FP32 precision) recalculates distances for the candidate set, keeping recall high despite the compressed index structure.

When estimating memory for an HNSW index, you must account for both the raw embeddings and the graph structure. Each vector in an HNSW index maintains connections to its neighbors. With a graph degree of 32, the graph requires approximately 128 MB for 1 million vectors, because each of the 32 links requires 4 bytes for 32-bit integer storage of neighbor IDs. Combined with the 512 MB required for raw 128-dimensional FP32 embeddings, the total usage is 640 MB. Using HNSW_PQ reduces the embedding portion to 8 MB, dropping the total footprint to 136 MB.

You prepare index parameters with the Python SDK. You can configure the M and efConstruction parameters to balance build time against search accuracy:

python

index_params = client.prepare_index_params()
index_params.add_index(
    field_name="vector_field",
    index_type="HNSW",
    metric_type="COSINE",
    params={"M": 32, "efConstruction": 256}
)

Three deployment modes: Lite, Standalone, Distributed

Milvus supports three deployment modes, so you can move from a notebook to a global cluster without changing your application code.

Three Milvus deployment modes by increasing data scale: Lite on a local file, Standalone on a single Docker host and Distributed on Kubernetes

Milvus Lite is a lightweight Python library designed for edge devices or local prototyping. It suits Jupyter Notebooks and handles up to a few million vectors. It requires no external dependencies or message queues, persisting data directly to a local file. This is the fastest way to get a RAG prototype running without infrastructure overhead.

Milvus Standalone bundles all components into a single Docker image, which makes it suitable for medium-scale production up to 100 million vectors. It uses an embedded version of Woodpecker as its message queue, so there is no separate Kafka or Pulsar service to manage on a single machine.

Milvus Distributed is the Kubernetes-native choice, handling datasets from 100 million up to tens of billions of vectors. This mode lets every component scale out as an independent microservice, which gives you the highest availability and resource efficiency. In this configuration, Woodpecker runs as a dedicated service to handle the Write-Ahead Log across the entire cluster.

Initializing the client is identical across modes — only the URI changes to point from your local environment to your production cluster:

python

# For Milvus Lite (local file)
client_lite = MilvusClient("./milvus_demo.db")
 
# For Standalone or Distributed (remote server)
client_remote = MilvusClient(
    uri="http://localhost:19530",
    token="user:password"
)

What changed in Milvus 3.0

Milvus 3.0 is a major shift toward a lake-native architecture. External Collections let you search data that sits directly in Iceberg, Parquet, or Lance formats on S3 or GCS without moving it. This ends the data-silo problem by letting Milvus build and serve indices over data that stays in your lake.

The lake-native architecture of Milvus 3.0 querying data directly in object storage instead of copying it through an ETL pipeline

The storage engine is now Loon (also known as Storage v3), a manifest-based columnar engine that uses the Vortex format. Vortex is optimized for the point-read patterns common in AI retrieval. In one internal benchmark using 3 million rows, 128-dimensional vectors, S3, and 256 concurrent readers, I/O per point read fell from about 9.4 MB for the Parquet baseline to 0.07 MB — roughly 135 times less.

The retrieval engine now handles complex post-processing with server-side ORDER BY and faceted search. For late-interaction models like ColBERT or ColPali, the StructArray type lets you store variable-length arrays of vectors in a single row. You can run entity-level search with the MAX_SIM and MAX_SIM_COSINE metrics, which sum the best match scores for every query token, keeping your document-to-embedding relationships intact.

Sparse retrieval has been overhauled with SINDI, an algorithm built for learned sparse embeddings like SPLADE. SINDI organizes postings into compact, SIMD-friendly windows, reaching up to about 10x the QPS of MaxScore across four SPLADE datasets, with a worst case around 5x. On one set of internal BM25 benchmarks it was roughly 3 times smaller than the Milvus 2.6 sparse index at comparable recall.

Milvus vs Pinecone, Qdrant, and Weaviate

Choosing a vector database in 2026 depends on your scale and latency requirements. Pinecone is a strong zero-ops choice for smaller datasets, but it carries higher costs at high query volumes and shows p50 latencies in the 20–30ms range. Qdrant, written in Rust, posts the lowest p50 latency at 4ms and offers excellent filtering, but it lacks the distributed architecture needed to scale effectively once you exceed 100 million vectors.

A comparison of four vector databases — Milvus, Pinecone, Qdrant and Weaviate — by data scale and p50 latency

Weaviate suits teams that need mature BM25 hybrid search and easy LLM agent integration via the Model Context Protocol (MCP). For deployments running from 100 million into billions of vectors, though, Milvus remains the primary choice, thanks to its Kubernetes-native microservices and deep hardware acceleration.

Milvus posts a 6ms p50 latency with GPU-accelerated indexing. At billion-scale, its cost efficiency comes from RaBitQ, a 1-bit quantization technique that compresses indices to 1/32 their original size at 95% recall, letting you store massive indices with a fraction of the RAM competitors need.

When you are designing for billion-scale, scaling query and data nodes independently while using RaBitQ compression is what makes Milvus economically viable for high-volume production.

When should you choose Milvus?

Choose Milvus if you are operating at 100 million vectors or more, or if your application needs the sub-10ms latencies that only GPU hardware acceleration provides. It is the right call for teams that need to index data directly in a data lake and avoid the overhead and synchronization risks of ETL pipelines.

If you are already standardized on Kubernetes and need to replace a fragmented stack — Elasticsearch for text plus a separate vector database for embeddings — Milvus gives you one integrated system. But if you are working with a few thousand vectors in a simple script, you are over-engineering: Milvus is built for throughput you do not have yet.