What Is DataHub? The Open-Source Metadata Platform for AI Data

DataHub is a third-generation metadata platform and AI data catalog originally built at LinkedIn to manage metadata at hyperscale.

It functions as a central nervous system for the modern data stack, providing a unified view of assets across fragmented environments.

A modern data stack spreads across warehouses, data lakes, business intelligence platforms, and ML systems, and each of those tools keeps its own private idea of what the data means. So the question that comes up most often — what breaks if I change this table? — has nowhere to be answered, because the answer depends on lineage (where a table came from and what depends on it) that no single tool can see end to end. DataHub connects those tools through a real-time streaming or batch ingestion framework so that whole picture lives in one unified metadata graph.

DataHub is built for developers, and you can tell from what it optimizes for: metadata you can act on rather than metadata you can read. Where a static catalog is passive documentation that somebody updates when they remember to, DataHub is a live system, and discovery, governance, and observability are all things that engineering teams and autonomous AI agents can query in real time.

DataHub's unified metadata graph connecting warehouses, data lakes, BI platforms and ML systems into a single management layer

What is DataHub?

DataHub is a model-first metadata platform, which is a formal way of saying the metadata model comes first and the systems plug into it rather than the other way around. More than 3,000 organizations run it today across millions of data assets. The "static" catalogs of earlier generations were documentation repositories, and that is the whole difference in one line: they described your data, while DataHub manages it as it changes.

It is open source under the Apache 2.0 license and stays vendor-neutral and community-driven. LinkedIn open-sourced it in 2020, and it has since become an independent project under the datahub-project GitHub organization, which matters more than the org name suggests, because the roadmap does not have to double as one vendor's upsell path. The architecture aims at a single foundation for data discovery, collaboration, governance, and end-to-end observability across everything that touches your data.

What does DataHub track about your data?

DataHub uses a metadata model built on "entities" and "aspects." In this model, an entity (like a dataset or dashboard) is the primary object, while an aspect is the atomic unit of metadata attached to it. That design keeps the model extensible, because you can attach new aspects to existing entities without breaking the core schema, and in practice it is the reason DataHub survives contact with a stack that gets rewired every quarter.

The "Context Foundation" is the map of how data moves through your organization, and it tracks:

Technical Metadata: Schemas, data types, and column-level lineage. For major warehouses like Snowflake and BigQuery, DataHub parses query logs to generate this lineage automatically, rather than requiring you to supply declared lineage manually.
Dataset Profiles: Row counts, total size, and last modified timestamps.
Usage Statistics: Query frequency, and who the heaviest users of a table actually are.
Data Quality: The quality metrics and assertions attached to an asset.
Governance Data: Ownership, documentation, and tags for classification, such as personally identifiable information (PII) markers.

How does DataHub work?

DataHub's architecture is stream-based and real-time, with a Kafka-based infrastructure underneath, so a metadata change is reflected across the platform within seconds. That is the part that makes the catalog usable while you are working rather than the morning after.

DataHub's stream-based architecture: sources push metadata through Kafka into the GMS service, which serves it back over GraphQL, REST and Avro APIs

The platform uses a "Schema-first" approach where the metadata model is described in a serialization-agnostic language. That supports several interaction methods, including REST, GraphQL, and an AVRO-based API over Kafka for subscribing to real-time changes. The ingestion framework ships over 80 production-grade connectors that support both push and pull models to extract deep metadata from sources like Snowflake, BigQuery, and dbt.

For organizations implementing a Data Mesh, where each domain team owns and publishes its own data instead of a central team owning everything, DataHub supports Federated Metadata Serving. This lets decoupled teams own and operate their own metadata services (GMS) while communicating with a central search index and graph via Kafka.

Getting a basic instance running locally to poke at takes one command:

bash

datahub docker quickstart

What does metadata do for AI systems?

DataHub supplies the context management that "Agentic AI" systems need. Without that context, Large Language Models (LLMs) have no way to tell whether a metric is trustworthy, where it originated, or what the business specifically means by it, and a confident answer built on the wrong revenue column is worse than no answer at all.

An AI agent querying DataHub's metadata graph over the Model Context Protocol to retrieve schema and lineage before acting

Key AI integrations include:

Model Context Protocol (MCP): This allows AI coding assistants like Cursor, Claude Desktop, or Cline to query DataHub metadata directly, so the assistant understands the data context while it writes code. If the protocol is new to you, see what MCP is.
Open Source Analytics Agent: An agent grounded in your DataHub catalog that allows users to ask data questions in plain English and receive SQL, results, and charts.
Context for Reliability: Metadata fills the "missing piece" for reliable autonomous agents by telling the agent what a metric means and whether it is safe to use, based on the governance policies you have already written down.

What does running DataHub yourself actually cost?

Self-hosting DataHub costs hardware first and people second, and the second bill is the one that surprises teams. The hardware fits in a line: a basic local Docker setup requires a minimum of 2 CPUs, 8GB of RAM, a 2GB Swap area, and 13GB of disk space. The people are the expensive half, because running DataHub properly consumes between a quarter and a full platform engineer's time to handle upgrades, connector repairs, and access requests. That is the figure I would put in front of whoever approves the project, because these are the "Month Six" costs that never show up in a proof of concept:

The Upgrade Nobody Owns: Release cycles are frequent. Skipping them creates technical debt that eventually requires a complex, high-risk migration project.
Broken Connectors: Upstream warehouse updates or API changes can silently break metadata ingestion, and your team is the one that has to debug and patch the connectors.
Key Person Risk: Deployment often depends on a single engineer who understands the specific wiring and patches. If they leave, the catalog often decays into an untrusted, out-of-date state.

When is DataHub the right choice — and when is it not?

DataHub fits engineering-led teams with a single warehouse, and it fits them even better if they intend to build custom, metadata-driven systems on top of a flexible, extensible model, with automated access control usually the first one anybody builds. For teams that already manage their own Kubernetes infrastructure, the engineering autonomy is real and they will use it.

However, the open-source version is often the wrong choice when external regulators — such as MAS (Singapore), APRA (Australia), or OJK (Indonesia) — require formal evidence of governance. While the open-source version holds the metadata, it lacks the "evidence layer" that audits require, such as immutable approval history, retained access records, and formal audit logs. In those regulated environments, building those features by hand costs more than the commercial platform you were trying not to pay for, and that is not a lesson worth spending a year to learn.