The modern data platform stack is maturing while AI shakes things up at the same time. Here is how LLMs touch each layer of the ten-layer stack — and what barely moves.
What is a modern data platform?
A modern data platform is the full path from raw data to useful insight. It pulls data in from databases, APIs, and streams. It shapes it into models you can use. It stores it in warehouses and lakehouses. It feeds dashboards, analysts, and ML models. And it governs the lot — who can see what, where it came from, and whether you can trust it.
Ten layers make up the stack:
- Ingestion — get data in (Kafka, Fivetran, Airbyte)
- Transform — shape and model it (dbt, Spark, Airflow)
- Storage — persist it (Snowflake, BigQuery, Iceberg, Delta)
- BI / visualisation — present it (Tableau, Looker, Power BI)
- Semantic — define metrics centrally (dbt metrics, Cube)
- Governance — control and document it (Unity Catalog, DataHub)
- ML / Analytics — model and analyse (MLflow, PyTorch, DuckDB)
- Security — protect it (masking, encryption, access control)
- Infra / Ops — run it (Terraform, Kubernetes)
- Observability — monitor it (Great Expectations, Monte Carlo)
Every large organisation runs some version of this. The tools vary — cloud-native, SaaS, open source — but the layers are universal.
Context: The Stack got quietly better
While everyone debated whether LLMs would replace data engineers, the modern data platform quietly built the foundation that makes AI useful:
- Open table formats won. Iceberg became the strategic default. Portable data means AI tools aren’t locked to one vendor.
- Catalogues became metadata APIs. Unity Catalog, Polaris, Dataplex — the context LLMs need to understand your data now lives in queryable systems.
- Semantic layers matured. dbt metrics, Cube, Snowflake Semantic Views — governed definitions AI can trust.
- Text-to-SQL went production. Snowflake Cortex Analyst, Databricks Genie, ThoughtSpot Sage — not demos, real features with 95% accuracy on governed datasets.
- Warehouses became AI runtimes. Snowflake Cortex AISQL, Databricks AI Functions — call LLMs from SQL. The warehouse isn’t just storage anymore.
- Local-first analytics emerged. DuckDB and Polars made laptop-scale prototyping viable. Teams can experiment with AI workflows before scaling.
- BI-as-code emerged. Evidence, Rill, Lightdash — dashboards as code that LLMs can generate and version-control.
- Lineage went portable. OpenLineage events flow across tools. AI can trace data from source to dashboard.
This foundation matters. LLMs don’t replace infrastructure — they query it. Better metadata means better AI.
The 10-Layer Stack: Before and After LLMs
Here is how each layer evolves:
| Layer | Current state | LLM evolution | Impact level |
|---|---|---|---|
| 1. Ingestion | Fivetran, Kafka, Airbyte, Debezium | AI agents generate connectors, handle schema drift | Supplemented |
| 2. Transform | dbt, Spark, Airflow, Dynamic Tables | Text-to-SQL, auto-generated models, natural language pipelines | Partially replaced |
| 3. Storage | Iceberg, Delta, Snowflake, BigQuery | Unchanged — still the foundation | No change |
| 4. BI / Viz | Looker, Tableau, Power BI, Metabase | Chat-based analytics replaces dashboards for exploration | Disrupted |
| 5. Semantic | dbt metrics, Cube, LookML | LLMs understand context natively; rigid definitions become optional | Disrupted |
| 6. Governance | Unity Catalog, DataHub, Collibra | Auto-classify PII, generate documentation, explain lineage | Supplemented |
| 7. ML / Analytics | MLflow, DuckDB, Polars, PyTorch | AI-assisted feature engineering, pipeline orchestration | Supplemented |
| 8. Security | Masking, encryption, access control | Anomaly detection, policy generation — humans still own keys | Supplemented |
| 9. Infra / Ops | Terraform, Kubernetes, monitoring | LLMs help write IaC, don’t replace it | No change |
| 10. Observability | Great Expectations, Monte Carlo, Soda | Root cause analysis in natural language, auto-fix suggestions | Supplemented |
Summary: 2 layers disrupted, 5 layers supplemented, 3 layers unchanged. Agents sit across those layers — there is a dedicated section after the layer-by-layer notes; they orchestrate tools and APIs and do not replace the foundation.
Layer-by-Layer: What Actually Changes
Current state: Choose between managed services (Fivetran, Zero-ETL) or build your own (Kafka, Debezium, Airbyte). How LLMs change it: What stays the same: You still need Kafka for streaming. You still need private connectivity. Operational complexity doesn’t disappear — it shifts from “build the connector” to “maintain the generated code.” The honest take: Managed ingestion isn’t dead. If you have 50 connectors and a small team, Fivetran still makes sense. But for straightforward sources with strong engineering teams, LLM-generated pipelines can shift the sums.Layer 1: Ingestion — AI-Assisted, Not AI-Replaced
Current state: dbt models, Spark jobs, Airflow DAGs, Dynamic Tables, declarative refresh. How LLMs change it: What stays the same: Complex business logic still needs human-authored, version-controlled SQL. dbt’s value isn’t the SQL — it’s testing, documentation, and lineage. AI-generated transforms still need governance. The honest take: The analyst who wrote 50 dbt models now writes 10. The rest get generated on demand or become unnecessary because users self-serve via natural language. The data engineer role shifts from “write SQL” to “design systems and review AI-generated code.”Layer 2: Transform — The Biggest Productivity Gain
Current state: Iceberg won the format war. Every vendor supports it. Catalogues (Unity Catalog, Polaris, Lake Formation) sit at the control plane. How LLMs change it: They don’t. You still need object storage. You still need Iceberg or Delta. You still need a catalogue for governance. LLMs query this layer — they don’t replace it. What matters more now: The better your metadata, the better LLM-powered analytics works. Invest in: The AI layer is only as good as the foundation it sits on.Layer 3: Storage — The Unchanging Foundation
Current state: Looker, Tableau, Power BI, Metabase, Superset. Self-serve exploration via drag-and-drop interfaces. How LLMs change it: The entire value proposition of BI tools was “explore data without writing SQL.” LLMs remove that barrier entirely. What survives: What dies: The 47-chart dashboard that took three sprints to build and nobody looks at. Self-serve exploration layers. The BI developer role as currently defined. The honest take: The landscape is splitting three ways: The winners won’t only be the tools with the slickest chat. They’ll be the ones where AI can build, change, and maintain dashboards in code — not just answer questions on top of them. This isn’t 5 years away. Snowflake Cortex claims 95% text-to-SQL accuracy on governed datasets. Kraken manages hundreds of Lightdash projects via CI/CD. The shift is happening now.Layer 4: BI / visualisation — the dashboard fades
Current state: dbt Semantic Layer, Cube, LookML, MetricFlow. Weeks spent defining what “revenue” means — handling edge cases, currency conversions, fiscal calendars. How LLMs change it: The semantic layer exists because machines couldn’t understand context. You had to spell out every business rule explicitly. LLMs understand context natively. What stays: The honest take: Tools like Cube and dbt’s semantic layer become optional for most queries. You’ll still want them for official metrics. But the tax of maintaining elaborate semantic models drops dramatically.Layer 5: Semantic — The Sleeper Disruption
Current state: Unity Catalog, DataHub, OpenMetadata, Collibra, Atlan. Manual cataloguing, PII tagging, documentation. How LLMs change it: What stays the same: The catalogue still matters — arguably more than ever. The data steward role shifts from “manually tag 10,000 columns” to “review and approve AI-generated classifications.” The honest take: Governance becomes easier, not optional. LLMs make the case for proper metadata stronger — the better your catalogue, the more useful AI becomes.Layer 6: Governance — AI-Powered, Human-Controlled
Current state: MLflow, feature stores, DuckDB, Polars, PyTorch, Feast, Evidently. How LLMs change it: What stays the same: PyTorch doesn’t go away. You still need MLflow, feature stores, and proper MLOps. The PhD who hand-crafted features now supervises an agent that proposes them.Layer 7: ML / Analytics — Copilots, Not Replacements
Current state: Column masking, row filters, dynamic masking, encryption, access control. How LLMs change it: What stays the same: Humans own the keys. “The AI gave me access” is not an acceptable audit trail. Security decisions require human oversight.Layer 8: Security — Smarter Detection, Human Control
Current state: Terraform, Kubernetes, Helm, ArgoCD, monitoring, cost management. How LLMs change it: LLMs help you write Terraform faster. They help you debug Kubernetes issues. They’re excellent pair programmers for infrastructure code. What stays the same: Kubernetes doesn’t go away. CI/CD, monitoring, cost management — all still needed. The pipes and plumbing remain.Layer 9: Infra / Ops — Better Tooling, Same Foundation
Current state: Great Expectations, Monte Carlo, Soda, Elementary, Lakehouse Monitoring. How LLMs change it: What stays the same: You need to detect the problem before you can explain it. Proper monitoring comes first.Layer 10: Observability — root cause in simple English
Agents: where they fit
Agents are not an eleventh layer — they cut across the ten. A single-turn chat answer is one pattern; an agent is closer to a bounded loop: plan → call tools (SQL, APIs, tickets, orchestrator triggers) → observe results → stop or escalate. The warehouse, formats, and orchestration underneath stay the same; the agent is the control surface that coordinates them.
Where they show up in practice:
- Ingestion and integration — scaffolding connectors, proposing mappings after schema drift, or driving repeatable “onboard this source” runbooks (humans still own production cutover).
- Transforms and analytics — multi-step workflows: clarify the question, run governed SQL, check row counts, produce a chart or narrative — not only one-shot text-to-SQL.
- Governance and catalogues — draft descriptions, suggest PII tags, queue human review (the “agentic catalog” pattern vendors are shipping).
- Observability — triage incidents, correlate failures, suggest fixes; execution and alerting stay in your existing tools.
- Orchestration — agents as a layer above Airflow/Dagster/Prefect: deciding when to trigger jobs or backfills from events and policies, not replacing the scheduler.
What stays non-negotiable: tool allowlists, session boundaries, audit trails, and human approval for anything sensitive. “The agent did it” is not a governance strategy.
Summary: chat lowers the floor for ad-hoc questions; agents raise the ceiling for repeatable, multi-step data work — still grounded in the same storage, semantics, and catalogues as before.
Build vs buy: what shifts
LLMs change what it costs to build things in-house versus pay a vendor:
| Before LLMs | After LLMs |
|---|---|
| Building connectors took weeks | LLMs draft configs in hours |
| Custom pipelines needed specialists | Natural language → working code |
| Managed services saved time | Generated code can be cheaper to run |
| Documentation was expensive | Often auto-generated from schema and usage |
The question isn’t only “which tool is best”. It’s what you’re paying for — and whether that still matches your team and sources.
For small teams with many connectors, managed services often still win. For strong engineering teams with simple sources, LLM-assisted code can tip the balance.
What Doesn’t Change
SQL still wins. Spark still scales. Airflow still runs (everyone still complains). The warehouse didn’t die — it learned to talk to lakes. Good modelling beats clever tooling. Security stays human.
What This Means
The stack is consolidating around fewer, more capable platforms:
- Warehouses absorbing everything — semantic layers, AI features, governance moving inside Snowflake/Databricks
- Chat with data going mainstream — Cortex Analyst, Databricks Genie, ThoughtSpot Sage, Claude rendering charts inline. The BI tool becomes a conversation.
- BI-as-code enabling AI-generated dashboards — Evidence, Rill, Lightdash turn dashboards into code artifacts that LLMs can write and maintain
- Orchestrators becoming control planes — Dagster and Airflow 3.0 pulling in catalogue, observability, cost management
- Catalogues going agentic — auto-documentation, auto-classification, auto-policy (Alation, Atlan, Informatica)
- Agents on top of the stack — planners with tools (SQL, orchestrator, tickets) for multi-step workflows; governance and allowlists matter more, not less
- Observability extending to AI — monitoring model inputs AND outputs, not just data pipelines
- Open formats winning — Iceberg as strategic default, OSI for semantic layer interoperability
The common thread: AI reduces manual work at every layer. Data engineers shift from building pipelines to governing AI-assisted pipelines.
What To Do Now
This article covers a lot. Here is how to act on it:
1. Audit your metadata readiness. LLM-powered analytics only works if the AI can understand your data. Check: Do your tables have column descriptions? Is PII classified? Can you trace lineage from source to dashboard? If not, start there — everything else depends on this foundation.
2. Run a chat-based analytics pilot. Pick one well-documented dataset and test Databricks Genie, Snowflake Cortex, or an open-source alternative against it. You’ll learn more in a week than from reading vendor whitepapers. The goal isn’t to replace BI — it’s to understand what works and what breaks.
3. Evaluate BI-as-code for your next dashboard. Before building another Tableau dashboard, try Evidence, Rill, or Lightdash. If your team can write SQL, they can write dashboards as code. The payoff: version control, CI/CD, and dashboards that LLMs can generate and modify.
4. Revisit your ingestion costs. That Fivetran contract made sense when building connectors took weeks. With LLM-assisted development, generating an Airbyte connector or Debezium config takes hours. Run the numbers again — especially for straightforward sources where you have engineering capacity.
5. Watch the orchestrator space. Airflow 3.0 shipped data assets. Dagster is pulling in catalogue and observability features. Your orchestrator is becoming your control plane. If you’re starting fresh or planning a migration, check whether your current tool will keep up.
6. Treat governance as AI infrastructure. Every investment in catalogues, semantic layers, and lineage pays off again when AI queries your data. The better your metadata, the better AI works. Governance isn’t overhead anymore — it’s the foundation that makes everything else possible.
7. If you pilot agents, scope them tightly. Pick one workflow (e.g. “monthly metrics pack” or “incident triage”) with explicit tools, a human-in-the-loop step, and logging. Agents amplify mistakes as well as speed — metadata and policies are your guardrails.
Closing Remarks
The modern data platform isn’t being replaced. It’s getting a new interface — and that interface speaks English. Agents add a second pattern: not only “ask once,” but “run a short, tool-backed workflow” with the same foundations underneath.
The biggest shift isn’t technical — it’s who can work with data. Natural language drops the barrier to zero. Data teams shift from “answer questions” to “build foundations that let anyone answer questions” — and, where it pays off, governed agent workflows on top.
The AI layer amplifies whatever foundation you’ve built. Better metadata, cleaner catalogues, governed metrics — these amplify everything you build on top. Weak foundations show up faster.
The stack isn’t static. Here is what is emerging across each layer — tools, features, and trends worth tracking in 2025–2026. Apache Paimon is the streaming-first table format to watch. Built from Flink Table Store, it uses LSM-tree architecture optimised for high-velocity writes with minute-level freshness — versus Iceberg’s hour-level recommended cadence. If you’re building streaming lakehouses, Paimon fills the gap between pure streaming (Kafka) and batch-optimised formats (Iceberg). Apache Fluss (incubating) takes it further: sub-second columnar streaming storage that tiers automatically into Iceberg or Paimon. Think of it as the “hot layer” for real-time data before it lands in your lakehouse. DuckLake (2025) rethinks metadata entirely — storing it in SQL databases instead of JSON/Avro manifests. Early days, but watch for simplicity gains. Iceberg v3 remains the strategic default. Every major vendor supports it now. The REST Catalog spec solved interoperability. Polaris Catalog (donated to Apache by Snowflake) is emerging as the open governance layer. The semantic layer war heated up in 2025: The pattern: semantic layers are moving from middleware to platform-native. Expect warehouse vendors to absorb this layer entirely. The native AI analytics features shipping now: These aren’t pilots anymore. They’re production features competing for how users interact with data. Monte Carlo launched Agent Observability (2025) — unifying data and AI observability to monitor both inputs and model outputs. Their Observability Agents can triage incidents and suggest fixes (though humans still execute). Also added unstructured data monitoring for GenAI pipelines. Emerging challengers: Sifflet, Metaplane, Sparvi offering lighter-weight alternatives. Elementary remains the dbt-native choice. Acceldata pushing enterprise hybrid environments. The trend: observability is extending to AI outputs, not just data inputs. Expect “LLM-as-judge” monitors and output quality scoring to become standard. AI is automating the grunt work: The shift: catalogues are becoming agentic. Instead of humans documenting data, agents propose descriptions, classify PII, and suggest policies. Humans review and approve. Airflow 3.0 (April 2025) was the biggest update in its history: Dagster continues gaining ground with asset-aware orchestration, built-in cataloguing, and 2x developer productivity claims vs. Airflow. The Components framework (GA October 2025) makes modular pipelines easier. Prefect remains the developer-friendly option with the simplest Python API. Kestra emerging for YAML-first, GitOps-style orchestration. The trend: orchestrators are absorbing catalogue, observability, and cost management features. The “orchestration platform” is becoming the “data control plane.” Worth watching in the broader ecosystem:Appendix: Watch list
Watch List: Emerging Players and Features
Storage and table formats
Semantic layer
Text-to-SQL and AI-BI
Data observability
Data governance and catalogues
Orchestration
AI-native infrastructure startups
References
Semantic layer and text-to-SQL
BI-as-code
Table formats and storage
Data observability
Data governance and catalogues
Orchestration
AI and data infrastructure