<div id="readability-page-1" class="page"><div><img src="https://devstarsj.github.io/assets/https://images.unsplash.com/photo-1460925895917-afdab827c52f?w=1200">The data warehouse is not dead. But it is unrecognizable.What used to be a monolithic proprietary system — an island of transformed data, accessible only through a specific query engine, locked to one vendor — has been decomposed. The storage layer is now separate from compute. The table format is open. The transformation logic is version-controlled. The catalog is federated.In 2026, the Modern Data Stack has matured into something more principled and more complex than the buzzword-heavy version we discussed in 2022. Let me break down what it actually looks like.<img src="https://images.unsplash.com/photo-1551288049-bebda4e38f71?w=800" alt="Abstract data visualization with flowing lines and nodes representing data connections"> Photo by Luke Chesser on Unsplash<hr><h2 id="the-lakehouse-architecture-what-it-means-in-practice">The Lakehouse Architecture: What It Means in Practice</h2>The “lakehouse” concept — combining the scale and openness of a data lake with the reliability and query performance of a data warehouse — has become the de facto architecture for companies with serious data workloads.The enabling technology: open table formats, most importantly Apache Iceberg.<h3 id="why-apache-iceberg-won">Why Apache Iceberg Won</h3>Three formats competed for the open table format standard: Delta Lake (Databricks), Apache Hudi (Uber), and Apache Iceberg (Netflix/Apple). By mid-2025, Iceberg had achieved decisive ecosystem dominance.What Iceberg provides:<div><pre><code>Raw Storage (S3/GCS/ADLS)
 └── Parquet/ORC data files
 └── Iceberg metadata layer
 ├── Catalog (table names → metadata pointers)
 ├── Metadata files (schema, partition info, snapshots)
 └── Manifest files (file lists with statistics)
</code></pre></div>ACID transactions on object storage: Multiple writers can commit without corrupting data. Time travel is built in. Schema evolution is safe.Query engine independence: The same Iceberg table can be queried by Spark, Trino, DuckDB, Snowflake, BigQuery, StarRocks, or any other Iceberg-compatible engine. You’re not locked to any single query engine.Partition evolution: Change how data is partitioned without rewriting all historical data.<div><pre><code>-- Create an Iceberg table
CREATE TABLE events (
 id BIGINT,
 user_id BIGINT,
 event_type STRING,
 properties MAP&lt;STRING, STRING&gt;,
 created_at TIMESTAMP
)
USING ICEBERG
PARTITIONED BY (days(created_at));

-- Time travel: query data as of 7 days ago
SELECT * FROM events
FOR SYSTEM_TIME AS OF TIMESTAMP '2026-02-28 00:00:00';

-- Evolve the partition strategy without rewriting data
ALTER TABLE events
REPLACE PARTITION FIELD days(created_at) WITH hours(created_at);
</code></pre></div>The result: you can store 10 years of raw event data in S3 at $23/TB/month and query it from any engine without vendor lock-in.<hr>dbt (data build tool) started as a way to write SQL transformations that get version-controlled. It has evolved into something closer to a full data development platform.<h3 id="what-dbt-does">What dbt Does</h3><div><pre><code>Raw source data (Iceberg/warehouse)
 └── dbt models (SELECT statements in .sql files)
 ├── Staging models (rename, cast, clean)
 ├── Intermediate models (joins, business logic)
 └── Mart models (final analytics tables)
 └── Tests, documentation, lineage
</code></pre></div>Each model is a SELECT statement. dbt compiles it to a CREATE TABLE or CREATE VIEW statement and handles the dependency graph, incremental loading, and test execution.<div><pre><code>-- models/staging/stg_events.sql
SELECT
 id,
 user_id,
 event_type,
 properties:page_url::STRING AS page_url,
 created_at::TIMESTAMP AS occurred_at
FROM 
WHERE created_at &gt;= '2020-01-01' -- Skip ancient garbage data

-- models/marts/user_activity_daily.sql
WITH daily_events AS (
 SELECT
 user_id,
 DATE_TRUNC('day', occurred_at) AS activity_date,
 COUNT(*) AS event_count,
 COUNT(DISTINCT event_type) AS unique_event_types
 FROM 
 GROUP BY 1, 2
)
SELECT
 u.user_id,
 u.email,
 d.activity_date,
 d.event_count,
 d.unique_event_types
FROM daily_events d
JOIN u USING (user_id)
</code></pre></div><h3 id="dbts-new-capabilities">dbt’s New Capabilities</h3>dbt Mesh: Teams can define cross-project dependencies, allowing large organizations to break their monolithic dbt project into domain-owned sub-projects while maintaining shared models.<div><pre><code># In a consuming project — reference a model from another project
models:
 - name: user_lifetime_value
 columns:
 - name: user_id
 data_tests:
 - not_null
 - relationships:
 to: ref('platform', 'dim_users') # Cross-project reference
 field: id
</code></pre></div>dbt + AI (dbt Copilot): Write a description of what you want, get a dbt model. It’s genuinely useful for standard patterns — aggregations, joins, slowly-changing dimensions. Less useful for complex business logic.Semantic Layer: Define metrics once in dbt, query them from any BI tool. The metric definition lives in code, not scattered across five different dashboard tools.<div><pre><code># models/metrics/revenue_metrics.yml
metrics:
 - name: monthly_recurring_revenue
 label: MRR
 description: "Monthly recurring revenue from active subscriptions"
 model: ref('fct_subscriptions')
 calculation_method: sum
 expression: mrr_amount
 timestamp: subscription_date
 time_grains: [month, quarter, year]
 dimensions:
 - plan_type
 - country
 - acquisition_channel
</code></pre></div><hr><h2 id="the-streaming-lakehouse-near-real-time-analytics">The Streaming Lakehouse: Near-Real-Time Analytics</h2>The traditional batch-processed data warehouse had a fundamental limitation: data was always hours or days old. The streaming lakehouse closes that gap.<img src="https://images.unsplash.com/photo-1518770660439-4636190af475?w=800" alt="Digital stream of glowing data particles flowing through a dark tunnel"> Photo by Alexandre Debiève on UnsplashThe architecture:<div><pre><code>Events → Kafka/Kinesis
 → Flink/Spark Streaming (transforms)
 → Iceberg tables (with streaming writes)
 → Query engines (seconds of latency)
</code></pre></div>Apache Flink has become the standard streaming processor for this pattern. Its Iceberg sink allows streaming writes while maintaining ACID semantics:<div><pre><code># Flink → Iceberg streaming write
from pyflink.table import TableEnvironment

env = TableEnvironment.create(...)

# Read from Kafka
env.execute_sql("""
 CREATE TABLE kafka_events (
 user_id BIGINT,
 event_type STRING,
 event_time TIMESTAMP(3),
 WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
 ) WITH (
 'connector' = 'kafka',
 'topic' = 'user-events',
 'format' = 'json'
 )
""")

# Write to Iceberg with streaming semantics
env.execute_sql("""
 CREATE TABLE iceberg_events (
 user_id BIGINT,
 event_type STRING,
 event_time TIMESTAMP(3)
 ) WITH (
 'connector' = 'iceberg',
 'catalog-name' = 'prod_catalog',
 'catalog-database' = 'events',
 'catalog-table' = 'user_events'
 )
""")

# Stream from Kafka to Iceberg
env.execute_sql("INSERT INTO iceberg_events SELECT * FROM kafka_events")
</code></pre></div>The result: data lands in queryable Iceberg tables within 30-60 seconds of the originating event. For analytics use cases, that’s effectively real-time.<hr><h2 id="the-query-engine-landscape">The Query Engine Landscape</h2>With open table formats, you can choose your query engine independently of your storage. The current landscape:<table><thead><tr><th>Engine</th><th>Best For</th><th>Tradeoffs</th></tr></thead><tbody><tr><td>Trino/Presto</td><td>Ad-hoc queries, data exploration</td><td>Complex to operate, great horizontal scale</td></tr><tr><td>DuckDB</td><td>Local analysis, single-machine</td><td>Single node, excellent for laptop/small server</td></tr><tr><td>Spark</td><td>Batch processing, large-scale ETL</td><td>Heavy, high operational overhead</td></tr><tr><td>Snowflake</td><td>Managed warehouse, SQL analytics</td><td>Expensive at scale, proprietary</td></tr><tr><td>StarRocks</td><td>Real-time analytics, dashboards</td><td>Best query speed for dashboard workloads</td></tr><tr><td>ClickHouse</td><td>Time-series, log analytics</td><td>Column-oriented, extremely fast for aggregations</td></tr></tbody></table>DuckDB deserves special attention. It has become the standard for local data work — you can query Iceberg tables on S3 directly from your laptop with zero infrastructure:<div><pre><code>import duckdb

# Query Iceberg on S3 from your laptop — no servers needed
conn = duckdb.connect()
conn.execute("INSTALL iceberg; LOAD iceberg;")

result = conn.execute("""
 SELECT
 event_type,
 COUNT(*) as event_count,
 APPROX_COUNT_DISTINCT(user_id) as unique_users
 FROM iceberg_scan('s3://my-data-lake/events/user_events/')
 WHERE created_at &gt;= CURRENT_DATE - INTERVAL 7 DAYS
 GROUP BY event_type
 ORDER BY event_count DESC
""").fetchdf()
</code></pre></div><hr><h2 id="ai-in-the-data-stack-whats-real">AI in the Data Stack: What’s Real</h2>The most overhyped and most misunderstood component of the modern data stack is AI. Let me separate signal from noise.Real and useful:<ul><li>Text-to-SQL: Natural language queries that get translated to SQL. Works well for standard analytical queries. Requires good catalog documentation (dbt docs help enormously).</li><li>Anomaly detection: ML models that flag unusual patterns in metrics. Genuinely catches issues humans would miss.</li><li>Data quality: AI-assisted schema inference, duplicate detection, PII identification in new datasets.</li><li>Documentation generation: AI-written column descriptions and table summaries, seeded from sample data and column names.</li></ul>Not ready for production:<ul><li>Fully autonomous data pipelines that “figure out the schema and transformations by themselves”</li><li>AI that replaces data engineers for complex transformation logic</li><li>Natural language as the primary interface for business analysts (most still prefer dashboards)</li></ul>The practical recommendation: use AI to accelerate data engineers, not to replace them. Text-to-SQL for exploration, AI-generated docs to fill the documentation gap, anomaly detection to catch data quality issues early.<hr><h2 id="the-stack-we-recommend-in-2026">The Stack We Recommend in 2026</h2>For a company processing 100GB-10TB/day:<table><thead><tr><th>Layer</th><th>Tool</th><th>Why</th></tr></thead><tbody><tr><td>Ingestion</td><td>Airbyte (managed) or Fivetran</td><td>300+ connectors, minimal maintenance</td></tr><tr><td>Streaming</td><td>Kafka + Flink</td><td>Standard, large ecosystem</td></tr><tr><td>Storage</td><td>S3 + Apache Iceberg</td><td>Open, cheap, portable</td></tr><tr><td>Catalog</td><td>Project Nessie or AWS Glue</td><td>Git-like branching for data</td></tr><tr><td>Transformation</td><td>dbt Core</td><td>Version-controlled, testable SQL</td></tr><tr><td>Orchestration</td><td>Airflow or Dagster</td><td>Dagster for better observability</td></tr><tr><td>Query Engine</td><td>Trino (shared) + DuckDB (local)</td><td>Best cost-to-performance ratio</td></tr><tr><td>BI/Visualization</td><td>Metabase or Superset</td><td>Open-source; Looker if budget allows</td></tr></tbody></table>Total infrastructure cost for this stack at medium scale: roughly $3,000-8,000/month on AWS, compared to $25,000-50,000/month for equivalent Snowflake + Fivetran + Looker.<hr><h2 id="the-bottom-line">The Bottom Line</h2>The Modern Data Stack in 2026 is both more powerful and more complex than it was three years ago. Open table formats have delivered on their promise of vendor independence. dbt has matured from a transformation tool into a data development platform. The streaming lakehouse has closed the gap between real-time and batch analytics.The teams that are winning have three things in common:<ol><li>They invested in data quality and testing from day one (dbt tests, Great Expectations)</li><li>They treat data as a product with SLAs and ownership, not a byproduct of engineering work</li><li>They resist adding new tools until they’ve exhausted what the existing tools can do</li></ol>The data stack is rich enough now that almost any analytical problem can be solved with open-source tools. The limiting factor is no longer technology — it’s discipline, data culture, and the unglamorous work of keeping pipelines healthy and data trustworthy.That part, no amount of tooling will automate away. <div>이 글이 도움이 되셨다면 공감 및 광고 클릭을 부탁드립니다 :) </div></div></div>

Dev Note

The data warehouse is not dead. But it is unrecognizable.
What used to be a monolithic proprietary system — an island of transformed data, accessible only through a specific query engine, locked to one vendor — has been decomposed. The storage layer is now separate from compute. The table format is open. The transformation logic is version-controlled. The catalog is federated.
In 2026, the Modern Data Stack has matured into something more principled and more complex than the buzzword-heavy version we discussed in 2022. Let me break down what it actually looks like.
Photo by Luke Chesser on Unsplash
The Lakehouse Architecture: What It Means in Practice
The “lakehouse” concept — combining the scale and openness of a data lake with the reliability and query performance of a data warehouse — has become the de facto architecture for companies with serious data workloads.
The enabling technology: open table formats, most importantly Apache Iceberg.
Why Apache Iceberg Won
Three formats competed for the open table format standard: Delta Lake (Databricks), Apache Hudi (Uber), and Apache Iceberg (Netflix/Apple). By mid-2025, Iceberg had achieved decisive ecosystem dominance.
What Iceberg provides:
Raw Storage (S3/GCS/ADLS)
 └── Parquet/ORC data files
 └── Iceberg metadata layer
 ├── Catalog (table names → metadata pointers)
 ├── Metadata files (schema, partition info, snapshots)
 └── Manifest files (file lists with statistics)
ACID transactions on object storage: Multiple writers can commit without corrupting data. Time travel is built in. Schema evolution is safe.
Query engine independence: The same Iceberg table can be queried by Spark, Trino, DuckDB, Snowflake, BigQuery, StarRocks, or any other Iceberg-compatible engine. You’re not locked to any single query engine.
Partition evolution: Change how data is partitioned without rewriting all historical data.
-- Create an Iceberg table
CREATE TABLE events (
 id BIGINT,
 user_id BIGINT,
 event_type STRING,
 properties MAP<STRING, STRING>,
 created_at TIMESTAMP
)
USING ICEBERG
PARTITIONED BY (days(created_at));
-- Time travel: query data as of 7 days ago
SELECT * FROM events
FOR SYSTEM_TIME AS OF TIMESTAMP '2026-02-28 00:00:00';
-- Evolve the partition strategy without rewriting data
ALTER TABLE events
REPLACE PARTITION FIELD days(created_at) WITH hours(created_at);
The result: you can store 10 years of raw event data in S3 at $23/TB/month and query it from any engine without vendor lock-in.
dbt in 2026: From Transformation Tool to Data Platform
dbt (data build tool) started as a way to write SQL transformations that get version-controlled. It has evolved into something closer to a full data development platform.
What dbt Does
Raw source data (Iceberg/warehouse)
 └── dbt models (SELECT statements in .sql files)
 ├── Staging models (rename, cast, clean)
 ├── Intermediate models (joins, business logic)
 └── Mart models (final analytics tables)
 └── Tests, documentation, lineage
Each model is a SELECT statement. dbt compiles it to a CREATE TABLE or CREATE VIEW statement and handles the dependency graph, incremental loading, and test execution.
-- models/staging/stg_events.sql
SELECT
 id,
 user_id,
 event_type,
 properties:page_url::STRING AS page_url,
 created_at::TIMESTAMP AS occurred_at
FROM 
WHERE created_at >= '2020-01-01' -- Skip ancient garbage data
-- models/marts/user_activity_daily.sql
WITH daily_events AS (
 SELECT
 user_id,
 DATE_TRUNC('day', occurred_at) AS activity_date,
 COUNT(*) AS event_count,
 COUNT(DISTINCT event_type) AS unique_event_types
 FROM 
 GROUP BY 1, 2
)
SELECT
 u.user_id,
 u.email,
 d.activity_date,
 d.event_count,
 d.unique_event_types
FROM daily_events d
JOIN u USING (user_id)
dbt’s New Capabilities
dbt Mesh: Teams can define cross-project dependencies, allowing large organizations to break their monolithic dbt project into domain-owned sub-projects while maintaining shared models.
# In a consuming project — reference a model from another project
models:
 - name: user_lifetime_value
 columns:
 - name: user_id
 data_tests:
 - not_null
 - relationships:
 to: ref('platform', 'dim_users') # Cross-project reference
 field: id
dbt + AI (dbt Copilot): Write a description of what you want, get a dbt model. It’s genuinely useful for standard patterns — aggregations, joins, slowly-changing dimensions. Less useful for complex business logic.
Semantic Layer: Define metrics once in dbt, query them from any BI tool. The metric definition lives in code, not scattered across five different dashboard tools.
# models/metrics/revenue_metrics.yml
metrics:
 - name: monthly_recurring_revenue
 label: MRR
 description: "Monthly recurring revenue from active subscriptions"
 model: ref('fct_subscriptions')
 calculation_method: sum
 expression: mrr_amount
 timestamp: subscription_date
 time_grains: [month, quarter, year]
 dimensions:
 - plan_type
 - country
 - acquisition_channel
The Streaming Lakehouse: Near-Real-Time Analytics
The traditional batch-processed data warehouse had a fundamental limitation: data was always hours or days old. The streaming lakehouse closes that gap.
Photo by Alexandre Debiève on Unsplash
The architecture:
Events → Kafka/Kinesis
 → Flink/Spark Streaming (transforms)
 → Iceberg tables (with streaming writes)
 → Query engines (seconds of latency)
Apache Flink has become the standard streaming processor for this pattern. Its Iceberg sink allows streaming writes while maintaining ACID semantics:
# Flink → Iceberg streaming write
from pyflink.table import TableEnvironment
env = TableEnvironment.create(...)
# Read from Kafka
env.execute_sql("""
 CREATE TABLE kafka_events (
 user_id BIGINT,
 event_type STRING,
 event_time TIMESTAMP(3),
 WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
 ) WITH (
 'connector' = 'kafka',
 'topic' = 'user-events',
 'format' = 'json'
 )
""")
# Write to Iceberg with streaming semantics
env.execute_sql("""
 CREATE TABLE iceberg_events (
 user_id BIGINT,
 event_type STRING,
 event_time TIMESTAMP(3)
 ) WITH (
 'connector' = 'iceberg',
 'catalog-name' = 'prod_catalog',
 'catalog-database' = 'events',
 'catalog-table' = 'user_events'
 )
""")
# Stream from Kafka to Iceberg
env.execute_sql("INSERT INTO iceberg_events SELECT * FROM kafka_events")
The result: data lands in queryable Iceberg tables within 30-60 seconds of the originating event. For analytics use cases, that’s effectively real-time.
The Query Engine Landscape
With open table formats, you can choose your query engine independently of your storage. The current landscape:
Engine
 Best For
 Tradeoffs
 
Trino/Presto
 Ad-hoc queries, data exploration
 Complex to operate, great horizontal scale
 
DuckDB
 Local analysis, single-machine
 Single node, excellent for laptop/small server
 
Spark
 Batch processing, large-scale ETL
 Heavy, high operational overhead
 
Snowflake
 Managed warehouse, SQL analytics
 Expensive at scale, proprietary
 
StarRocks
 Real-time analytics, dashboards
 Best query speed for dashboard workloads
 
ClickHouse
 Time-series, log analytics
 Column-oriented, extremely fast for aggregations
 
DuckDB deserves special attention. It has become the standard for local data work — you can query Iceberg tables on S3 directly from your laptop with zero infrastructure:
import duckdb
# Query Iceberg on S3 from your laptop — no servers needed
conn = duckdb.connect()
conn.execute("INSTALL iceberg; LOAD iceberg;")
result = conn.execute("""
 SELECT
 event_type,
 COUNT(*) as event_count,
 APPROX_COUNT_DISTINCT(user_id) as unique_users
 FROM iceberg_scan('s3://my-data-lake/events/user_events/')
 WHERE created_at >= CURRENT_DATE - INTERVAL 7 DAYS
 GROUP BY event_type
 ORDER BY event_count DESC
""").fetchdf()
AI in the Data Stack: What’s Real
The most overhyped and most misunderstood component of the modern data stack is AI. Let me separate signal from noise.
Real and useful:
Text-to-SQL: Natural language queries that get translated to SQL. Works well for standard analytical queries. Requires good catalog documentation (dbt docs help enormously).
Anomaly detection: ML models that flag unusual patterns in metrics. Genuinely catches issues humans would miss.
Data quality: AI-assisted schema inference, duplicate detection, PII identification in new datasets.
Documentation generation: AI-written column descriptions and table summaries, seeded from sample data and column names.
Not ready for production:
Fully autonomous data pipelines that “figure out the schema and transformations by themselves”
AI that replaces data engineers for complex transformation logic
Natural language as the primary interface for business analysts (most still prefer dashboards)
The practical recommendation: use AI to accelerate data engineers, not to replace them. Text-to-SQL for exploration, AI-generated docs to fill the documentation gap, anomaly detection to catch data quality issues early.
The Stack We Recommend in 2026
For a company processing 100GB-10TB/day:
Layer
 Tool
 Why
 
Ingestion
 Airbyte (managed) or Fivetran
 300+ connectors, minimal maintenance
 
Streaming
 Kafka + Flink
 Standard, large ecosystem
 
Storage
 S3 + Apache Iceberg
 Open, cheap, portable
 
Catalog
 Project Nessie or AWS Glue
 Git-like branching for data
 
Transformation
 dbt Core
 Version-controlled, testable SQL
 
Orchestration
 Airflow or Dagster
 Dagster for better observability
 
Query Engine
 Trino (shared) + DuckDB (local)
 Best cost-to-performance ratio
 
BI/Visualization
 Metabase or Superset
 Open-source; Looker if budget allows
 
Total infrastructure cost for this stack at medium scale: roughly $3,000-8,000/month on AWS, compared to $25,000-50,000/month for equivalent Snowflake + Fivetran + Looker.
The Bottom Line
The Modern Data Stack in 2026 is both more powerful and more complex than it was three years ago. Open table formats have delivered on their promise of vendor independence. dbt has matured from a transformation tool into a data development platform. The streaming lakehouse has closed the gap between real-time and batch analytics.
The teams that are winning have three things in common:
They invested in data quality and testing from day one (dbt tests, Great Expectations)
They treat data as a product with SLAs and ownership, not a byproduct of engineering work
They resist adding new tools until they’ve exhausted what the existing tools can do
The data stack is rich enough now that almost any analytical problem can be solved with open-source tools. The limiting factor is no longer technology — it’s discipline, data culture, and the unglamorous work of keeping pipelines healthy and data trustworthy.
That part, no amount of tooling will automate away.

The Modern Data Stack in 2026: dbt, Iceberg, and the End of the Data Warehouse as We Knew It

댓글