Welcome back to Data This Week!
This edition brings major architecture-level shifts across the modern data stack. We look at the trade-offs of serving low-latency APIs directly off lakehouse tables, Databricks pushing Spark into a fully declarative paradigm, and the upcoming deprecations and primitives in Apache Iceberg 1.12. On the storage and transactional front, we break down Postgres search internals, serverless CDC patterns on AWS, browser-native persistence in DuckDB, and how Notion revamped concurrent editing using CRDTs.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
Serving APIs Directly Off S3 Tables: The Reality Check
Engineers often dream of ditching the operational overhead of transactional read caches and serving customer-facing APIs straight from object storage. Manish Kumar Yadav details what actually happens when querying Amazon S3 Tables (managed Iceberg) via AWS Lambda. The evolution went from a classic PyIceberg + PyArrow setup (great for simple point lookups, but unable to handle native multi-table joins) to pairing PyIceberg with in-memory DuckDB, and finally letting DuckDB attach directly to the Iceberg REST catalog. The key takeaway for platform architects: while DuckDB collapses multiple moving parts into a unified SQL interface, extension cold-start penalties (1–3s) and read tail latencies mean this architecture excels at internal analytics or flexible GraphQL reporting, but cannot yet displace operational OLTP stores for sub-50ms SLAs.
Getting Started with Spark Declarative Pipelines
Databricks is shifting Spark workloads from imperative DAG wiring to end-to-end declarative engineering with Spark Declarative Pipelines (SDP, the foundation of Lakeflow). Rather than hand-rolling Airflow DAGs, stateful checkpointing, and bespoke incremental MAX(updated_at) filters in PySpark or dbt, SDP lets engineers declare what datasets should exist using SQL or Python decorators (@dp.table, @dp.expect_or_drop). The underlying engine automatically handles dependency graphs, incremental state tracking, and failure recovery. For teams running massive batch and streaming transformations, moving dependency and backfill logic directly into the engine planner promises significantly less glue code to maintain.
Streaming PostgreSQL Changes to Amazon S3 with AWS Fargate
Setting up Change Data Capture (CDC) from operational databases to data lakes often defaults to running heavy self-managed Debezium clusters or paying premium SaaS connector fees. AWS architectural guidance outlines a lightweight, serverless CDC pipeline that streams Postgres WAL logs straight into Amazon S3 using logical replication (wal2json) and AWS Fargate. By running a containerized CDC reader that continuously polls replication slots without managing persistent EC2 instances, this pattern provides an isolated, cost-effective alternative for teams feeding raw event streams directly into S3 landing zones for downstream lakehouse ingestion.
Preview of Apache Iceberg 1.12 & Iceberg Rust 0.11
Apache Iceberg 1.12 and Iceberg Rust 0.11 are hitting the ecosystem, packing major foundational updates alongside breaking changes. In Iceberg 1.12 Java, Spark 3.4 runtime support is officially dropped, and writing Position Deletes with Row Data (PDWR) is removed — actions will now fail explicitly on files retaining row data. On the feature side, native geospatial types (geometry and geography) are finally wired through Parquet logical types, and early V4 spec plumbing (relative paths, TrackedFile builders) is introduced. Meanwhile, Iceberg Rust 0.11 brings end-to-end table encryption, native Variant support, and revamped REST authentication session management.
Anatomy of a Postgres Search Engine
Before reaching for an external Elasticsearch or OpenSearch cluster, data and database engineers should understand how inverted indexes function inside relational storage. PlanetScale’s deep dive breaks down the internal mechanics of building search in Postgres: dissecting term dictionaries (B-trees mapping tokens to postings lists), compressed postings lists (delta encoding and SIMD-friendly bitmap representations), positional offset tracking for phrase proximity queries, and BM25 scoring. If you are designing low-latency full-text search directly within your transactional or analytical database, this is an essential read on index compression and memory bandwidth trade-offs.
Standardizing Data Contracts & Products with Bitol (ODCS & ODPS)
As decentralized data platforms mature, governance often falls apart at the interface between data producers and consumers. The Bitol project (under the LF AI & Data Foundation) has formalized the Open Data Contract Standard (ODCS v3.2) and Open Data Product Standard (ODPS v1.1). ODCS establishes a vendor-agnostic YAML schema defining schema types, data quality rules, SLAs, and security classifications, while ODPS bundles those datasets into discoverable, versioned, and reusable data products. If your organization is standardizing automated contract testing in CI/CD, this specification is the emerging industry standard to watch.
How Notion Handles Concurrent Editing with CRDTs
Building collaborative text architectures across distributed clients requires ditching coarse locking mechanisms. Notion’s engineering team shared an in-depth breakdown of how they rebuilt their block-based text engine using Conflict-free Replicated Data Types (CRDTs). Operating on top of the Peritext algorithm for rich-text annotations, Notion solved the classic “block splitting” dilemma (when concurrent users split a paragraph while another edits text) using concepts like “text slice trees” and deterministic search labels (L/R branch identifiers). A brilliant read for senior engineers interested in distributed conflict resolution and local-first architecture.
Persistent Databases in the Browser with DuckDB-Wasm and OPFS
Previously, DuckDB-Wasm was ephemeral — closing a tab wiped the in-memory Wasm heap unless you serialized Parquet into IndexedDB manually. DuckDB’s latest architectural update introduces native support for the Origin Private File System (opfs://). This gives DuckDB-Wasm random-access read/write capabilities directly against sandboxed browser storage. By bringing DuckDB’s Write-Ahead Log (WAL) and checkpointing mechanisms straight into client-side workers, teams can build zero-infrastructure, persistent local-first analytical applications that cache multi-gigabyte analytical datasets right on the user’s machine.
🛠️ Tools
Kafgres
What it is: A lightweight, high-performance PostgreSQL extension that embeds a singleton Kafka broker directly inside the Postgres process. Instead of provisioning and operating a dedicated Kafka or Redpanda cluster for low-to-medium scale queueing, Kafgres allows producers and consumers to interact using standard Kafka wire protocols while writing topic logs directly to disk via background workers. By avoiding Postgres table heap overhead for raw message streams, it clocks sustained write throughput exceeding 700 MB/s while unlocking seamless transactional coordination.
Blog & Architecture → | GitHub →
💬 Community Sentiments
Does AI Actually Struggle at Data Modeling?
A widely discussed thread on r/dataengineering tackled a common industry observation: while modern LLMs excel at generating boilerplate dbt SQL, writing ingestion scripts, and explaining regex, they consistently fall apart when asked to produce robust, domain-aligned data models (3NF, Kimball dimensional stars, or Data Vault).
The community’s consensus boils down to three core realities:
-
Lack of business context: Data modeling is not a syntax exercise; it is an organizational mapping exercise. AI models cannot infer tribal operational context — such as whether a “cancelled” order should be filtered out of revenue metrics or how churn is legally recognized.
-
Context window limitations for complex lineage: While an LLM can model isolated toy tables, it struggles to evaluate multi-hop downstream trade-offs, fan/chasm traps, or grain mismatches across sprawling enterprise schemas.
-
The senior data engineer moat: As code generation becomes commoditized, deep conceptual modeling, data contract definition, and semantic architecture remain the most durable skills for senior practitioners. AI makes a great copilot for linting and drafting documentation, but human judgment is strictly required to decide schema grain and entity boundaries.
That’s all for this week! See you in the next edition.