Welcome back to Data This Week!
This edition focuses heavily on physical layout mechanics, metadata-driven architecture, and cross-system observability. We unpack why sorting individual Parquet files doesn’t automatically give you an ordered scan in Apache DataFusion, how rank-space Hilbert curves keep data skew from ruining row-group pruning in DuckDB, and how to guarantee point-in-time correctness when building graph layers on top of Apache Iceberg. On the platform reliability front, we examine how Netflix and Uber model dependencies and observability across massive distributed architectures, alongside a practical runbook for Apache Polaris disaster recovery.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
Why Sorted Parquet Files Do Not Guarantee an Ordered Scan in DataFusion
Writing Parquet files sorted by a key (e.g., customer ASC) and declaring WITH ORDER (customer ASC) on an external table feels like it should eliminate hash repartitioning and sort overhead during a GROUP BY customer query. Christian Del Monte traces the physical execution plan in Apache DataFusion (ProjectionExec, AggregateExec, RepartitionExec) to show why the engine still drops output_ordering at the scan level and falls back to hash partitioning.
The culprit is overlapping key ranges across files. Because DataFusion’s scan operator groups files into execution partitions (target_partitions), placing multiple files whose min/max key ranges overlap into the same partition stream, or reading overlapping ranges concurrently across partitions without a sort-preserving merge (SortPreservingMergeExec), destroys global stream ordering. Even enabling datafusion.execution.split_file_groups_by_statistics = true cannot salvage an ordered scan if every daily file spans the full customer alphabet. To actually unlock streaming ordering_mode in AggregateExec, your compaction and write pipelines must produce non-overlapping key ranges across files or explicitly trade parallelism for sort-preserving merge operators.
Build a Leak-Free Graph on Your Lakehouse Gold Tables
When serving ML features, fraud detection context, or RAG pipelines from a graph layer built over Lakehouse Gold tables, temporal data leakage is a silent killer: graph traversals easily pull edges or properties modified after the model’s decision timestamp. Santosh Shinde demonstrates how to construct a strictly point-in-time correct, zero-infrastructure property graph using Apache Iceberg tags and LadybugDB (the embedded columnar property graph engine built on the Kùzu architecture).
By pinning immutable Iceberg snapshot tags to model training and inference windows, the pipeline materializes version-locked Gold state into a single-file .lbug embedded database. Combined with temporal boundary predicates enforced via a single extra Cypher constraint on edge validity intervals, multi-hop graph traversals are guaranteed to evaluate only the state the model knew at decision time. This eliminates future-state leakage without standing up a dedicated, stateful graph cluster.
Large-Scale Automated Dependency Analysis Across Uber’s Service Mesh
In a microservice architecture at Uber’s scale, knowing whether a downstream call is a hard (fail-close) or soft (fail-open) dependency is critical for reducing Mean Time to Mitigate (MTTM) and avoiding noisy on-call pages. Relying on standard distributed trace sampling to correlate upstream and downstream failures takes days of observation to gather just 10 matching failure samples along deep call chains (A → B → C → D → E), by which time deployments have already mutated the call graph.
Uber bypassed trace-sampling bottlenecks by instrumenting inbound (Handle) and outbound (Call) RPC middleware directly at the Go transport layer, crucially after all client retry attempts resolve, to log edge failure pairs. By computing conditional failure probabilities (Pc) from co-occurring caller/callee errors and applying heuristic thresholds (Pc ≥ 0.8 for fail-close, Pc ≤ 0.2 for fail-open), Uber continuously maps true critical-path dependencies across its service mesh in near real-time.
Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale
Netflix unpacks why a typical 4-hour incident investigation rarely stalls on raw compute, and instead stalls on reconstructing entity relationships across isolated tools. While metrics, events, logs, and traces (MELT) capture raw telemetry, systems like Atlas (telemetry), Radar (alerting), Spinnaker (deployments), PagerDuty, and incident.io disagree on entity boundaries. What one system calls an “application,” another calls a “service” or “cluster,” with slightly different naming conventions. Without a shared semantic contract, both human responders and SRE AI agents hallucinate or bottleneck on tribal knowledge.
Netflix solved this as a data modeling problem by defining a formal, machine-readable Ontology backed by QuipuDB, their homegrown distributed RDF graph database. A 5-layer enrichment pipeline runs dozens of sequential enrichers that consume noisy operational events and emit validated (subject, predicate, object) triples. During an incident, their correlation engine traverses this knowledge graph, scoring causal chains by temporal proximity and severity. For example, it surfaces a 0.87 confidence hypothesis linking a client latency spike to a GraphQL rollout 6 hops downstream.
dbt and Amazon Redshift Data Sharing: Multi-Cluster Pipelines Without Moving a Byte
For platform teams splitting ingestion/ETL workloads and BI consumption across isolated Amazon Redshift clusters, copying tables across boundaries introduces latency, storage duplication, and governance drift. This guide details how to wire Amazon Redshift Data Sharing directly into multi-cluster dbt pipelines so consumer clusters transform live producer data with transactional consistency and zero data movement.
By enabling datasharing: true in profiles.yml, dbt resolves shared objects across database boundaries using Redshift’s cross-database system views. A critical architectural constraint for senior engineers to note: downstream dbt models reading from datashare sources must use table (or incremental) materializations, as standard Redshift views cannot bind to shared cross-cluster objects without late-binding configurations.
Rank-Based Clustering for Improved Pipeline Performance
Multi-column sorting via space-filling curves (Morton/Z-order or Hilbert curves) is a staple for improving Parquet row-group min/max pruning, but raw value-space encoding falls apart on skewed production data, where most rows bunch into a tiny fraction of the coordinate grid. Isaac Harvey (creator of Duckstring) benchmarks rank-space Hilbert clustering on a 288-million-row (14.4 GB) TPC-DS store_sales table in DuckDB.
Instead of bit-interleaving raw column values, the pipeline first maps skewed columns (like ss_net_paid and ss_sold_date_sk) into uniform buckets using DuckDB’s approx_quantile, then orders rows along a Hilbert curve via the lindel community extension (hilbert_encode), targeting roughly one grid cell per 122,880-row group. Because each row’s Hilbert key depends only on its own quantile bucket after the lightweight quantile pass, DuckDB streams the key calculation with minimal memory overhead. This dramatically cuts the share of row groups read on selective multi-column range scans, especially over high-latency object storage.
Getting Started with Apache Iceberg Write Support in Amazon Redshift (Part 1)
As Amazon Redshift expands native read/write interoperability with lakehouse storage, this walkthrough covers the foundational mechanics of creating and mutating Apache Iceberg tables backed by Amazon S3 Tables (@s3tablescatalog) and AWS Glue Data Catalog directly from Redshift SQL.
It walks through configuring federated IAM roles (RedshiftIcebergRole) with Lake Formation permissions, executing CREATE TABLE ... USING ICEBERG PARTITIONED BY (DAY(order_date)) and CREATE TABLE AS (CTAS) with custom TABLE PROPERTIES (such as snappy vs. zstd or uncompressed), and simplifying cross-catalog SQL queries using three-part notation ("bucket@s3tablescatalog".namespace.table) alongside session USE and set search_path directives.
We Need to Talk About JEV in Data Pipelines
Using frontier generative LLMs to classify hundreds of thousands of messy text records inside batch ETL pipelines is slow, expensive, and prone to formatting hallucinations. João Pedro shares lessons from processing 630,000 public election expense records (about 200,000 distinct, unstandardized descriptions) using JEV (by TypeSafe AI), a specialized, constrained language model restricted to answering three structured classification question types with confidence scores.
Rather than throwing every row at an API, the architecture uses a hybrid funnel. SQL keyword rules deterministically classify the easy ~90K descriptions first, leaving 110,000 ambiguous records (~68.2M tokens) for JEV. By enforcing strict class-specific probability thresholds (e.g., 60% to 95% confidence, depending on how blurry the class boundary is) and discarding low-confidence “Other” noise, the pipeline classified the entire dataset in under 30 minutes for just $2.60.
Disaster Recovery for Apache Polaris: Backup and Restore
Recovering an Apache Polaris catalog deployment requires coordinating two asymmetric systems: append-mostly object storage (Iceberg data and metadata files) and a live transactional JDBC metastore (holding catalogs, principals, RBAC grants, and current table metadata pointers). Moreover, because Polaris interleaves multi-tenant realm IDs into primary keys within the same JDBC tables, a database restore is all-or-nothing across realms.
Dremio lays out a battle-tested DR runbook that highlights the #1 failure mode teams overlook: restoring a catalog that is newer than object storage. If the catalog points to metadata JSONs or snapshots that haven’t replicated or were lost in storage, tables fail to load entirely. The golden rule is storage first, catalog second. A slightly older catalog merely leaves unreferenced orphan files in S3, whereas a newer catalog breaks table resolution until you manually roll back snapshots table by table. The post also covers continuous WAL archiving vs. logical dumps, why you must pre-create the JDBC schema before Polaris bootstrap, and three scheduled chaos drills to measure your true RTO.
🛠️ Tools
duckxlsx
What it is: A lightweight, high-performance open-source utility by me, designed to bridge the gap between DuckDB and complex Excel (.xlsx) workflows. While data engineers spend years building petabyte-scale lakehouses, finance and operations stakeholders still inevitably demand multi-tab, formatted Excel exports or drop messy spreadsheets into ingestion landing zones. duckxlsx streamlines reading and writing .xlsx files directly within DuckDB pipelines without round-tripping through heavy, memory-bloated Pandas/openpyxl intermediaries, keeping single-node analytical and reporting pipelines fast, scriptable, and low-footprint.
💬 Community Sentiments
How Do You Find Data Mismatches Between Two Large Tables?
A classic production war story surfaced on r/dataengineering this week: when validating a migration or debugging pipeline drift between source and target tables containing hundreds of millions (or billions) of rows, how do you pinpoint missing records and column-level value mismatches without blowing up warehouse compute?
The community consensus broke down into four battle-tested patterns:
- Hierarchical Aggregate & Bucket Narrowing (Divide and Conquer): Never start with a full-table row-level join. Engineers overwhelmingly recommend grouping by date/partition first (
COUNT(*),SUM(numeric_col),COUNT(DISTINCT pk),MIN/MAX(updated_at)) to isolate the exact daily or hourly partitions that drifted before inspecting individual rows. - Row-Hash Comparison over Wide Joins: Instead of joining or running
EXCEPTacross 80 columns (which triggers massive network shuffles and uglyNULL-safe equality checks), compute a deterministic hash of concatenated column values (MD5orxxHash64overCOALESCE(CAST(col AS VARCHAR), '__NULL__')) alongside the primary key. Joining only on(pk, row_hash)immediately isolates missing keys vs. mutated payloads at a fraction of the I/O cost. EXCEPT/FULL OUTER JOINon Pruned Slices: Once narrowed to a faulty partition,SELECT ... FROM source EXCEPT SELECT ... FROM target(and vice versa) remains the cleanest SQL idiom for exact row diffs, followed by unpivoting (UNPIVOTor JSON object diffing) on mismatched primary keys to see exactly which column diverged.- Specialized Bisection Tools: For cross-database validation (e.g., Postgres source to Snowflake/Iceberg target), veterans pointed to bisection-based checksum tools (inspired by
data-diff/ Merkle-tree style segment hashing) or pulling narrow partition slices into local DuckDB to diff out-of-band without burning cloud warehouse credits.
That’s all for this week! See you in the next edition.