Data Engineering Pipeline Questions I Prep
Four pipeline-shaped questions I rehearse before data-engineering loops: incremental ingestion, idempotent upserts, late-arriving data, and a SQL window-function read. Python throughout, light on framework lock-in.
By @sarahwilson
November 22, 2025
·
Updated May 20, 2026
202 views
3
4.5 (12)
An hourly job pulls new rows from a source table into a warehouse. What is the column you key the incremental cutoff on, what is the bug if you key on created_at, and what is the operationally honest answer?
Sample
Sample run: pull_new_rows(source, sink, last_high_watermark_ms=1715347200000) reads WHERE updated_at > 1715347170000 (a 30s overlap), upserts on id, and returns the new max updated_at. The bug if you key on created_at instead: any row inserted before the cutoff but updated after will fall outside the window, so the update is silently missed.
3 more questions and all solutions are locked.
Purchase this item to access all questions, code snippets, and solutions.