Data Engineering on App Coding

A Database Where Every Value Remembers Its Source, Confidence and Extraction Time

Mon, 05 Oct 2026 00:00:00 +0000

Somebody pastes a population figure into a shared spreadsheet: 1,234,567. It might have come from a census table, a Wikipedia infobox, a city press release or a model’s summary of one of those, and three weeks later nobody can say which. The paste kept the digits. Everything that made them believable stayed in a browser tab that closed long ago.

A value without provenance is a rumor with a data type.

A Telemetry Database That Forgets on Purpose Keeps Rollups and Anomalies Instead of Raw Rows

Mon, 05 Oct 2026 00:00:00 +0000

At 300 requests a second, one latency metric adds about 26 million rows a day. Nobody will read most of them again. What people ask for a month later is the p99 per route for last Tuesday, and the one request that took nine seconds. A rule that deletes rows after 30 days throws away both answers. Keeping every row forever pays storage bills for data nobody reads.

The telemetry store worth building decides at write time what has to survive. It folds everything else into coarser buckets on a schedule and keeps the few raw events that carry information, whole. Forgetting becomes part of the schema, declared in a policy, instead of a cron job that deletes by age. This is the argument for reducing data near the source, applied to time series.

A Tiny SQL Engine for JSON Streams Fits Between jq and a Stream Processing Cluster

Mon, 05 Oct 2026 00:00:00 +0000

Your service writes one JSON object per line to stdout, and you want the average cpu for host web-3 over the last five minutes. jq can’t answer that, because it forgets each object once it has printed it and has no idea what “five minutes” means. You can fake both with jq -n 'reduce inputs as $e (...)' plus a timestamp comparison, and you’ll have a script nobody wants to touch twice. DuckDB answers the question in one line, but only against a file as it exists right now. ksqlDB, Materialize, RisingWave, Arroyo and Flink SQL answer it continuously, once you’ve deployed a service and agreed to operate it.

An Embedded Sketch Database: Billions of Events, Megabytes of Storage, Error Bars on Every Answer

Mon, 05 Oct 2026 00:00:00 +0000

Counting unique visitors exactly means remembering every visitor. Say 6 million distinct IPv4 addresses hit a checkout service today. Packed at 4 bytes each, that set is 24 MB before any hash table overhead, and you need one per service, per region, per day you want to compare. Redis answers the same question with a HyperLogLog in about 12 KB per key, at a standard error of about 0.81%.

The thin part is the database around it: a SQLite-shaped file of sketches with time buckets, rollups and a SQL front door (check before you start; this space moves). The version worth building stores nothing but mergeable sketches, grouped by dimension and time bucket, and every aggregate returns its error bound next to the value. You never keep the events. It’s the extreme case of reducing data near the source: keep the answer, drop the rows.

An Evidence Graph Built From Extracted Claims Keeps Contradictions Attached to Their Sources

Mon, 05 Oct 2026 00:00:00 +0000

Feed forty articles about a rumored plant restart to a summarizer and you get one smooth paragraph: several outlets report that production will resume in March, though the ministry has pushed back. It reads well. It also throws away what you’d need to act on it. You can’t tell which outlets reported it, whether each had its own source or all copied one wire story, or what the ministry actually said, a denial or a polite refusal to comment.

DuckDB Already Queries Every File in a Folder, So Build the Part That Infers the Joins

Mon, 05 Oct 2026 00:00:00 +0000

Someone hands you a zip: customers.csv from the CRM, a JSON dump from the billing API, and a SQLite file from an internal app nobody has touched in two years. DuckDB will make all three queryable in a few lines of SQL, with no import step (SELECT * FROM 'customers.csv' works as written). Then you spend the afternoon working out which column joins to which. The CRM says customer_id and stores 00123. The billing API says customerId and stores 123. The SQLite table has a column called customer holding the text 123. Same customer, three spellings.

Facts Should Expire: A Database That Tells an AI Agent When Its Knowledge Has Gone Stale

Mon, 05 Oct 2026 00:00:00 +0000

An agent keeps a memory table, and two rows matter for today’s task: a supplier’s headquarters city, written eleven months ago, and a freight quote, written three hours ago. A SELECT returns both in the same shape. The headquarters is probably still right. The quote may already be wrong. Nothing in either row says so, and the agent repeats both in the same flat tone.

A row has no idea how old its truth is. It might carry a created_at, and a careful person can compute an age from it, but the decay rate belongs to the kind of fact, not to the table. A three-week-old headquarters address is fine. A three-week-old shipping rate is fiction. Server health goes bad in half a minute. No timestamp column holds that knowledge, and nothing forces a query to read it anyway.

Incremental View Maintenance Small Enough to Embed Turns Dashboard Queries Into Lookups

Mon, 05 Oct 2026 00:00:00 +0000

A dashboard tile shows today’s US traffic, so every page load runs SELECT count(*) FROM events WHERE country = 'US' over a table that grew by a few hundred thousand rows since yesterday. SQLite answers correctly. It also walks every matching index entry to do it, because its b-trees don’t keep counts, and it does the same walk for the next viewer and the one after. Between two page loads the answer moves by a handful of rows, yet each load pays for all of them.

Kafka for One Machine: An Embedded Event Log With Offsets, Compaction and No Broker

Mon, 05 Oct 2026 00:00:00 +0000

A desktop app wants an undo history that survives a crash. An edge gateway buffers sensor readings while its uplink is down and has to replay them, in order, once the link is back. An AI agent writes down every action it takes so a bad run can be replayed later. All three are the same data structure: an append-only sequence of records, plus readers that remember where they stopped. Kafka is that structure, and also a cluster, partitions and the people who tune them.