Opening the field notes · Labs
Setting up the experiment.
Opening the field notes · Labs
Setting up the experiment.
See how a storage engine saves incoming writes to files, then merges those files in passes.
TryCompare strategies: flush a burst, delete its latest key, add a burst before flushing, then compact when eligible.
Watch forTombstones clear only when a safe merge includes every older copy; Size-tiered needs four similarly sized files.
The linked story describes related work; the scenarios and results here are synthetic.
Skill practicedBalance write effort, read work, and storage space
Connection to my workNeuralEye documents PostgreSQL and Redis; this lab explores a separate storage-engine design.
A log-structured merge-tree (LSM) stores new writes in memory, flushes them to files, then merges those files in passes. Compare leveled and size-tiered strategies, then follow a tombstone as it shadows an older value and is removed only after a safe merge. The synthetic workload is not a reproduction of a particular storage engine.
This schematic model uses simplified file sizes and merge passes, not RocksDB, Cassandra or another engine. Space amplification counts on-disk SSTable bytes divided by all live logical bytes; staged memtable keys count as live data but are not on disk yet, so this interim ratio can be below 1 until flush. In this workload, leveled passes rewrite more bytes while reducing overlapping versions; size-tiered passes rewrite less often while retaining stale versions longer. Actual amplification depends on workload and implementation details.
NeuralEye’s project story documents analytics storage with PostgreSQL and Redis, not an LSM compaction engine. This separate teaching model lets me reason about write amplification and background compaction as adjacent storage-engine concepts; it does not describe NeuralEye’s storage implementation or configuration.
See the broader engineering briefWaiting for writes
Nothing has been written yet. Add a burst, flush the staged keys into an L0 file, then create enough files for the selected strategy to compact.
Workload
Write a burst to choose a key to delete.
Write a burst to stage keys in memory, then flush them to create an L0 table.
Leveled compaction is ready when L0 has multiple files or a level reaches its byte budget.
Compaction strategy
Changing strategy resets the experiment to an empty run, clearing staged keys and files. Note this result first, then repeat the same actions to compare.
Each rectangle represents one on-disk SSTable. Its width estimates that level’s average table size relative to the largest level; the model exposes aggregate byte totals.
Write amplification
—
Available after the first logical write
Space amplification
—
Available once live data exists
L0 SSTables
0
Below stall threshold
Compaction passes
0
0 flushes total