How writes create small files

Every write to a Delta table adds new Parquet files. Lots of small writes - dozens of single-row inserts or merges - leave the table as many tiny files, and reads slow down because the engine must open and scan all of them. This is the "small-files problem", and the fix is compaction.

Why it happens

• Each insert_rows / merge_rows call writes new file(s) and a new version.
• Updates and deletes don't rewrite in place - they write new files and mark old ones removed.
• Over time the file count grows even if the row count doesn't.

Two things that make it worse

• Drip writes - 1000 single-row calls create far more small files (and versions) than one batched call of 1000 rows.
• Over-partitioning - partitioning on a high-cardinality column scatters rows into thousands of tiny per-partition files.

The fixes

Next

• Compact small files · A maintenance cadence.