Decide partitioning
When partitioning a Delta table helps query performance - and when it just hurts.
Partitioning splits a Delta table's files into subdirectories by the value of one or more columns. It can speed up queries that filter on those columns - or quietly make everything slower. For master data, the answer is usually don't.
What it does
A table partitioned by, say, region stores files under
…/region=EU/, …/region=US/. A query filtered on region = 'EU' then reads
only that partition's files ("partition pruning") instead of scanning the whole
table.
When it helps
- Large tables (many millions of rows) that are almost always filtered on a low-cardinality column (region, year, tenant).
- The partition column has a handful to a few hundred distinct values.
When it hurts
Heads up
- Small/medium tables - the overhead outweighs any pruning benefit.
- High-cardinality columns - explodes file count.
- Columns you rarely filter on - no pruning, all cost.
Rule of thumb
Start unpartitioned. Reach for partitioning only when a table is genuinely
large and a clear, low-cardinality filter column dominates your queries. When in
doubt, leave it off and rely on optimize to keep
file sizes healthy.