Optimize & vacuum

Compact small files with optimize_table, then reclaim storage with vacuum_table - in that order.

Maintenance is two periodic chores: optimize compacts a table's many small Parquet files into fewer larger ones (the single biggest read-performance win after a run of small writes), then vacuum deletes the old, unreferenced files the compaction left behind.

Optimize

optimize_table creates a new version but does not change your data.

eddytor optimize table eddytor sales orders

POST /v1/tables/eddytor/sales/orders/optimize
Authorization: Bearer edd_live_…

{ "targetSizeMb": 256 }

{ "table": "eddytor.cfg_xxx.<uuid>_products", "target_size_mb": 256 }

When to run it:

After sequences of many small writes (dozens of inserts/merges) - that's where it pays off most. Every write adds new Parquet files (updates and deletes don't rewrite in place), so the file count grows even when the row count doesn't.
Not after every write - batch first, then optimize periodically.
Skip tiny tables (< ~1000 rows) - minimal benefit.

Good to know

optimize_table is idempotent - running it again on an already-optimized table is harmless. It shows up in history as an OPTIMIZE version; don't mistake it for a data change when picking a rollback point.

Vacuum

vacuum_table deletes old, unreferenced data files (the ones optimize and updates/deletes left behind), reclaiming storage. It is not reversible - so always dry-run first.

eddytor vacuum table eddytor sales orders --dry-run
eddytor vacuum table eddytor sales orders            # execute after reviewing

POST /v1/tables/eddytor/sales/orders/vacuum
Authorization: Bearer edd_live_…

{ "retentionHours": 168, "dryRun": true }

Then send again with "dryRun": false to execute.

{ "table": "eddytor.cfg_xxx.<uuid>_products", "retention_hours": 168, "dry_run": true }

Then run again with "dry_run": false to execute.

Heads up

Vacuum is NOT idempotent - deleted files are gone forever. The dry run is your safety net: it previews exactly what would be removed. Always run it first and review the file count / space savings.

retention_hours (default 168 = 7 days) sets how far back files are kept.

Heads up

Setting retention_hours below 168 can break time-travel. After vacuum, rollback_table and restore_table to versions older than the retention window fail - their files are deleted. Keep retention_hours >= 168 unless you're certain you won't need to recover that far back.

Order matters: optimize, then vacuum

Run optimize first (creates compacted files but leaves the old small ones), then vacuum (removes them). Confirm both ran in history.

How often

Table activity	Optimize	Vacuum
Frequent small writes	after each burst of writes	periodically (weekly)
Steady/batch loads	after the batch	weekly/monthly
Rarely written	only if reads slow	rarely
Tiny (< ~1000 rows)	skip - negligible benefit	rarely

Optimize

Vacuum

Order matters: optimize, then vacuum

How often

On this page