Scaling the engine
The engine is stateless - scale it out to handle more query and I/O load.
The engine does the data-plane work: reading and writing Delta tables and
running queries. It's stateless, so it scales horizontally - the server fans
work out across whatever engine replicas exist (via DNS discovery in the :k8s
edition). The server and Postgres rarely need scaling.
Scale on demand
kubectl -n eddytor scale deploy/eddytor-engine --replicas=5That's it - new replicas register and start taking work. No restart of the server needed.
Autoscaling
The chart enables an HPA on the engine by default, so it scales with load automatically. Tune or disable it via values:
# disable autoscaling and pin a fixed count
--set engine.autoscaling.enabled=false --set engine.replicas=3Trimming for small clusters
On a small node pool, drop to single replicas so the pods fit:
--set engine.replicas=1 --set server.replicas=1 --set engine.autoscaling.enabled=falseWhat you can't do
Heads up
eddytor-engine / eddytor-engine-headless) and the server
discovers replicas by those names - don't rename them or run two releases in one
namespace.This scaling only applies to the :k8s edition (DNS discovery). Docker
Compose runs a single engine over direct TCP - to scale, move to
Kubernetes.
Why this works
The engine holds no state between requests - table data lives in your object storage, metadata in Postgres. Any engine replica can serve any request, so adding replicas linearly adds throughput for concurrent queries and writes. See Architecture.