Warehouses and scaling
Choose, isolate, start, stop and scale VegaDB compute for each workload.
A VegaDB warehouse is a pool of workers that runs queries for one workspace. It reads the catalogs authorized for its workspace and does not own the underlying data. A workspace can have several warehouses, each with its own size, scaling limits and idle time, all reached through the same stable endpoint.
Choose a warehouse profile
| Workload | Starting profile | Optimize for |
|---|---|---|
| Development and notebooks | X-Small or Small, short idle time | Low idle cost and fast startup |
| Scheduled transformation | Medium, short idle time | Throughput during a bounded window |
| BI and dashboards | Medium or Large, higher maximum workers | Concurrency and predictable latency |
| Large joins or backfills | Memory-optimized | Working-set headroom and throughput |
| Customer-facing analytics | Dedicated warehouse, higher minimum workers | Isolation, concurrency and availability |
The profile sets the worker size; minimum and maximum workers set how far the warehouse scales. Profile names and quotas depend on the account plan. The warehouse detail shows the worker size, minimum and maximum workers, idle time and regional availability.
Lifecycle
Create
Choose a region, profile, minimum and maximum workers, and the idle time after which the warehouse stops.
Start
The first query starts a stopped warehouse. You can also start it explicitly, from the console or the API, before a known workload.
Scale
Queued work adds workers up to the maximum; workers are removed again when demand stays lower.
Stop
An idle warehouse stops by itself. Stopping ends compute billing while preserving catalogs, tables, permissions, query history, sessions and the endpoint.
A warehouse is stopped, starting, running, resizing, stopping or failed.
Start and stop
Start on the first query. A query sent to a stopped warehouse starts it. The query is parsed, authorized and planned while the workers start, and reports the state resuming to every interface: Flight SQL polling, Spark Connect and the HTTP API. Concurrent queries join the same start rather than starting it again. When the minimum number of workers is ready, the query is placed on them and runs. If the warehouse does not become ready within its resume timeout, the query fails with WAREHOUSE_RESUME_TIMEOUT.
Metadata requests and reads of a stored result do not start a warehouse.
Stop when idle. When a warehouse has no running or queued queries for its idle time, it finishes its remaining work and stops. Open sessions stay open; their next query starts the warehouse again.
Explicit start and stop. Start a warehouse before a known workload, or stop it at once, from the console or the control API. An explicit stop also suspends automatic start: queries on that warehouse fail rather than starting it, until someone starts it again.
BASE="https://<warehouse endpoint>/api/v1/vegadb/orgs/<organization id>/workspaces/<workspace id>/warehouses"
curl "$BASE/<warehouse id>" --header "Authorization: Bearer $VEGADB_TOKEN"
# → {"id": "...", "name": "bi", "state": "running", "suspended": false, "desired_workers": 4, ...}
curl --request POST "$BASE/<warehouse id>/stop" --header "Authorization: Bearer $VEGADB_TOKEN"
curl --request POST "$BASE/<warehouse id>/start" --header "Authorization: Bearer $VEGADB_TOKEN"Running queries on a warehouse needs USAGE on it. Starting and stopping it needs OPERATE. The workspace permission to use a warehouse grants USAGE; the permission to manage it grants USAGE, OPERATE and MODIFY.
Scaling
A warehouse scales between its minimum and maximum workers on demand:
- Out. Queued statements and memory reservation pressure raise the worker count, about one worker for every eight waiting or active statements, up to the maximum.
- In. When lower demand lasts for a minute, the warehouse drains its highest workers: they finish the tasks they hold and take no new ones before they leave. A worker selected for removal returns to service if demand comes back first.
Plans are fitted to the workers present, because partition counts, broadcast and repartition choices depend on the worker count. Scaling is driven by queued work and memory, not CPU, because analytical load is bursty and memory bound.
Workload isolation
Create separate warehouses when one workload must not consume another workload's concurrency or memory. A common production layout is:
ingestion: VegaFlow loads and scheduled transformations;bi-serving: dashboards and semantic tools;data-science: notebooks and exploratory SQL;app-serving: latency-sensitive product queries.
Isolation does not grant data access. Users and service principals still need permission on every referenced catalog and object. Select the warehouse per session; see choose a warehouse.
Concurrency and queues
When a warehouse is busy, statements wait in its queue, report the state queued, and add workers as described above. A statement's timeout includes its time in the queue and while the warehouse starts. Use a dedicated warehouse for bursty dashboard traffic rather than letting it compete with long transformations.
Inspect query history for queue time, warehouse wait, execution time, bytes scanned, spill and cancellation reason before resizing. More compute helps parallel work; it cannot fix unnecessary scans, severe data skew or an unbounded result.
Idle time
Set the idle time long enough to avoid cycling between closely spaced jobs: the first query after a stop waits for the warehouse to start. Scheduled VegaFlow work can start its target warehouse before execution. Applications that need continuous low latency should use a longer idle time and enough minimum workers for their steady load.