VegaDB overview

VegaDB is Vegalake’s distributed database and SQL warehouse. It gives applications, analysts and data teams one endpoint for managed data and federated DuckLake, Apache Iceberg and Delta Lake tables, reached through Arrow Flight SQL, Spark Connect or an HTTP SQL API.

What you get

Warehouses and data are independent

A warehouse supplies compute for a workload. Catalogs, schemas, tables, permissions and history belong to the workspace. This means you can stop idle compute without deleting data and run separate warehouses for ingestion, BI, data science and high-concurrency applications.

                       ┌─ BI warehouse ───── dashboards
Governed data layer ───├─ ELT warehouse ──── transformations
                       └─ App warehouse ───── APIs and products

Each warehouse has its own size, minimum and maximum workers and idle time. All of them are reached through the workspace's stable endpoint, which does not change when a warehouse starts, stops or scales.

Query across clouds and formats

Registered catalogs make external data look like ordinary qualified SQL objects. Credentials stay in Vegalake Secrets and access is governed at workspace, catalog, schema and table level.

SELECT c.segment, sum(o.net_amount) AS revenue
FROM aws_iceberg.sales.orders o
JOIN azure_ducklake.crm.customers c USING (customer_id)
WHERE o.ordered_at >= current_date - INTERVAL '30 days'
GROUP BY c.segment
ORDER BY revenue DESC;

VegaDB keeps a query consistent with the selected table snapshots and versions, including when one statement spans several catalogs.

SQL surface

The SQL language includes joins and subqueries, CTEs and recursive CTEs, set operations, window functions, QUALIFY, PIVOT and UNPIVOT, ASOF JOIN, lateral expressions, prepared parameters, nested types, JSON, arrays, maps, structs, approximate aggregates and vector-distance functions.

Start with the SQL reference for syntax, types, statements, functions and compatibility notes.

Query provenance

Query requests accept an optional provenance boolean, defaulting to false. Set provenance: true on a query or one-shot query run to request complete read provenance. Query responses include a dependencies array of workspace relation IDs when requested and established; otherwise it is empty. This is separate from permission to run a query or retrieve its results.

Provenance covers read-only queries over fully qualified managed tables and views, including supported built-in scalar and aggregate functions, windows, nested values, clock expressions such as now() and current_date, and expression subqueries. Function calls may be unqualified or use main; calls qualified with another schema or a catalog have no provenance. Dependencies include each view and every base table it reads, with up to eight view levels and 200 distinct relations in total. If your warehouse defines any SQL macro or user-defined type, or your session has a temporary one, query results carry no provenance, even when the query does not use it. Top-level unqualified relation names, CTEs, SHOW/SUMMARIZE/DESCRIBE, table functions such as range and unnest, time-travel queries, cycles and unresolved or ambiguous names also carry no provenance. Functions that inspect SQL, query plans, settings, variables, files or catalog information, and unsupported list-aggregation or row-expanding helpers, have no provenance. Statements that create or change objects in the warehouse's internal memory catalog are refused (UNSUPPORTED_SQL). Switching to that catalog with USE and whole-database COPY FROM DATABASE statements are also refused; USE requires identifiers rather than string targets. Writes that use a relation alias or struct named memory, or two-part names beginning with memory., are also refused; use another alias or qualify managed names fully as vegadb.<schema>.<table>. A statement that contains the unquoted words COPY FROM DATABASE in that order is refused even when it only reads, so quote such identifiers. An empty dependencies array means provenance is unavailable, not that the returned data is non-sensitive. Turing requests provenance for its queries and exports and considers classification facts on both views and tables. Published asset views carry the MODELED badge; a view that reads another asset's stored result may show MIXED. Turing keeps conservative sensitivity handling when provenance is unavailable.