Lakehouse federation

VegaDB queries open lakehouse data in place. Register a storage integration, add a catalog or Delta table registry, grant access and attach it to a SQL session. You do not need to copy the data into a proprietary format first.

Supported combinations

Write availability depends on catalog ownership and your account release. External Iceberg, external DuckLake and registered Delta are read-only unless the console explicitly shows a writable capability.

Operation matrix

“Yes” means VegaDB can plan the operation when the principal and storage integration allow it. It does not turn a read-only external registration into a writer.

Storage schemes

The registered prefix is a security boundary. A catalog cannot redirect a query to an object outside the integration scope.

1. Create a storage integration

In Workspace → Data → Storage integrations, choose the provider and the narrowest storage prefix that contains the table data. Store credentials in Secrets and reference the secret from the integration.

Provider     Amazon S3 / S3-compatible
Location     s3://company-lake/analytics/
Region       me-central-1
Credential   analytics-reader-role
Endpoint     leave empty for AWS; set for compatible storage

Use Test access before saving. The test checks identity, endpoint, TLS, list access and one bounded object read from the configured prefix.

Provider option reference

SQL alternative: create a secret directly

Administrators can provide credentials through SQL with CREATE PERSISTENT SECRET. This is useful for initial setup and controlled deployment automation when a storage integration has not already been created in the console.

The statement must run over TLS using the PostgreSQL simple-query protocol. It cannot be prepared, parameterized or executed inside a transaction. The principal needs SECRET_ADMIN in the current workspace.

CREATE PERSISTENT SECRET analytics_s3 (
  TYPE S3,
  PROVIDER CONFIG,
  KEY_ID '<access-key-id>',
  SECRET '<secret-access-key>',
  SESSION_TOKEN '<temporary-session-token>',
  REGION 'me-central-1',
  ENDPOINT 'https://s3.me-central-1.amazonaws.com',
  PATH_STYLE FALSE,
  SCOPE 's3://company-lake/analytics/'
);

Catalog credentials

An external DuckLake catalog usually needs a PostgreSQL credential for its metadata database in addition to its object-storage secret:

CREATE PERSISTENT SECRET finance_catalog_postgres (
  TYPE POSTGRESQL,
  PROVIDER CONFIG,
  HOST 'catalog.example.com',
  PORT 5432,
  DATABASE 'finance_lake',
  USERNAME '<catalog-user>',
  PASSWORD '<catalog-password>',
  TLS_MODE VERIFY_FULL,
  CA_REFERENCE 'finance-catalog-ca'
);

For an Iceberg REST catalog using OAuth 2.0 client credentials:

CREATE PERSISTENT SECRET iceberg_catalog_oauth (
  TYPE ICEBERG_OAUTH2,
  PROVIDER CONFIG,
  CLIENT_ID '<oauth-client-id>',
  CLIENT_SECRET '<oauth-client-secret>',
  TOKEN_ENDPOINT 'https://identity.example.com/oauth/token'
);

If the catalog supplies a bearer token directly, use the alternate form:

CREATE PERSISTENT SECRET iceberg_catalog_token (
  TYPE ICEBERG_OAUTH2,
  PROVIDER CONFIG,
  BEARER_TOKEN '<bearer-token>'
);

Rotate, inspect and revoke

CREATE OR REPLACE adds a new active version without exposing or overwriting the encrypted history:

CREATE OR REPLACE PERSISTENT SECRET analytics_s3 (
  TYPE S3,
  PROVIDER CONFIG,
  KEY_ID '<new-access-key-id>',
  SECRET '<new-secret-access-key>',
  REGION 'me-central-1',
  SCOPE 's3://company-lake/analytics/'
);

SHOW SECRETS;
DROP PERSISTENT SECRET IF EXISTS analytics_s3;

SHOW SECRETS returns metadata such as name, type, state, active version, timestamps and a redacted scope. There is no SQL statement that returns a decrypted value. DROP PERSISTENT SECRET revokes the secret; it does not reveal or reuse its previous value.

2. Register a format

Name                 finance_ducklake
Metadata connection  finance-catalog-postgres
Storage integration  finance-azure-lake
Default database     finance

3. Grant and attach

Grant USAGE on the registered catalog and schema plus SELECT on the tables. Then attach the registered catalog under a session alias:

ATTACH CATALOG prod_iceberg AS iceberg;
ATTACH CATALOG finance_ducklake AS finance;
ATTACH CATALOG prod_delta AS delta;

SHOW DATABASES;
SHOW ALL TABLES;

The attachment lasts for the SQL session. The workspace’s default managed catalog is attached automatically.

An attachment alias must be unique within the session. Use stable aliases in application SQL and fully qualify alias.schema.table. DETACH alias removes the session attachment without deleting the workspace registration or cloud data.

DETACH iceberg;

4. Query one or several catalogs

SELECT d.region, count(*) AS orders, sum(i.net_amount) AS revenue
FROM iceberg.sales.orders i
JOIN delta.crm.customers d USING (customer_id)
JOIN finance.reference.fx_rates f
  ON f.currency = i.currency AND f.rate_date = CAST(i.ordered_at AS DATE)
GROUP BY d.region
ORDER BY revenue DESC;

Use fully qualified catalog.schema.table names in shared queries and views. VegaDB resolves one consistent version for each referenced table before execution.

Planning and consistency

• Catalog authorization and table metadata are resolved before data tasks begin.
• Each table in one statement is pinned to the snapshot or version selected for that statement.
• A cross-catalog query is a consistent set of per-table versions; it is not a transaction spanning the external catalog services.
• Predicate and column pushdown are applied only when file metadata and expression semantics prove the result remains correct.
• Remote object listings, catalog calls and file reads are retried only where the request is safe to repeat. Expired credentials and changed snapshots surface as errors instead of switching versions silently.

For reproducible processing, select explicit snapshots or versions. A query against current tables can observe a later version when it is executed again.

Time travel

Iceberg catalogs preserve snapshots and named references. Delta registries preserve table versions. The exact selectors available to your catalog appear in the table’s History panel and SQL help.

-- Examples; use an ID/version shown by the table history surface.
SELECT * FROM iceberg.sales.orders AT (SNAPSHOT => 918273645);
SELECT * FROM delta.sales.orders AT (VERSION => 42);

Troubleshooting

Operational checklist

1. Scope the storage integration to the smallest usable prefix.
2. Test catalog authentication and object access independently.
3. Register the format and confirm table locations remain inside the integration.
4. Grant catalog, schema and table privileges to roles rather than individual users where possible.
5. Attach with a stable alias and run DESCRIBE before production queries.
6. Validate time-travel retention for jobs that pin historical versions.
7. Monitor catalog latency, files scanned, bytes read, retries and authorization failures.