VegaFlow overview

VegaFlow connects systems and runs the work that turns raw data into products, models and decisions. Use a QuickFlow for a direct source-to-destination sync or a versioned pipeline for a complete data or AI workflow.

VegaFlow records connections, datasets, pipeline versions, runs, owners, environments and lineage in VegaGraph. You can query these records alongside the systems that depend on them.

Explore VegaFlow

Quickstart

Validate PostgreSQL connections, prepare compute and run your first QuickFlow.

Concepts

Choose QuickFlows or pipeline projects and understand versions, environments, runs and evidence.

Connections

Reusable, governed source and destination connections with managed secrets and catalog discovery.

Connector guides

Browse PostgreSQL, GitHub and Airtable by connector type and availability.

QuickFlows

Direct batch or incremental data syncs with mapping, schedules and stream-level recovery.

Pipelines

Versioned DAGs for SQL, Python, data quality, ML training and operational work.

Compute environments

Isolated managed compute that can be sized, started, stopped and observed independently.

Operations

Schedules, triggers, run timelines, logs, cancellation, retry and troubleshooting.

API reference

Workspace-scoped resource groups, request conventions and lifecycle operations.

How data moves through the platform

Applications + databases + files + events
                    │
              VegaFlow connections
                    │
         QuickFlows + versioned pipelines
                    │
        VegaDB tables + models + applications
                    │
             VegaGraph context
       lineage · owners · policy · business impact

Teams can see what ran, what changed, which data product it produced and which downstream systems depend on it.

Reliable by default

Every run points to the version that was executed. Tasks and streams keep their attempts, timing, structured errors and outputs. Cancellation and retry are explicit operations; retrying failed stream work does not silently replay successful streams.

Compute is not the definition

A compute environment can stop, start or change size without changing a QuickFlow or pipeline definition. Use inexpensive compute for periodic ingestion, a larger environment for backfills and isolated capacity for production ML or customer-facing workflows.

Access and governance

Connections, environments, QuickFlows and pipelines are permissioned workspace objects. A service principal needs permission to execute the flow and use each referenced connection and environment. Secrets remain references; they are not copied into definitions, run history or VegaGraph.