Visual pipelines
Build workflows on a canvas, run them with their own run settings, trigger them by schedule, webhook or app event, and test or retry them step by step.
A visual pipeline is a workflow you build on a canvas: a trigger, then steps that call connections, run Python, map data, branch, loop, wait for approval or call other pipelines. It is a workspace resource, like a QuickFlow. It needs no pipeline project, environment or deployment.
Use a visual pipeline for event-driven work across apps and data. Use a pipeline project when the work lives in a Git repository, and a QuickFlow for a plain source-to-destination sync.
Pipelines and folders
Visual pipelines are listed under VegaFlow › Visual pipelines. Folders organise the list:
- Every workspace has a Default folder. It cannot be renamed, moved or deleted.
- Folders nest up to three levels. Deleting a folder moves its pipelines and subfolders to Default.
- Folders carry no permissions. Members who can create visual pipelines can manage folders. Moving a pipeline needs
updateon that pipeline.
The list shows how each pipeline is doing. Four tiles count the active pipelines, runs and failed runs in the last 24 hours, and approvals waiting for a decision. Each row shows the description, folder, apps the pipeline uses, what starts it, its status, a bar for its last 30 production runs (oldest on the left), the last run and the owner. Test runs are not counted. Filter by status (All, Active, Off, Drafts, Failing: pipelines whose latest finished run failed), search by name or description, or show only the pipelines you created. The counts cover the pipelines loaded in the list, 200 at a time; a count with a plus sign (200+) is a lower bound.
New pipeline asks for a name and a folder and starts from a blank canvas or a template. Templates lists the built-in recipes and your workspace's templates.
Pipeline names are unique within a workspace. Deleting a pipeline archives it: it leaves the list, its triggers stop, and its name can be reused. Its runs stay readable to people who hold read on it until retention removes them.
A pipeline can be exported as JSON and imported into a workspace. The export holds the graph and the portable parts of its run settings.
Steps
Add a step from the + on the canvas. A step's panel edits its fields; a field takes a fixed value, a value from an earlier step, or a formula.
| Group | Steps |
|---|---|
| Apps | An action of a connected app, such as creating an issue or inserting a row. |
| Logic | If / else (condition rows, AND within a group and OR between groups, or an expression), Route by value (named routes; see below), Filter on a connection (see below), For each (one item at a time or several at once, and whether one failed item stops the loop or is collected), Wait, Paths meet (joins the paths that ran, by picking fields or by matching two lists on a field). |
| Data | Edit fields (shape a new object) and Python code (code, inputs, a declared output). |
| People | Approval, Run a pipeline and Ask an agent. |
Insert data places a value from an earlier step, from the trigger, or the current loop item. Text fields can hold values inside the text, shown as chips, such as [critical] Disk full on {{host}}.
Route by value
A route step compares one value with each route. It sends the run down the first route that matches, or down every route that matches, one after the other in the order shown; then its output is a list with one item per route that ran. When nothing matches, the run goes down a Default path (left empty, it carries on with the next step), ends quietly, or fails with your message.
Filter on a connection
A filter sits on the connection between two steps and is drawn as a label on the line. The run continues only when its conditions are true. Otherwise the run ends quietly: it succeeds with no output and shows as Filtered in tests and on the Runs page. A filter can be turned into an If / else.
Paths meet
Match by a field pairs the items of two lists that share a value, for example issues and owners by service. Keep only the items that match, everything from the first list, or everything from both. The output is pairs, a list of { "left": …, "right": … }; a side with no partner is null. The same function is available in formulas as merge_by(left, left_field, right, right_field, keep) with keep one of "matches", "left" or "all". Only text, numbers and true/false values match; empty values, objects and lists never do.
Approval
An approval asks people to decide before the run goes on. Pick the people and teams who decide (nobody means anyone who can run the pipeline), and whether someone other than the person who started the run must decide. How many must approve is any one, everyone (when every decider is a named person), or at least a number from 2 to 20. One rejection rejects, whatever this is set to, and a person counts once. The step's answers come from the approval that completed the count.
An approval can expire. When nobody decides in time it can reject, fail the run, take its own path, or approve automatically; an automatic approval is recorded as decided by the pipeline, not by a person. Approvals appear in Approvals and on the run in the Runs page, with who has approved so far.
Formulas
fx on a value opens a formula editor. Formulas can use:
| Kind | Functions and operators |
|---|---|
| Compare | =, !=, <, <=, >, >=, and, or, not, contains |
| Text | concat, upper, lower, trim, replace, split, join, length |
| Lists | first, last, unique, append, length, merge_by |
| Numbers | add, sub, mul, div, round |
| Other | if(condition, then, otherwise), coalesce, now() |
now() is the time the run started and is the same in every step and every replay of the run.
Wait
A wait holds the run for a length of time, or until a date and time. A time that has already passed does not wait. Waiting runs show as waiting and hold no compute.
Draft, compile and publish
The editor works on the pipeline's draft. Saving keeps the draft (see Saving); Compile checks it and lists any issues above the canvas. Publish opens Review changes, which compares the draft with the live version step by step and field by field, marks new and edited steps on the canvas and takes an optional one-line note (up to 200 characters). Publishing there turns the compiled draft into a new numbered version. Versions are immutable: runs started earlier keep the version they started with.
Runs, schedules, webhooks and app events always use the active published version. Editing the draft changes nothing until you publish.
Saving
The header shows Unsaved changes, Saving… or the time of the last save. The draft is saved:
- when you select Save or press Ctrl+S (⌘S on a Mac);
- at most a minute after you change something;
- before a test, before Publish and before you ask Turing a question from the editor;
- when you switch to another browser tab or leave the editor. The browser warns before you close a page with unsaved changes.
Undo and redo work on your own edits in the editor and do not depend on saving.
Two people can edit the same draft. When someone else saved since you opened it, your save merges their changes with yours, step by step: if you changed different steps, both are kept, the canvas shows the merged draft, and a message names who you were merged with. Moving steps on the canvas never conflicts. If you both changed the same step, or the same pipeline setting, Steps were changed in two places lists each one: choose Keep mine or Keep theirs for each and select Save. Cancel keeps your changes in the editor, unsaved.
New steps get IDs with a short random ending, such as mapping_k3f9, so two people adding steps at the same time do not collide.
Versions
Versions lists every published version with its note and who published it. Open one to see it read only on the canvas, compare it with the draft, or Restore as draft. Restoring replaces the draft and can be undone in the editor. A version published before notes were kept shows no note and compares without step names.
When a step fails
By default a failing step stops the run. An app, Python, approval or call step can instead carry on with empty outputs, or take an error path that runs other steps. Each of those steps can also retry when it fails (up to 999 more times, with a fixed or doubling wait of up to a day) and has a time limit that counts every try and the waits between them.
Build with Turing
Turing, VegaLake's AI assistant, can explain, build and change visual pipelines. Open it from Turing in the editor header. It sees the open pipeline and the selected step, so "this step" works, and offers Explain this pipeline, Add error handling to app steps and Find steps that are not ready. It reads step settings and test samples, never secret values.
Turing never changes a pipeline on its own. When you ask for a change, it checks its draft against the compiler and sends a proposal. Show on canvas opens the editor with the proposal drawn over the draft: new steps marked NEW, changed steps EDIT and removed steps REMOVED, with a list of the changes. Editing is off while a proposal is shown.
- Review each steps through the changes one at a time and centres each on the canvas.
- Apply all applies the whole proposal to the draft. Ctrl+Z (⌘Z) undoes all of it at once. If the draft changed since Turing made the proposal, the changes are merged as for a save; a step that both changed opens the same choice, with Keep Turing's and Keep the draft's.
- Discard drops the proposal.
Applying a proposal changes the draft only. Turing cannot publish a pipeline, turn it on or run it; you review and publish as usual.
To start a pipeline from a description, choose Describe it in New pipeline, write what it should do and select Draft. Turing creates the pipeline in the folder you chose and links to it in the conversation.
Asking Turing to propose changes needs update on the pipeline; applying one needs update too. See Turing conversations.
Settings page
The Settings page has these parts:
| Part | What it holds |
|---|---|
| Details | Name, description, who created the pipeline and when, and its ID. |
| Publishing | Whether the pipeline is active and which version its trigger starts, whether the draft has changes not published, and the published versions with who published them and their notes. |
| Import and export | Export the pipeline as JSON. |
| Archive | Archive the pipeline. |
| How it runs | The run settings below. |
Settings apply from the next run and are not part of a version, so they need no publish. Changes to Details and How it runs are saved together from the save bar.
Run settings
Each pipeline has its own run settings, edited under How it runs on its Settings page:
| Setting | Meaning |
|---|---|
| Compute | A dedicated cluster or serverless compute. |
| Run-as identity | The service principal the pipeline runs as. |
| Allowed connections | The connections steps may use. |
| Allowed secrets | The secrets steps may read. |
| Egress | Where steps may send network traffic. |
| Intermediate retention | How long intermediate files are kept, in days. |
Saving checks that you may use each cluster, connection and secret and act as the service principal. Every save creates a new immutable snapshot. A run keeps the snapshot it started with, including when it is retried, so changing settings never changes a run that already started.
A pipeline cannot run until its run settings are saved.
Triggers
The trigger is the first step on the canvas and is published with the version. Automatic triggers fire only while the pipeline is Active: published, with its Active switch on. Manual runs and calls from other pipelines are allowed whenever a version is published.
| Trigger | Starts a run when |
|---|---|
| Manual | Someone starts it. An input schema on the trigger becomes the run form. |
| Schedule | A cron expression or interval (at least one minute) comes due, in the chosen time zone. |
| Webhook | A request reaches the pipeline's webhook URL. |
| App event | A connection reports an event, by polling or by the app's webhook. |
Pipelines can also be started by a call step in another pipeline. The caller's run-as identity needs execute on the called pipeline, and the called pipeline runs with its own run settings.
Schedules
A schedule sets what happens when a run is still active at the next time (overlap: skip, queue or allow) and when times were missed while the pipeline was off (missed: skip, run the latest, or catch up). Turning the pipeline off pauses the schedule.
Webhooks
Creating a webhook gives the pipeline a URL and a signing secret. The secret is shown once; rotate it to get a new one. Each delivery starts a run with this event:
{
"body": { "...": "the request body" },
"headers": { "content-type": "application/json" },
"received_at": "2026-10-27T09:00:00Z",
"delivery_id": "8f9c..."
}App events
Pick a connection, the app's event and its settings. Polling events take an interval from 5 seconds to one day. The connection, and the signing secret when the app needs one, must be allowed in the run settings. Events arrive as:
{ "id": "evt_123", "type": "issue.opened", "time": "2026-10-27T09:00:00Z", "data": { "...": "..." } }Events received while the pipeline is off are dropped, so turning it on never replays a backlog.
Sample events
Listen for a test event on a webhook or app-event trigger keeps the first event received in the next two minutes as the trigger's sample. The event still starts a run if the pipeline is Active. Tests from the draft use the sample when the trigger has no pinned event.
Test from the draft
Tests run the saved draft without publishing it. A test is real: it uses the pipeline's run settings and calls apps for real. Testing needs update on the pipeline.
| Test | What runs |
|---|---|
| Whole pipeline | Every step, from the trigger. |
| From a step | The chosen step and every step after it. |
| Only a step | The chosen step alone. |
Only top-level steps can be chosen. A step inside a loop or branch runs with its container, which can be chosen.
When a test skips earlier steps, each step it skips that a tested step reads from is given an output instead of running:
- its pinned output, if it has one;
- otherwise its output from the most recent test that ran it successfully.
If a needed step has neither, the test is refused and names that step. Pin it, or test from an earlier step.
Pins are kept per step of the draft, by step ID, and survive moving the step. Pin a step's output in the step panel's Test tab. Pinning the trigger sets the test event. Without a pinned or sample event, the editor asks for one.
Each test compiles the draft into a test build. Test builds never become versions: they cannot be published, activated or used by triggers.
While a test runs, the canvas shows each step's state, and the panel below the canvas lists the steps with their output and logs. Steps that were given an output show Reused output. A test that ended at a filter shows as Filtered. Test runs also appear on the Runs page as test runs.
Debug with a past run's data
Debug with a past run's data loads a run's step data into the editor. Change the draft, then test from the failed step: the event and earlier steps' outputs come from that run instead of pins or earlier tests.
Runs, retry and retry from a step
The Runs page lists runs newest first, with what started each, its state and how long ago. Filter by All runs, Failed, Running or waiting or Test runs; Run now starts a run when the pipeline can be started by hand. Open a run to see when it started, how long it took, its version and ID, and the canvas with each step's state. Each step opens its input, output and logs; a loop shows its items as a table. A failed run names the step that failed and links to it in the editor. A run waiting for approval says who can decide and when the request expires, and people who may decide can approve or reject there. A run that ended at a filter shows as Filtered.
From a run you can retry or cancel it, copy its ID, download its output, test the draft with the run's input, or open it in Monitor for the technical view.
A failed run can be retried. The retry is a new run in the same execution, with the failed run's run settings.
- Start from the beginning, or from a top-level step. Starting from a step reuses the failed run's recorded output for the steps before it; the chosen step and everything after it run again. The failed step is selected by default.
- Retry with the same version, or pick another published version, for example one that fixes the failed step.
If an earlier step's output is no longer available, for example a step inside a loop or an output past retention, the retry is refused and names the step.
Permissions
| Permission | Allows |
|---|---|
create (type level) | Create and import pipelines; manage folders. |
list, read | See the pipeline, its draft, versions, runs and outputs. |
update | Edit the draft and run settings, manage triggers and pins, test the draft. |
publish | Publish versions; turn the pipeline on and off. |
execute | Start runs, retry and cancel, decide approvals. |
delete | Archive the pipeline. |
iam:modify_permission | Grant permissions on the pipeline. |
The creator gets every permission on the pipeline. See access control.
API
All routes are under {api-base}/orgs/{org_id}/workspaces/{workspace_id}/vegaflow/visual-pipelines.
| Route | Permission |
|---|---|
GET /, POST / | list / type-level create |
GET /{id}, PATCH /{id}, DELETE /{id} | read / update / delete |
GET /{id}/draft, PUT /{id}/draft, POST /{id}/compile | read / update |
POST /{id}/publish, POST /{id}/enable, POST /{id}/disable | publish |
GET /recent-runs?workflow_id={id},{id} | list |
GET /{id}/versions, GET /{id}/versions/{version_id} | read |
GET /{id}/run-settings, PUT /{id}/run-settings | read / update |
GET /{id}/trigger, POST /{id}/trigger/listen, GET /{id}/trigger/sample | read / update / read |
POST /{id}/trigger/webhook, PATCH /{id}/trigger/webhook, POST /{id}/trigger/webhook/rotate-secret | update |
POST /{id}/executions | execute |
GET /{id}/executions, GET /{id}/executions/{execution_id} | read |
POST /{id}/executions/{execution_id}/runs/{run_id}/retry | execute |
POST /{id}/executions/{execution_id}/cancel | execute |
GET /{id}/approvals, POST /{id}/approvals/{approval_id}/decision | read / execute |
POST /{id}/test-runs | update |
POST /{id}/proposals, GET /{id}/proposals/{proposal_id} | update / read |
POST /{id}/proposals/{proposal_id}/apply, POST /{id}/proposals/{proposal_id}/discard | update |
GET /authoring-catalog, GET /authoring-apps, POST /compile-check, GET /{id}/authoring, POST /{id}/compile-check | type-level list / read |
GET /{id}/draft/pins, PUT /{id}/draft/pins/{node_id}, DELETE /{id}/draft/pins/{node_id} | read / update |
GET /{id}/export, POST /import | read / type-level create |
GET /folders, POST /folders, PATCH /folders/{folder_id}, DELETE /folders/{folder_id} | type-level list / create |
Publish with a note. The body is optional:
POST /orgs/{org_id}/workspaces/{workspace_id}/vegaflow/visual-pipelines/{id}/publish
If-Match: "{draft_version}"
Content-Type: application/json
{ "note": "Handle empty orders" }Save the draft. If-Match is the version you loaded. Send base_source, the source you loaded or last saved, so a save after someone else's merges instead of failing:
PUT /orgs/{org_id}/workspaces/{workspace_id}/vegaflow/visual-pipelines/{id}/draft
If-Match: "{draft_version}"
Content-Type: application/json
{ "source": "{graph JSON}", "base_source": "{graph JSON you loaded}" }When the versions match, the source is saved as sent. When they differ and the changes do not overlap, the merged draft is saved and the response carries "merge": {"merged": true, "with": ["user:{id}"]}. Without base_source, a stale version is refused with 409. Overlapping changes return 409 with code VISUAL_DRAFT_CONFLICT, the conflicts and the current draft:
{
"code": "VISUAL_DRAFT_CONFLICT",
"conflicts": [{ "path": "", "node_id": "build_summary", "title": "Build summary", "kind": "both_changed" }],
"current": { "workflow": { "...": "..." }, "draft": { "resource_version": 7, "...": "..." } }
}kind is both_changed, changed_removed, both_added or field (a pipeline setting, with node_id field:<name>). Retry with If-Match of current.draft.resource_version, the same base_source, and resolutions mapping each conflict to mine or theirs. The key is the node_id, or path/node_id for a step inside a branch, loop or error path:
{ "source": "{graph JSON}", "base_source": "{graph JSON you loaded}", "resolutions": { "build_summary": "mine" } }Resolutions count only against the version they were made for; if someone saved again meanwhile, the save merges again and may return new conflicts. Applying a proposal takes the same optional resolutions in its body and answers the same way; a proposal with a conflict stays pending.
Read the latest runs of up to 50 pipelines at once, as the list does. Each item holds the newest 30 production runs, with more set when there are older ones. Pipelines you cannot read are left out:
GET /orgs/{org_id}/workspaces/{workspace_id}/vegaflow/visual-pipelines/recent-runs?workflow_id={id},{id}Test a draft from a step:
POST /orgs/{org_id}/workspaces/{workspace_id}/vegaflow/visual-pipelines/{id}/test-runs
Content-Type: application/json
{ "mode": "from_step", "node_id": "build_summary" }mode is whole, from_step or step. input sets the test event; without it the trigger's pin, then its sample, is used. seed_run ({"execution_id", "run_id"}) uses a past run's event and outputs instead.
Retry a failed run from a step with a fixed version:
POST /orgs/{org_id}/workspaces/{workspace_id}/vegaflow/visual-pipelines/{id}/executions/{execution_id}/runs/{run_id}/retry
Content-Type: application/json
{ "from_node": "build_summary", "version_id": "{version_id}" }Pin a step's output:
PUT /orgs/{org_id}/workspaces/{workspace_id}/vegaflow/visual-pipelines/{id}/draft/pins/{node_id}
Content-Type: application/json
{ "value": { "total": 3 } }Pinned values are limited to 1 MiB.
Decide an approval. Each person's decision counts once; the approval is decided when enough people approve or anyone rejects:
POST /orgs/{org_id}/workspaces/{workspace_id}/vegaflow/visual-pipelines/{id}/approvals/{approval_id}/decision
If-Match: "{resource_version}"
Content-Type: application/json
{ "decision": "APPROVED", "values": { "likely_cause": "Cache stampede" } }The approval read shows required_approvals and approvals, the people who approved so far with when. A run's read shows "filtered": true when it ended at a filter.