Runs and observability

Execution states, task history, logs, timelines, cancellation, retry and retention.

VegaFlow preserves the distinction between a pipeline execution, a run within that execution and the evidence emitted by compute. Use the parent execution for overall state and the child history/timeline for diagnosis.

QuickFlow runs

Before starting a manual run, validate both connections and confirm that the selected cluster is ready or can auto-resume. Start the run from the QuickFlow and open its run history.

Run lists and details are scoped to the parent QuickFlow, organization and workspace. Keep the returned run ID as the handle for that execution and follow its state to completion; request acceptance alone does not establish success.

A cluster transition or worker delay can leave work pending without making the definition invalid. Check cluster readiness, capacity and permissions before retrying. See the QuickFlow lifecycle for stream state, cancellation and failed-stream recovery.

Pipeline execution API

GET  /pipelines/{pipeline_id}/executions
POST /pipelines/{pipeline_id}/executions
GET  /pipelines/{pipeline_id}/executions/{execution_id}
POST /pipelines/{pipeline_id}/executions/{execution_id}/cancel
GET  /pipelines/{pipeline_id}/executions/{execution_id}/timeline
GET  /pipelines/{pipeline_id}/executions/{execution_id}/runs/{run_id}/history
POST /pipelines/{pipeline_id}/executions/{execution_id}/runs/{run_id}/retry

State model

queued → preparing → running → succeeded
                      ├──────→ failed
                      └──────→ canceling → canceled

Preparation resolves the immutable version, deployment generation, environment, execution identity, connection access and runtime inputs. Returned API state is authoritative if a connector-specific substatus uses different wording.

Execution timeline

The timeline orders control-plane and compute events with sequence IDs and occurrence times. Use it to answer when work was queued, prepared, started, retried, canceled or completed.

Timeline order is stronger than sorting log timestamps from different machines. Logs are diagnostic streams; the timeline is the lifecycle record.

Compute logs

GET /pipelines/{pipeline_id}/executions/{execution_id}/runs/{run_id}/logs
GET /pipelines/{pipeline_id}/executions/{execution_id}/runs/{run_id}/logs/{log_id}/content

The list returns log metadata such as task identity, stream, byte size, dropped bytes, storage state and occurrence time. Content is fetched separately and can be encoded for safe transport.

Logs have bounded size and retention. dropped_bytes > 0 means the retained content is incomplete. Persist durable business outputs to an approved destination instead of relying on logs as storage.

Cancellation

Cancellation is idempotent. Queued work can stop before compute starts. Running work moves through canceling while VegaFlow asks active tasks to reach a safe stop boundary. External systems may already contain committed effects; cancellation is not a cross-system rollback.

Retry

A retry creates new attempt/execution evidence and retains the failure it follows. Before retrying, determine whether previous task outputs can be reused safely.

Failure	Typical action
Temporary network or capacity failure	Retry when the operation is idempotent.
Expired credential	Rotate the managed secret, validate access, then retry.
Invalid source/configuration	Publish a corrected version and start a new execution.
Partial external write	Reconcile destination state before retry.
Upstream schema change	Review mapping/evolution policy and publish an intentional change.

QuickFlow retry-failed-streams follows the same principle but selects only failed stream work.

Metrics to watch

queue and environment preparation time;
run duration and retry count;
source records/bytes read;
destination records/bytes committed;
per-task or per-stream lag;
log truncation and retention state;
cancellation latency;
schedule/event delivery delay;
recurring errors by connector, pipeline version and environment.

Sensitive data

VegaFlow removes managed secret values from normal definitions and lifecycle records. Application payloads and user code can still print sensitive fields. Configure logging deliberately and avoid emitting credentials, personal data or unrestricted records to stdout/stderr.

Runs and observability

On this page