Pipeline runs

A run processes one video file through one version of a pipeline. This page describes how a run and its steps progress, how to cancel and retry runs, and how to decide reviews.

Follow a run

getPipelineRun returns the run with every step of its pipeline version, ordered by depth and then by step key. A step that needs no other step has depth 0, and any other step is one deeper than the deepest step it needs, so each step comes after the steps it needs:

import { Client, getPipelineRun } from "@editframe/api/pipelines";

const client = new Client(process.env.EDITFRAME_API_KEY);
const run = await getPipelineRun(client, runId);

console.log(run.status, run.error);
for (const step of run.steps) {
  console.log(step.step_key, step.status, step.attempt, step.watermark_ms);
}

npx editframe pipelines runs get RUN_ID --watch
npx editframe pipelines runs list --status failed --pipeline-id PIPELINE_ID

A step that has not been scheduled yet has the status pending and an id of null. For a segment step, watermark_ms is how far through the media its output is final.

listPipelineRuns returns runs newest first, and takes status, pipeline_id, file_id, group_id and metadata filters. For notifications instead of polling, subscribe to the pipeline webhook topics.

Run statuses

succeeded, failed, filtered and cancelled are final. isPipelineRunTerminal(run) returns true for them.

What filtered means

A final step is a step that no other step needs. A filter or a review stops its branch when a filter's predicate was false, a reviewer rejected, or a review expired with on_expiry: "reject". The stopped step is skipped, and every step that needs it, directly or indirectly, is also skipped.

A run is filtered when every branch stopped, so no final step succeeded. filtered is not an error, and the outputs of the steps that succeeded remain available.

A run whose file reaches at least one final step ends succeeded, even when other branches stopped. A pipeline with two complementary filters ends succeeded when the taken branch finishes. The steps of the other branch are skipped, and each sends pipeline_step.skipped.

Step statuses

Automatic retries

Each step, and each window of a segment step, has 3 attempts. After a retryable failure, Editframe waits 10 seconds, doubles the wait for each later attempt up to 10 minutes, and starts the next attempt. A longer Retry-After from your endpoint replaces the wait. Timeouts, connection errors and 408, 429 and 5xx endpoint responses are retryable. A transcribe, sample_frames or detect_scenes attempt whose decoder was stopped, for example because the worker ran out of memory, fails with the code decoder_killed and is retryable. A file that the step cannot decode fails it with decode_failed, without a retry.

When a step fails without a retry, Editframe cancels the unfinished steps and the run fails. run.error has the message, the code and the step_key of the failed step.

Cancel a run

import { cancelPipelineRun } from "@editframe/api/pipelines";

await cancelPipelineRun(client, runId);

npx editframe pipelines runs cancel RUN_ID

The request returns 202 with the run. The run becomes cancelled asynchronously: unfinished steps are cancelled, and no other steps start. A run that has succeeded, failed or ended filtered returns 409 run_not_cancellable.

Retry a run

import { retryPipelineRun } from "@editframe/api/pipelines";

await retryPipelineRun(client, runId);

npx editframe pipelines runs retry RUN_ID --watch

Only failed and cancelled runs can be retried. Other runs return 409 run_not_retryable. A retry runs the failed and cancelled steps and windows again, each with 3 more attempts. Steps and windows that succeeded keep their outputs. A review that failed when it expired opens a new review task.

The request returns 202 with the run, and the run continues asynchronously. When the pipeline is paused, the retried run stays pending until you resume the pipeline. Repeated retry requests for the same failure count as one retry. A retried run of a completed group makes the group closed again until the run finishes.

Reviews

A review step opens a review task and waits in waiting_review. List the open reviews, oldest first:

import { listPipelineReviews, getPipelineReview } from "@editframe/api/pipelines";

const { reviews } = await listPipelineReviews(client, { status: "pending" });
const review = await getPipelineReview(client, reviews[0].id);

console.log(review.instructions, review.expires_at, review.input_namespaces);

npx editframe pipelines reviews list --status pending
npx editframe pipelines reviews get REVIEW_ID

A review has the status pending, approved, rejected or expired, the run_id, file_id and step it belongs to, the instructions, on_expiry and expires_at of the step, and the recorded decision. input_namespaces lists the namespaces of the steps the review step needs, directly or indirectly, in the order of the run's steps. Read their outputs with getPipelineRunOutputs(client, review.run_id, namespace).

Approve

An approval can correct annotations. The corrected annotations, and every kind in reviewed_kinds, replace the annotations of those kinds from earlier steps for every later step. A kind in reviewed_kinds without annotations leaves later steps with none of that kind. Kinds that you do not name keep their original annotations.

import { approvePipelineReview } from "@editframe/api/pipelines";

await approvePipelineReview(client, reviewId, {
  metadata_patch: {
    annotations: [
      {
        kind: "face",
        start_ms: 846000,
        end_ms: 849000,
        keyframes: [
          { t_ms: 846000, x: 0.41, y: 0.22, w: 0.12, h: 0.18 },
          { t_ms: 849000, x: 0.45, y: 0.21, w: 0.12, h: 0.18 },
        ],
      },
    ],
    reviewed_kinds: ["face", "plate"],
  },
});

npx editframe pipelines reviews approve REVIEW_ID --patch corrections.json

The review step succeeds. Its output has the corrected annotations and the metadata decision, decided_by, decided_at, review_task_id and reviewed_kinds.

Reject

import { rejectPipelineReview } from "@editframe/api/pipelines";

await rejectPipelineReview(client, reviewId, { reason: "Not dashcam footage" });

npx editframe pipelines reviews reject REVIEW_ID --reason "Not dashcam footage"

A rejection skips the review step and every step after it. The run ends filtered unless another branch reaches a final step. reason has at most 2000 characters.

Decision rules

The HTTP route is POST /api/v1/pipeline-reviews/:id/decision with { "decision": "approved", "metadata_patch": … } or { "decision": "rejected", "reason": … }. It returns 202 with the review. The review stays pending until the run applies the decision.

When nobody decides before expires_at, the step's on_expiry setting applies. See Review steps.

Dashboard

When pipelines are enabled for your organization, the dashboard navigation has a Pipelines section with the Pipelines, Runs, Groups, Reviews and Endpoints pages. The pages update while runs are active.

• Pipelines shows, for the last 24 hours, 7 days or 30 days, how many runs finished and succeeded, how many are running, and how many reviews wait for a decision. Under Needs attention it groups failed runs by pipeline, step and error code, and links to them. Each pipeline lists its steps, its mix of finished runs and its median run time. A paused pipeline has the label Paused.
• A pipeline's page draws the steps of one version as stations. Each station shows how many runs are queued, working or waiting at the step now, and how the step ended in the chosen window: succeeded, failed, filtered out, rejected, expired or not run. A note names the step with the most waiting work. Select a station to open its runs, or a count under Steps to open the runs whose step is in that state. Choose a version to see its flow and definition. Admins and editors can run the pipeline on a video file: choose the file and a version, the latest by default, and confirm. They can also pause and resume the pipeline. The confirmation step states how many pending runs will wait and how many running runs finish their current steps and then wait. A paused pipeline shows who paused it and when. The page lists each pause and resume, with who made it, in the change history.
• On the Runs, Groups, run and group pages, the runs and steps that a pause holds show Waiting: pipeline paused.
• Runs lists runs newest first. Filter them by status, pipeline, step and step status, and creation time. Admins and editors can select up to 100 runs to cancel or retry them together, or retry every failed run that matches the filters, up to the newest 500.
• Groups lists your groups newest first. Filter them by status and pipeline. Each group shows its status, pipeline version, external_id, metadata, a bar of its runs by status, the runs that are running and pending now against max_in_flight, and when it was created, closed or completed. The Groups link on a pipeline's page opens the groups of that pipeline.
• A group's page shows the progress of its runs and the numbers of the group report: queue wait, run duration, input minutes, usage, and failures by step with their top error codes. Each failed step links to the runs of the group that failed at that step. The page also lists the most recent failed runs, the pending runs that wait only because of max_in_flight, and the change history with who made each change. Admins and editors can close an open group, change max_in_flight, cancel the unfinished runs and retry the failed runs, up to the newest 500 runs. Each action has a confirmation step that states its effect and the number of runs.
• A video file's page has a Pipelines section. It lists the newest runs of the file, links to all of them, and shows the file's current outputs by namespace, with the number of outputs and the run that produced them. Admins and editors can run a pipeline on the file: choose the pipeline and a version, the latest by default, and confirm.
• A run started in the dashboard is admitted as a run created with POST /api/v1/pipeline-runs is: it needs a video file, an archived pipeline refuses it, and on a paused pipeline it waits as pending.
• By default, a run started in the dashboard has no API key and sends no webhooks. To send them, choose an API key under Webhooks in the confirmation step. The run then sends its events to that key's webhook URL, for the topics that the key subscribes to, as a run that the key creates through the API does. Renders of the run's redact_render steps also belong to that key. The list shows the API keys of your organization that have a webhook URL and are not expired. A key that subscribes to no run, step or review topic cannot be chosen.
• A run's page shows who started the run, from the dashboard or through the API, the API key that receives its webhooks, the step graph, a timeline of the steps, and for a selected step its attempts, errors and events. It also lists the webhook events about the run and its steps, with their delivery attempts. Admins and editors can cancel or retry the run.
• Reviews lists open reviews oldest first, with their instructions and expiry. On a review's page, admins and editors can remove annotations, correct their times and labels, and then approve, or reject with an optional reason.
• Endpoints lists your endpoints with their header names, limits, requests in flight now and the outcomes of their requests in the last 24 hours and 7 days. An endpoint's page shows, for the chosen window, how many requests succeeded, failed or timed out, the failure codes, the median and p95 request time, the callbacks that wait now, and the recent failures with links to their runs. It also lists the pipelines and steps that use the endpoint, and the history of limit changes with who made them. Admins and editors can change the request timeout, the callback timeout and the maximum concurrency after a confirmation step. The dashboard never shows the signing secret or header values.

Members with the Reader role can view the pages but cannot act.