Pipelines workspace

Stages panel

View and edit your data transformations as a sequence of collapsible stages and operations.

Stage

Structure your data transformations with Join, Union and Transform stages.

Operation

Within a Transform stage, use sequences of Operations to apply transformations to your data.

Input tabs

In the Pipeline view, the main panel contains tabs to manage pipeline inputs: uploads, reference datasets and partner data, allowing you to add, view and manage files used in the current pipeline. See the pipeline inputs page for more details.

Pipeline Outputs

A summary of all outputs of the pipeline, including validation reports, mapping reports, output datasets, the run log and results of previous runs.

Action bar

Run

Run the pipeline to perform the data transformations and create the output datasets, validation report and mapping reports. Running the pipeline will open the Run Log popup, from where the results of previous pipeline runs can be viewed, shared or downloaded. Learn more pipeline runs.

Queue

If the pipeline has been set to Auto Run, or can be run via API, it can receive multiple instructions to begin a run. Upcoming runs are placed in a queue, which can be managed via the queue popup.

History

Every time a user makes an edit or causes a trace to run, the pipeline is automatically saved and a new version is recorded. The history panel records the versions and allows rollback to previous versions. Learn more about history.

Sharing

Pipeline owners can click the share button on the top right to share a pipeline with members of their organisation. Learn more about sharing.

Create and name pipelines

Click the folder button to view and manage all your pipelines.

Click the + button to create a new pipeline.

Click the pipeline name to rename your pipeline.

Learn more about creating, naming and managing pipelines.