Append Metadata

About Append Metadata

Append metadata allows dataset and pipeline metadata to be appended to the output data as new columns. The fields that can currently be appended are:

• Pipeline ID
  The unique reference Quantemplate gives to every pipeline and is used in the pipeline’s URL. This is useful for audit purposes where it is necessary to prove that a dataset originated from a specific pipeline.
• Pipeline name
  The name of the pipeline at run time. This is useful to augment the Pipeline ID above.
• Pipeline run date
  The date and/or time of the start of the pipeline run. Date-time will be recorded in the timezone set in this operation, which may be different from the user’s local timezone. This ensures consistent results when working with teams across multiple time zones, or across daylight-saving time periods. The field can optionally include the date, time and timezone in the same column.
• Pipeline run number
  The pipeline run number which generated the output dataset. This is useful to link a dataset to a specific run number, enabling review of validations and seeing time of run.
• Stage input filename
  The name of the input file entering the stage. This is useful for tracing the input file or extracting a company name or date from the file name.

If there are more metadata fields which would be useful to you, please contact support.

Using the Append Metadata operation

To add and configure the Append Metadata operation:

1. From the operations menu, select the Append Metadata operation.
2. Select the metadata fields to be added.
3. If needed, edit the names of the new columns which will be created.
4. Click the plus button to add another metadata field. Each field can only be added once.
5. To remove a metadata field, click the cross button which appears on hovering a field block.

Example

Append the filename to multiple stage inputs, then union the files.

Append Metadata operation

Field: Filename
Column name: Input file

Union stage

Feed both outputs from Append Metadata into a Union stage.

Note: the Append Metadata operation replaces the Append Filename operation. Append Filename will continue to be supported in existing pipelines but should not be added to new pipelines.