Append Metadata

About Append Metadata

Append metadata allows dataset and pipeline metadata to be appended to the output data as new columns. The fields that can be appended are:

Section	Function	Description
Pipeline	Pipeline Auto Run by	If a pipeline run was initiated by an Auto Run trigger updating, record the trigger datasets or feeds that caused the run.
	Pipeline ID	The ID is the unique reference Quantemplate gives to every pipeline and is used in the pipeline’s URL. This is useful for audit purposes where it is necessary to prove that a dataset originated from a specific pipeline.
	Pipeline input row count	The row counts of inputs to a stage that come from feeds, uploads or reference datasets. Inputs from Stage Outputs are not counted. Rows are counted at the point at which they enter the stage. Use in combination with Stage Input Filename to identify the input whose rows are being counted.
	Pipeline name	The name of the pipeline. This is useful to augment the Pipeline ID above.
	Pipeline revision number	The revision number of the pipeline at run time. This is useful for audit purposes where it is necessary to ascertain that which pipeline revision number produced a dataset.
	Pipeline run by user	Name and/or email of user who ran the pipeline.
	Pipeline run date	The date and/or time of the start of the pipeline run. Date-time will be recorded in the timezone set in this operation, which may be different from the user’s local timezone. This ensures consistent results when working with teams across multiple time zones, or across daylight-saving time periods. The field can optionally include the date, time and timezone in the same column.
	Pipeline run number	The pipeline run number which generated the output dataset. This is useful to link a dataset to a specific run number, enabling review of validations and seeing time of run.
	Stage input filename	The name of the input file entering the stage. This is useful for tracing the input file or extracting a company name or date from the file name.
Feeds	Deadline status	When using a schedule, records whether a submission-subsection was submitted on time or was overdue. If overdue, returns ‘Overdue’. If not overdue, returns a blank.
	Feed ID	The ID is the unique reference Quantemplate gives to every feed and is used in the feed’s URL. This is useful for audit purposes where it is necessary to prove that a dataset originated from a specific feed.
	Feed name	The name of the feed. This is useful to augment the feed ID above.
	Period end date	If the feed uses a schedule to create submissions, this records the date-time of end of the submission’s schedule period.
	Period start date	If the feed uses a schedule to create submissions, this records the date-time of start of the submission’s schedule period.
	Resubmission status	If data from a feed was a resubmission, returns the value ‘Resubmission’, or otherwise a blank.
	Schedule end date	If the feed uses a schedule to create submissions, this records the date-time the schedule ceases running.
	Schedule name	If the feed uses a schedule to create submissions, this records the name of the schedule.
	Schedule start date	If the feed uses a schedule to create submissions, this records the the date-time the schedule started running.
	Stage input is additive / only latest	If auto update has been set in the stage inputs, this records whether the ‘Additive’ or ‘Only latest’ has been set. Can be used to indicate whether data is Inception-to-Date or Transactional. Returns the values ‘Additive’ or ‘OnlyLatest’.
	Submission created date	Date-time the submission was created. This could include when the submission was created as a draft.
	Submission data version	Version number of submission data.
	Submission first requested date	Date-time the data was first requested.
	Submission name	Name of the submission.
	Submission requested by user	Name and/or email of user who created the feed.
	Submission uploaded date	Date-time the data was uploaded.
	Submitted by user	Name and/or email of the user who submitted the data.
	Subsection name	Name of the submission-subsection.
Reference datasets	Dataset information	Select one of the reference datasets used in the pipeline then toggle on options to write the ID, name, row count, updated date or revision number of the dataset as columns of metadata. To get the metadata for additional datasets, add another ‘Dataset information’ function to the Append Metadata operation.

If there are more metadata fields which would be useful to you, please contact support.

Using the Append Metadata operation

To add and configure the Append Metadata operation:

From the operations menu, select the Append Metadata operation.
Select the metadata fields to be added.
If needed, edit the names of the new columns which will be created.
Click the plus button to add another metadata field. Each field can only be added once.
To remove a metadata field, click the cross button which appears on hovering a field block.

Example

Append the filename to multiple stage inputs, then union the files.

Input name: Acme-bdx-2022-08
Policy No	Premium
H24788	100
H84729	120
H89722	130

Input name: NewCo-bdx-2022-08
Policy No	Premium
8723XYZ	140
8738GHI	150
8264GUI	160

Append Metadata operation

Field: Filename
Column name: Input file

Union stage

Feed both outputs from Append Metadata into a Union stage.

Output from Union stage
Policy No	Premium	Input file
H24788	100	Acme-bdx-2022-08
H84729	110	Acme-bdx-2022-08
H89722	130	Acme-bdx-2022-08
8723XYZ	140	NewCo-bdx-2022-08
8738GHI	150	NewCo-bdx-2022-08
8264GUI	160	NewCo-bdx-2022-08