Quantemplate pipelines can be reused to perform the same set of transformations on data sources which are regularly updated. Once you’ve configured a Quantemplate pipeline to transform a particular set of data sources, it’s easy to feed in identically formatted new data as you receive it.
For example, each month you receive a dataset from three different sources. You have configured a pipeline to transform the data to the desired output format. When new data comes in next month, it can be fed through the pipeline and exported to the data repo.
To run new data through a pipeline:
If you’re creating a pipeline that will be frequently re-used, consider using this structure:
If something changes in one of your source data formats, your pipeline may produce unexpected results. Quantemplate allows you to edit transformations to accommodate the new data format.
For example, one of your data providers has moved to a new system, generating slightly different column header names. To fix this, go into the Map Column Headers operation and configure the correct mappings.