Quantemplate pipelines can be reused to perform the same
set of transformations on data sources which are regularly updated. Once you’ve
configured a Quantemplate pipeline to transform a particular set of data sources,
it’s easy to feed in identically formatted new data as you receive it.
For example, each month you receive a dataset from three
different sources. You have configured a pipeline to transform the data to the desired
output format. When new data comes in next month, it can be fed through the pipeline
and exported to the data repo.
To feed new data into a pipeline:
1.
Go to the inputs tab and remove the old pipeline inputs. Click on the [-] button to remove.
2.
Upload the latest data.
3.
Now thread the new data through the pipeline.
Go to the pipeline tab, open the first stage and select the newly uploaded files as
stage inputs. The stage outputs are generated automatically.
4.
Configure the inputs for the next stage, until
all stages have their inputs configured. Union stages create a single output based
on the stage name, so for all subsequent stages the new data is threaded through
automatically.
5.
Run the pipeline to generate your output
data, then export it to your data repo.
Best Practice
If you’re creating a pipeline that will be frequently re-used,
consider using this structure:
1.
Create a
transform
stage to remove rows, detect
headers and map to a common schema.
2.
Add a
union
stage to combine the source data.
This creates a single output with the output name based on the stage name, meaning
the rest of the pipeline can flow in the new data automatically.
3.
Add subsequent transform and join stages.
If your data transformation requirements mean you’re not able
to use this kind of pipeline structure, then the new data can still be threaded through
from stage to stage individually. Be sure to select the right inputs for each stage.
Dealing with source data format changes
If something changes in one of your source data formats, your
pipeline may produce unexpected results. Quantemplate allows you to fix this by editing
your transformations to accommodate the new data format.
For example, one of your data providers has moved to a new
system, generating slightly different column header names. To fix this, go into the
column mapping stage and configure the correct mappings.
