Detect Headers

Quantemplate requires a defined set of column headers to perform downstream operations. When using files uploaded directly to a pipeline, headers will be automatically titled ‘unnamed’, unnamed (0)’, etc. A typical workflow uses Remove Rows to strip out any blank or unwanted rows above the data, then Detect Headers to define one or more rows as the header.

Detect Headers can convert a multi-row header into a single row header and can handle multiple files with varying numbers of header rows.

Using Detect Headers

If needed, use a Remove Rows operation to strip out unwanted rows above the data. Then add a Detect Headers operation from the Operations menu. Enter the number of rows to select and merge into a single header row.

Detect Headers can merge up to 10 rows into the header row.

Example

Take an input file with 2 rows of headers and merge into a single row

Input

unnamed	unnamed (0)	unnamed (1)	unnamed (2)	unnamed (3)	unnamed (4)	unnamed (5)	unnamed (6)
2015				2016
Q1	Q2	Q3	Q4	Q1	Q2	Q3	Q4
27492	20144	17594	85993	104288	49004	40100	38299

Use first 2 rows as the header

Output

2015:Q1	2015:Q2	2015:Q3	2015:Q4	2016:Q1	2016:Q2	2016:Q3	2016:Q4
27492	20144	17594	85993	104288	49004	40100	38299

Ignore inputs with named headers

When enabled, this option reads the headers in the file. If all of them start with ’unnamed‘ the headers will be defined, otherwise the file will be skipped.

Datasets in the Data Repo and Partner Datasets already have their headers defined. If bringing these datasets into a stage alongside uploads to the pipeline – which do not have headers defined – enabling this option will define the headers for the uploads and skip the Data Repo / Partner Data items. If the option is not enabled, the Data Repo / Partner Data items will have their first row(s) of data converted to headers.

If an operation adds a column to a pipeline upload before Detect Headers is applied, the uploaded file will be treated as if its headers have been defined, so will be skipped if this option is enabled. In this case, the option should not be enabled.

Reference Datasets used as a Master Schema for Map Column Headers should have zero rows of data, so do not require this option to be enabled.

Variable header formats

In scenarios where input files have differing numbers of rows in the header, it can sometimes be useful to assume that the first row containing only numbers is the first record in the data, whilst the rows above are headers.

Selecting the option to ‘Interpret non-numeric rows as headers’ will look at the first rows, up to the maximum number defined in the operation, and select them as headers if they contain non-numeric data (i.e. text, blanks). The first row will always be taken as a header, regardless of content

Input 1

unnamed	unnamed (0)	unnamed (1)	unnamed (2)	unnamed (3)	unnamed (4)	unnamed (5)	unnamed (6)
2015-Q1	2015-Q2	2015-Q3	2015-Q4	2016-Q1	2016-Q2	2016-Q3	2016-Q4
27492	20144	17594	85993	104288	49004	40100	38299

Input 2

unnamed	unnamed (0)	unnamed (1)	unnamed (2)	unnamed (3)	unnamed (4)	unnamed (5)	unnamed (6)
2015				2016
Q1	Q2	Q3	Q4	Q1	Q2	Q3	Q4
35322	12344	12345	65322	43234	34466	23456	23455

Use the first 2 rows as the header. Interpret non-numeric rows as headers.

This will take the first row and any subsequent rows which do not have a numeric value as the header, up to a maximum of 2 rows.

Output 1

2015-Q1	2015-Q2	2015-Q3	2015-Q4	2016-Q1	2016-Q2	2016-Q3	2016-Q4
27492	20144	17594	85993	104288	49004	40100	38299

Output 2

2015:Q1	2015:Q2	2015:Q3	2015:Q4	2016:Q1	2016:Q2	2016:Q3	2016:Q4
35322	12344	12345	65322	43234	34466	23456	23455