The key file required for the data upload is the manifest file. This file contains information on what and where to upload. The file is in JSON format. It could be created in any simple text editor or using a variety of online JSON editors.
Important.
- The manifest file must be named manifest.json since Timeline accepts only manifest files with this name.
- All files to be uploaded must be listed in a single manifest.json file and zipped along with it.
- Place manifest.json only in the root of the ZIP file, and data files can be placed in subfolders.
- The manifest file must contain either repositoryToken or projectToken but not both.
For instructions on setting up the manifest, select one of the options below, depending on where you plan to upload the data.
Manifest with the repositoryToken to upload data to a repository
The table below describes the nodes that are in the manifest file.
| Node |
Description |
| repositoryToken |
Encrypted strings identifying the repository you are uploading data to.
To obtain this string:
- Open your repository
To do this, click Data in the left bar. The Repository opens. If you have multiple repositories, select the one you need. To do this, go to Details > General, click Change, and select the desired.
- Go to the Data sources tab.
- Select Scheduled file upload.
- Click Show and copy token to clipboard.
|
| email |
A list of email addresses, separated by commas.
The reports about successful or failed uploads will be sent to the specified emails. Timeline also always sends an email to the repository owner.
|
| commands |
The ordered list of various commands which will be executed once the upload package is received and unzipped.
Available commands with arguments:
- upload
Parses the file(s) and puts the data into the table.
Arguments:
- file or fileRegex – name of the file from the ZIP. Only CSV files are supported to upload.
See a table with the options description below.
- table – name of repository table into which the data from file(s) will be uploaded. If the table doesn’t exist, it will be created.
- keepOldData – if true, Timeline keeps the old data in the selected table.
Note. The argument is optional.
- clone
Copies a repository table and all data in it.
Arguments:
- src – name of the table to be copied.
- dst – name of the table into which the data from the file will be copied. If the table doesn’t exist, it will be created.
- todo-list
Executes a to-do list from the repository.
You can create to-do lists within a Timeline Repository, which are essentially a list of data transformation functions. These typically end in uploading that data to a project. If a data file does not require any transformation, the to-do list can specify that it be uploaded directly to a project using the predefined project table mapping. For more information about To-do list, see ETL in the Cloud.
Important. To upload file(s) to a project you must first perform a manual upload and map the data. Timeline will automatically save a mapping after the data file has been successfully uploaded to a project. When uploading files of the same type, Timeline maps the uploaded data automatically. If no previously saved mapping is found the upload fails.
- Arguments:
- table – name of the table on which the to-do list will be executed.
- list – name of an existing to-do list.
|
Arguments for assigning file names in manifest
To perform upload commands, you should specify the file name(s) from the ZIP. Follow the descriptions in the table, to choose and use a file argument that suits your case.
| Argument |
Description |
Example |
| file |
Use this argument when you have a fixed file name to upload. As a value, you can specify a constant with the name of the uploaded file. |
|
| fileRegex |
With this argument, you can enter a regular expression which specifies a rule for multiple file names. The program uploads in sequence all files from the ZIP whose names fit the rule.
Note. https://regex101.com/ offers an online service that helps you create and parse regular expressions, explaining the purpose of each operator. There you can also test your regular expression and get detailed information on how it works. Simply enter your regular expression in the REGULAR EXPRESSION field and see it parsed and explained in the EXPLANATION pane.
|
Regular expression:
Argument in the manifest file:
|
Manifest with the projectToken to upload data to a project
The table below describes the nodes that are in the manifest file.
| Node |
Description |
| projectToken |
Encrypted strings identifying the project you are uploading data to.
To obtain this string:
- Open your project and click the project name.
The Project details window opens.
- Go to the Data sources tab and select Scheduled file upload.
- Click Show and copy token to clipboard.
Important. If you expect to upload large datasets to your project, be sure to turn on Automatically switch to Large Data Mode in Project details > Data sources.
|
| email |
A list of email addresses, separated by commas.
The reports about successful or failed uploads will be sent to the specified emails. Timeline also always sends an email to the project owner.
|
| commands |
A command to be executed once the upload package is received and unzipped.
Available commands with arguments:
- upload
The action parses file(s) and puts the data to the existing project using the mapping predefined for this project.
Arguments:
- file or fileRegex – name of the file from the ZIP. Only CSV files are supported to upload.
See a table with the options description below.
- clearProject – if true, old data from the project will be deleted before new data is loaded.
Note. The argument is optional.
- mapNewColumnsAsAttributes – if true, all optional columns except the mandatory columns Timeline ID, Timestamp, and Event Name columns are added to the project as attributes.
Note. The argument is optional.
Example. Use this argument if you plan to upload a file with more optional columns in the table than defined in the preconfigured mapping. When uploaded, data from the new columns become new attributes. If you omit the mapNewColumnsAsAttributes argument from the manifest file, any optional columns that were not previously mapped will not be uploaded.
Important. To upload file(s) to a project you must first perform a manual upload and map the data. Timeline will automatically save a mapping after the data file has been successfully uploaded to a project. When uploading files of the same type, Timeline maps the uploaded data automatically. If no previously saved mapping is found the upload fails.
|
Arguments for assigning file names in manifest
To perform upload commands, you should specify the file name(s) from the ZIP. Follow the descriptions in the table, to choose and use a file argument that suits your case.
| Argument |
Description |
Example |
| file |
Use this argument when you have a fixed file name to upload. As a value, you can specify a constant with the name of the uploaded file. |
|
| fileRegex |
With this argument, you can enter a regular expression which specifies a rule for multiple file names. The program uploads in sequence all files from the ZIP whose names fit the rule.
Note. https://regex101.com/ offers an online service that helps you create and parse regular expressions, explaining the purpose of each operator. There you can also test your regular expression and get detailed information on how it works. Simply enter your regular expression in the REGULAR EXPRESSION field and see it parsed and explained in the EXPLANATION pane.
|
Regular expression:
Argument in the manifest file:
|
Examples of manifest
The following manifest file takes file events.csv from the ZIP, parses it into the existing repository table mytable, then copies the table into yourtable (existing data in the table is kept), and then applies to-do list mylist to it. A report will be sent to the repository owner, yourname@company.com, and yourcolleague@company.com.
The next example parses files whose names match the regular expression ER_.*\.csv and uploads them to a project identified by the token. Old data from the project will be deleted.