Batch data versus API requests
Why use batch data uploads?
Our Batch API allows you to upload millions of records with a single upload. There are lots of reasons you might want to use this instead of our standard API requests. The main function is pretty simple, if you don’t have the capability to provide events in real time, batch uploads might be a more effective solution for you.
NDJSON formatting
The preference for formatting is Newline delimited JSON (NDJSON). We prefer this format for a few reasons. It helps with validation of your data, and allows for some amount of malformed data without completely rejecting the file. This provides a great redundancy. The specification for NDJSON formatting can be found on Github.
For more information about the formatting requirements for your data, have a look at our Data section on Learn. The Data formatting and validation page would be a great place to start.
File zipping and upload
Our file preference is a Gzipped NDJSON file. Gzip is a free compression software that has a great compression ratio, making your files smaller, and reducing the time to transfer your data. To learn more about Gzip and access the free software, check it out here. We will also accept regular NDJSON files if you’d prefer.
Get your batch data uploaded into the Hub
Throughout the process you will need to work closely with the Lexer team in order to get everything uploaded and ready to go.
The process you will need to follow:
- Create an .ndjson file that abides by the NDJSON formatting. Other formatting requirements for the file can be found in our Data section on Learn.
- Ensure the file is Gzipped.
- Once your file is Gzipped and ready to send to the team, follow either the SFTP, or AWS S3 process to upload your file.
- Please make sure to let your Success Manager know you would like to batch upload as part of the onboarding process.
- If this data transfer is part of a regular process, set up a schedule to transfer the deltas. This schedule should be outlined in your solution design.
Where to from here?
Batch data uploads could be the perfect solution to help you upload large volumes of data infrequently, but there might be an option that fits your business even better. Make sure you have a chat with your Success Manager so they can help you find the best solution. This helps you get the most out of your data as quickly as possible.
