File upload API

Sending us your customer data via our File Upload API can be easier and more flexible than SFTP or manual file upload. Here we’ll guide you to creating your API token, requesting permission to deliver a file, and sending that file.

File types

You can use Lexer’s File Upload API for CSV or NDJSON file types. The steps for each file type vary quite significantly. If you have a CSV, you can convert a CSV to an NDJSON file using our [.code]csv_to_ndjson[.code] schema.

NDJSON files get processed faster
If converted, your file upload process becomes automated meaning there is less back and forth required to get your data Hub ready. This speeds up the process, meaning you can see your data in the Lexer Hub even sooner. If you want to convert your files, please follow the steps indicated below - Skip to Uploading NDJSON instructions

Uploading CSV files

If you're unable to convert your CSV file to NDJSON, you can still upload your CSV file by following these steps.

  1. Create an Activity API token.
  2. Authenticate and request permission to send your file. This will generate a URL that will be used to upload your file. 
  3. Use this URL to send us your data file.

API Tokens

You can find detailed instructions about how to set up an API Token in our Authentication and API token creation article. Make sure you select the Activity API Type when generating a token.

Authenticate and request permission to send your file

To successfully authenticate and run your file transfer you will need to use the following command, replacing the API token and File name.

Make sure your file name uses the following naming convention:
- For a CSV: {content_descriptor}-{date}-{version}.csv
- For a JSON: {specification_name}-{date}-{version}.json
File names are case-sensitive, and may not contain special characters other than an hyphen '-', or an underscore '_'.

Run the following command after replacing the [.code]API_token[.code] with your API code (created in step 1). Replace the [.code]File_Name[.code] section with the name of your .csv or .json file (make sure to include the appropriate suffix eg. .json).

curl -XPOST -H 'Auth-Api-Token: API_token' https://api.lexer.io/v1/uploads -H 'Content-type: application/json' -d '{"filename": "File_Name"}'

For example, when using a JSON file you would use the following command:

 # Request
curl -XPOST -H 'Auth-Api-Token: 12345678-1234-1234-1234-123456789' https://api.lexer.io/v1/uploads -H 'Content-type: application/json' -d '{"filename": "{specification_name}-{date}-{version}.json"}'

A successful request will return a response that looks like this:

# Response
{ id: 123, url: "https://lexer-client-upload-au..." }


The URL field created is where you will be sending your file. Copy the URL for use in the next step.

Send your data file

To send your file, run the command below with the [.code]URL[.code] inserted from the response above and your [.code]File_name[.code].

curl -XPUT "URL" --upload-file File_Name

For example, when using a JSON file you would use the following:

 curl -XPUT "https://lexer-client-upload-au..." --upload-file {specification_name}- {date}-{version}.json 

Done! Our team will receive a notification that your file has been received, and we will reach out with any questions or issues. We advise getting in touch to let us know you have sent the file.

File Details
• Maximum file size is 5GB.
• File format preference is NDJSON.
• JSON or CSV can be sent if unable to convert to NDJSON.

Troubleshooting

Please note that the URL received from first request contains [.code]\u0026[.code] in place of [.code]&[.code]. Having the ampersand represented as hexadecimal can cause the second request to return an error similar to that below.

<?xml version="1.0" encoding="UTF-8"?><Error><Code>AuthorizationQueryParametersError</Code><Message>X-Amz-Algorithm only supports "AWS4-HMAC-SHA256"</Message><RequestId>...</RequestId><HostId>...+.../....=</HostId></Error>

This can be avoided by saving the URL in a variable as demonstrated in the workflow below.

url=$(curl -XPOST -H 'Auth-Api-Token:<TOKEN>' https://api.lexer.io/v1/uploads -H 'Content-type: application/json' -d '{"filename": "<FILE_NAME>"}' | jq -r .url)
curl -XPUT "${url}" --upload-file <FILE_NAME>

Uploading NDJSON or JSON files

Uploading NDJSON files via the File Upload API using the following steps, allows for automatic loading of your data. To enable automatic loading it is critical to include the dataset ID and record type in the file upload request.

Have an NDJSON file ready to upload? Follow these steps: 

  1. Ensure the data you'd like to upload matches one of Lexer’s predefined schemas Lexer schema's.
  2. Find the Dataset ID for the dataset you'd like to upload your files to. You can find instructions on how to use Datasets here. If you need to create a new dataset, you can find the instructions here.
  1. Requesting and uploading files can be done using the script found in our [.code]file_upload_api schema[.code].  Alternatively, you can use your own code to produce a URL that you can use to upload a file. You can find instructions on how to do this here
  2. To check how the data ingestion is progressing you can view your file upload in Logs. This will allow you to see the status of the job, the potential statuses are: Success, Pending, In Progress or Failed
  1. ‍You can then check to see whether the upload was successful in Datasets.

             a) Open Manage > Datasets

             b) Open the target dataset. 

             c) Click “View” in the top right of the page. 

             d) Open the Jobs tab to see the status of the most recent job.

             e) Click on the job to open up the Job View.

  1. The Job View provides you with additional information about the job that can help you confirm a successful upload. 

If everything looks correct you should be able to utilize this data in the Hub.

Python Implementation

Below is a python implementation of the approach to uploading files with the File Upload API.

#!/usr/bin/env python3
import requests
TOKEN = "<TOKEN>"
FILE_PATH = "<FILE_NAME>"

def request_presigned_url(token, file_path):
    url = "https://api.lexer.io/v1/uploads"
    payload = {"filename": file_path}
    headers = {"Auth-Api-Token": token}
    response = requests.request("POST", url, json=payload, headers=headers)
    return response.json()
    
def upload_file(url, file_path):
    response = requests.put(url, data=open(file_path, "r").read())
    return response.status_code
    
presigned_url = request_presigned_url(TOKEN, FILE_PATH)
upload_status = upload_file(presigned_url["url"], FILE_PATH)
print(upload_status)

This is the simplest example. For more detail, you can look at our repository here.

Other options?

There are other options for file transfer that might be worth considering:

Its always a good idea to discuss with your Success manager if you are unsure which method would work best.

That's a wrap

That's it for uploading API files! If you get stuck, try following the steps above and please don't hesitate to reach out to Lexer Support at support@lexer.io if you have any questions.

Updated:
October 14, 2022
Did this page help you?
Thank you! Your feedback has been received!
Oops! Something went wrong while submitting the form, for assistance please contact support@lexer.io