File upload API

Sending us your customer data via our File Upload API can be easier and more flexible than SFTP or manual file upload. Here we’ll guide you to creating your API token, requesting permission to deliver a file, and sending that file.

File types

You can use Lexer’s File Upload API for CSV or NDJSON file types. The steps for each file type vary quite significantly. If you have a CSV, you can convert a CSV to an NDJSON file using our [.code]csv_to_ndjson[.code] schema.

NDJSON files get processed faster
If converted, your file upload process becomes automated meaning there is less back and forth required to get your data Hub ready. This speeds up the process, meaning you can see your data in the Lexer Hub even sooner. If you want to convert your files, please follow the steps indicated below - Skip to Uploading NDJSON instructions

Uploading CSV files

If you're unable to convert your CSV file to NDJSON, you can still upload your CSV file by following these steps.

  1. Create an Activity API token.
  2. Authenticate and request permission to send your file. This will generate a URL that will be used to upload your file. 
  3. Use this URL to send us your data file.

API Tokens

You can find detailed instructions about how to set up an API Token in our Authentication and API token creation article. Make sure you select the Activity API Type when generating a token.

Authenticate and request permission to send your file

To successfully authenticate and run your file transfer you will need to use the following command, replacing the API token and File name.

Make sure your file name uses the following naming convention:
- For a CSV: {content_descriptor}-{date}-{version}.csv
- For a JSON: {specification_name}-{date}-{version}.json
File names are case-sensitive, and may not contain special characters other than an hyphen '-', or an underscore '_'.

Run the following command after replacing the [.code]API_token[.code] with your API code (created in step 1). Replace the [.code]File_Name[.code] section with the name of your .csv or .json file (make sure to include the appropriate suffix eg. .json).

curl -XPOST -H 'Auth-Api-Token: API_token' https://api.lexer.io/v1/uploads -H 'Content-type: application/json' -d '{"filename": "File_Name"}'

For example, when using a JSON file you would use the following command:

 # Request
curl -XPOST -H 'Auth-Api-Token: 12345678-1234-1234-1234-123456789' https://api.lexer.io/v1/uploads -H 'Content-type: application/json' -d '{"filename": "{specification_name}-{date}-{version}.json"}'

A successful request will return a response that looks like this:

# Response
{ id: 123, url: "https://lexer-client-upload-au..." }


The URL field created is where you will be sending your file. Copy the URL for use in the next step.

Send your data file

To send your file, run the command below with the [.code]URL[.code] inserted from the response above and your [.code]File_name[.code].

curl -XPUT "URL" --upload-file File_Name

For example, when using a JSON file you would use the following:

 curl -XPUT "https://lexer-client-upload-au..." --upload-file {specification_name}- {date}-{version}.json 

Done! Our team will receive a notification that your file has been received, and we will reach out with any questions or issues. We advise getting in touch to let us know you have sent the file.

File Details
• Maximum file size is 5GB.
• File format preference is NDJSON.
• JSON or CSV can be sent if unable to convert to NDJSON.

Troubleshooting

Please note that the URL received from first request contains [.code]\u0026[.code] in place of [.code]&[.code]. Having the ampersand represented as hexadecimal can cause the second request to return an error similar to that below.

<?xml version="1.0" encoding="UTF-8"?><Error><Code>AuthorizationQueryParametersError</Code><Message>X-Amz-Algorithm only supports "AWS4-HMAC-SHA256"</Message><RequestId>...</RequestId><HostId>...+.../....=</HostId></Error>

This can be avoided by saving the URL in a variable as demonstrated in the workflow below.

url=$(curl -XPOST -H 'Auth-Api-Token:<TOKEN>' https://api.lexer.io/v1/uploads -H 'Content-type: application/json' -d '{"filename": "<FILE_NAME>"}' | jq -r .url)
curl -XPUT "${url}" --upload-file <FILE_NAME>

Uploading NDJSON or JSON files

Uploading NDJSON files via the File Upload API using the following steps, allows for automatic loading of your data. To enable automatic loading it is critical to include the dataset ID and record type in the file upload request.

Have an NDJSON file ready to upload? Follow these steps: 

  1. Ensure the data you'd like to upload matches one of Lexer’s predefined schemas Lexer schema's.
  2. Find the Dataset ID for the dataset you'd like to upload your files to. You can find instructions on how to use Datasets here. If you need to create a new dataset, you can find the instructions here.
  1. Requesting and uploading files can be done using the script found in our [.code]file_upload_api schema[.code].  Alternatively, you can use your own code to produce a URL that you can use to upload a file. You can find instructions on how to do this here
  2. To check how the data ingestion is progressing you can view your file upload in Logs. This will allow you to see the status of the job, the potential statuses are: Success, Pending, In Progress or Failed
  1. ‍You can then check to see whether the upload was successful in Datasets.

             a) Open Manage > Datasets

             b) Open the target dataset. 

             c) Click “View” in the top right of the page. 

             d) Open the Jobs tab to see the status of the most recent job.

             e) Click on the job to open up the Job View.

  1. The Job View provides you with additional information about the job that can help you confirm a successful upload. 

If everything looks correct you should be able to utilize this data in the Hub.

Python Implementation

Below is a python implementation of the approach to uploading files with the File Upload API.

#!/usr/bin/env python3
import requests
TOKEN = "<TOKEN>"
FILE_PATH = "<FILE_NAME>"

def request_presigned_url(token, file_path):
    url = "https://api.lexer.io/v1/uploads"
    payload = {"filename": file_path}
    headers = {"Auth-Api-Token": token}
    response = requests.request("POST", url, json=payload, headers=headers)
    return response.json()
    
def upload_file(url, file_path):
    response = requests.put(url, data=open(file_path, "r").read())
    return response.status_code
    
presigned_url = request_presigned_url(TOKEN, FILE_PATH)
upload_status = upload_file(presigned_url["url"], FILE_PATH)
print(upload_status)

This is the simplest example. For more detail, you can look at our repository here.

Other options?

There are other options for file transfer that might be worth considering:

Its always a good idea to discuss with your Success manager if you are unsure which method would work best.

That's a wrap

That's it for File Upload API. If you get stuck, try following the steps above and please don't hesitate to reach out to Lexer Support at support@lexer.io if you have any questions.

For more information on all available APIs visit our API Reference Guide.
Updated:
August 29, 2024
Did this page help you?
Thank you! Your feedback has been received!
Oops! Something went wrong while submitting the form, for assistance please contact support@lexer.io
Welcome to Lexer!
Fundamentals
Getting started
Javascript Tag basics
Data Platform
Javascript Tag
Data in the CDXP
Fundamentals
Getting started
Our glossary
Fundamentals
Getting started
Integrations overview
Fundamentals
Integrations
Onboarding data with Lexer
Data Platform
Data Onboarding
Data onboarding process
Data Platform
Data Onboarding
Importing CSV data
Data Platform
Data Onboarding
Importing JSON data
Data Platform
Data Onboarding
Secure file uploads
Data Platform
Data Onboarding
SFTP uploads and exports
Data Platform
Data Onboarding
AWS S3 uploads and exports
Data Platform
Data Onboarding
Lexer's attributes
Data Platform
Data Types
Predictive attributes
Data Platform
Data Types
Lexer API overview
Data Platform
Developer APIs
Understanding APIs
Data Platform
Developer APIs
API authentication
Data Platform
Developer APIs
API rate limits
Data Platform
Developer APIs
Bulk write API
Data Platform
Developer APIs
Profile read API
Data Platform
Developer APIs
Activity overview
Insights
Activity
Team report
Insights
Activity
Cases report
Insights
Activity
NPS report
Insights
Activity
Listen overview
Insights
Listen
Searching in Listen
Insights
Listen
Tier filters
Insights
Listen
Saved dives
Insights
Listen
Boolean search
Insights
Listen
Listen CSV exports
Insights
Listen
Visualize overview
Insights
Visualize
Curate image feed
Insights
Visualize
Respond overview
Engagement
Respond
Identity Resolution
Fundamentals
Getting started
My account
Fundamentals
Setup
Manage team
Fundamentals
Setup
Group permissions
Fundamentals
Setup
Classifications
Fundamentals
Setup
Out of the box segments
Fundamentals
Setup
Lexi your AI companion
Fundamentals
Getting started
Browser guide
Fundamentals
Security
Corporate networks
Fundamentals
Security
Multi-factor authentication
Fundamentals
Security
Single sign-on
Fundamentals
Security
Lexer's ultimate troubleshooting guide
Fundamentals
Troubleshooting
Troubleshooting tech issues
Fundamentals
Troubleshooting
Troubleshooting integrations
Fundamentals
Troubleshooting
Troubleshooting Activate
Fundamentals
Troubleshooting
Troubleshooting Respond
Fundamentals
Troubleshooting
Help! My data is missing from the Hub
Fundamentals
Troubleshooting
Lexer data specification
Data Platform
Data Specification
Customer data specification
Data Platform
Data Specification
Commerce data specification
Data Platform
Data Specification
Marketing data specification
Data Platform
Data Specification
Compliance data specification
Data Platform
Data Specification
Data formatting and validation
Data Platform
Data Specification
Product imagery
Data Platform
Data Specification
Currency conversion
Data Platform
Data Specification
Dataset management
Data Platform
Data Management
Getting started with Logs
Data Platform
Data Management
Respond chatbot API
Data Platform
Developer APIs
Activity API
Data Platform
Developer APIs
Lexer Forms overview
Data Platform
Forms
Form builder workflow
Data Platform
Forms
Form conditional logic
Data Platform
Forms
Form settings
Data Platform
Forms
Form segmentation
Data Platform
Forms
Form response analysis
Data Platform
Forms
Hidden form fields
Data Platform
Forms
Javascript Tag technical guide
Data Platform
Javascript Tag
Javascript Tag use cases
Data Platform
Javascript Tag
Javascript Tag: Shopify Custom Pixel
Data Platform
Javascript Tag
CRM data
Data Platform
Data Types
Transaction data
Data Platform
Data Types
Email engagement data
Data Platform
Data Types
Experian data enrichment
Data Platform
Data Types
Customer Service data
Data Platform
Data Specification
GDPR and CCPA requests
Fundamentals
Compliance
File upload API
Data Platform
Developer APIs
Segment overview
Insights
Segment
Creating segments
Insights
Segment
Smart Search
Insights
Segment
Profile tab
Insights
Segment
Event Explorer
Insights
Segment
Attribute value types
Data Platform
Data Types
Compare segments
Insights
Compare
Compare attributes
Insights
Compare
Activate overview
Engagement
Activate
Segment activations
Engagement
Activate
Event activations
Engagement
Activate
Activation field mapping
Engagement
Activate
Audience splits
Engagement
Activate
A/B splits
Engagement
Activate
Control group splits
Engagement
Activate
Inbox filtering
Engagement
Respond
Ignored Senders
Engagement
Respond
Forms in Respond
Engagement
Respond
Workflow states
Engagement
Respond
Bulk changes
Engagement
Respond
Scheduled replies
Engagement
Respond
Message templates
Engagement
Respond
Finding conversations
Engagement
Respond
Customer profiles
Engagement
Respond
Grouped messages
Engagement
Respond
Automation rules
Engagement
Respond
Redact messages
Engagement
Respond
Routing customer replies
Engagement
Respond
Interact with comments
Engagement
Respond
Respond workflow tips
Engagement
Respond
Contact Reporting
Engagement
Contact
Contact Queue Filters
Engagement
Contact
Contact WhatsApp Queues
Engagement
Contact
Schedule overview
Engagement
Schedule
Serve overview
Engagement
Serve
Installing Serve
Engagement
Serve
Serve user management
Engagement
Serve
Forms in Serve
Engagement
Serve
Configuring Serve
Engagement
Serve
Serve reports
Engagement
Serve
Serve Currency
Engagement
Serve
Serve POS QR Code
Engagement
Serve
Activate CSV export
Engagement
Activate
Track overview
Insights
Track
Track Tables
Insights
Track
Report overview
Measure
Listen
Lexer Product Recommender → Klaviyo Integration
Grow
Lexer Product Recommender → Klaviyo Integration
Where do your best customers live? Use geographic insights to maximize media spend and conversion
Grow
Where do your best customers live? Use geographic insights to maximize media spend and conversion
Suppression audiences: How to use them and maximize your media effectiveness
Enrich
Suppression audiences: How to use them and maximize your media effectiveness
Measure campaign effectiveness: Track your KPIs and report on campaign performance
Enrich
Measure campaign effectiveness: Track your KPIs and report on campaign performance
Converting prospects: Acquire customers from your current, engaged prospect base (and save your ad spend!)
Acquire
Converting prospects: Acquire customers from your current, engaged prospect base (and save your ad spend!)
Understanding customer intent: Effective message personalization for buyers who shop outside their gender
Grow
Understanding customer intent: Effective message personalization for buyers who shop outside their gender
Deepen customer understanding: Use third-party data to identify insights for personalized messaging and increased engagement
Grow
Deepen customer understanding: Use third-party data to identify insights for personalized messaging and increased engagement
Basket analysis: Increasing customer lifetime value through targeted product bundling
Grow
Basket analysis: Increasing customer lifetime value through targeted product bundling
Persona building: Personalize your messaging for maximum return on your campaign investments
Grow
Persona building: Personalize your messaging for maximum return on your campaign investments
Driving customer loyalty: Identify, keep, and grow your most loyal customers
Retain
Driving customer loyalty: Identify, keep, and grow your most loyal customers
Increasing customer lifetime value: Upselling strategies
Grow
Increasing customer lifetime value: Upselling strategies
Reactivating lapsed customers: Retention and growth
Retain
Reactivating lapsed customers: Retention and growth
Welcome offer strategies: Using the Hub for analysis
Acquire
Welcome offer strategies: Using the Hub for analysis
Creating customer forms: Best practices
Enrich
Creating customer forms: Best practices
Connecting your MarTech stack with Lexer!
Enrich
Connecting your MarTech stack with Lexer!
Identifying win-back customers using the Second Last Order: Date attribute
Retain
Identifying win-back customers using the Second Last Order: Date attribute
Understanding customers' purchasing habits: Order Sequence filter
Grow
Understanding customers' purchasing habits: Order Sequence filter
Maximizing customer communications in Respond
Retain
Maximizing customer communications in Respond
Using Product Recommendation in the Hub to encourage repeat purchases
Grow
Using Product Recommendation in the Hub to encourage repeat purchases
Encouraging customer loyalty using birthday campaigns
Retain
Encouraging customer loyalty using birthday campaigns
Increasing customer lifetime value: Converting your one-time buyers into two-time buyers
Grow
Increasing customer lifetime value: Converting your one-time buyers into two-time buyers
Collecting zero-party data using Lexer Forms
Enrich
Collecting zero-party data using Lexer Forms
Measuring your data over time with Track
Enrich
Measuring your data over time with Track
UTM Creation Guidelines
UTM Creation Guidelines
UTM Creation Guidelines
Brand Assets
Brand Assets
Brand Assets
GTM guide to creating marketing materials with Lexer
GTM guide to creating marketing materials with Lexer
GTM guide to creating marketing materials with Lexer
How to use the Lexer Form Builder
How to use the Lexer Form Builder
How to use the Lexer Form Builder
Explore our standard integrations
Explore our standard integrations
Explore our standard integrations
Operating a CDP Practice LP
Operating a CDP Practice LP
Operating a CDP Practice LP
LP Sales Enablement
LP Sales Enablement
LP Sales Enablement
Video Training Courses LP
Video Training Courses LP
Video Training Courses LP
Purpose and Mission
Purpose and Mission
Purpose and Mission
Brand & Marketing LP
Brand & Marketing LP
Brand & Marketing LP
Stage 3 - QBRs and Customer insights
Stage 3 - QBRs and Customer insights
Stage 3 - QBRs and Customer insights
Client Onboarding Journey
Client Onboarding Journey
Client Onboarding Journey
Start your CDP Practice
Start your CDP Practice
Start your CDP Practice
Lexer Messaging
Lexer Messaging
Lexer Messaging
Pain point conversations
Pain point conversations
Pain point conversations
Lexer Ideal Customer Profile & Positioning
Lexer Ideal Customer Profile & Positioning
Lexer Ideal Customer Profile & Positioning
Referral Partner Program Overview
Referral Partner Program Overview
Referral Partner Program Overview
Demo Hub Guide
Demo Hub Guide
Demo Hub Guide