Dataset management in the Hub

Datasets in Lexer: An explainer

Lexer’s Dataset Manager gives you the ability to explore the data and high-level statistics from each of your integrated accounts, directly in the Hub.

Each of your datasets contain record types specific to each integrated account that has data flowing into the Hub. These have been transformed and organised into datasets that follow Lexer’s standard schema.

For example, Shopify datasets will contain record types related to Shopify online transactions, including customer records, products, orders, and returns. Whereas Klaviyo datasets will contain record types relating to email events, including customer, emails sent, emails clicked, subscribed, etc.

Organizing data into a dataset

In the diagram below, we have used Klaviyo as an example to illustrate how data makes its way into a dataset from the original Klaviyo source. You are then able to view this dataset within the Lexer Hub.

  1. Klaviyo connects with Lexer via an API, and data is transferred into Lexer via this method.
  2. The raw data that comes in from Klaviyo is stored in an Amazon S3 bucket.
  3. The data from this S3 bucket is then transformed into Lexer's accepted data schema.
  4. The data is then arranged into a dataset than can be accessed and viewed via the Lexer Hub.

So, how can I use datasets?

Having your datasets easily accessible within the Hub has a number of benefits and uses, and we have outlined the use cases below.

Data QA and validation

Lexer’s Dataset Manager lets you quickly view and validate the health of your integration feed, including data continuity, volume, and freshness of data flowing into your datasets. Drilling down even further, you can click into each record to view the payload for individual customer records.

Lexer tip!
The permission to view dataset payloads is managed by your account administrator, as datasets can contain customer PII.

Dataset statistics

Within the Dataset Manager you can also access high-level statistics and charts, which allow you to quickly visualize important metrics for each dataset. These will vary depending on the type of records contained within the dataset.

As we continue to build out our API capabilities, the Dataset Manager will also give you the ability to create your own datasets, which can be written into directly using JSON and CSV uploads!

Finding your datasets in the Hub

You can view your datasets in the Hub by navigating to Manage > Datasets in the top navigation bar.

Please note that access to datasets is managed via Group Permissions and Lexer admins will be able to give access to this feature by selecting the checkbox for Can view and manage datasets when creating, or editing, a permissions group. Contact Lexer if you need yours updated to enable access to this feature.

Navigating the Dataset Manager UI

Once you find your datasets in the Hub, navigating this tool and understanding each different section is easy!

All your datasets will be listed in the left-side panel. You’ll be able to see the name of each dataset along with a brief description of the dataset, the status and time of the last job load, and when the dataset load job last ran.

Click on a dataset in this panel to open the detailed dataset view in the main window. You can also click on the View button in the top, right-hand corner of the dataset screen to see more details about your dataset and the jobs that have run.

The Jobs tab is especially useful because you’ll be able to see a history of your dataset. This includes: 

  • The status of the last job load: Did it run successfully, did it fail, or is it still pending?
  • When the run job started (dates and times are displayed in your local timezone).
  • How long the job took to run.
  • Which record types were updated.

To find out more information about each individual job, you can click on the row it belongs to, which will bring up the Job View panel.

Towards the bottom of the panel you’ll see a section called Stats. The table in this section displays a list of the record types that were updated, including:

  • Total Records: The sum of New Records + Updated Records.
  • New Records: All new records that will be loaded to the CDE.
  • Updated Records: All existing records that have been updated.
  • Rejected Records: All records that have been rejected and are not a part of the Total Records count.

We can then collapse these panels and move back to the main view where you will see a list of record types for the selected dataset. These form the basis of the dataset you are viewing.

The example below shows a list of Customer records in a Klaviyo dataset. You can view other record types within the dataset by selecting a different dataset record type from the tabs along the top.

Click on a record to view detailed payload information.

Use the date picker at the top of the page to change the timelines on the data you wish to view.

Lexer tip!
When you first access the Dataset Manager, the date range will default to the “Last 7 days”.

Dataset metrics and statistics

Relevant, top-line metrics for the selected record type are presented at the top of the main window, along with a chart that presents a view of high-level metrics relevant to the record type within the dataset across the date range selected.

What’s next for the Dataset Manager?

The next step with datasets is to give you the ability to create and edit your own datasets within the Dataset Manager, and to support direct integration using Lexer’s Write API, which is coming soon, so watch this space!

Updated:
October 25, 2022
Did this page help you?
Thank you! Your feedback has been received!
Oops! Something went wrong while submitting the form, for assistance please contact support@lexer.io
Welcome to Lexer!
Fundamentals
Getting started
Our glossary
Fundamentals
Getting started
Integrations
Fundamentals
Setup
My account
Fundamentals
Setup
Manage team
Fundamentals
Setup
Group permissions
Fundamentals
Setup
Classifications
Fundamentals
Setup
Out of the box segments
Fundamentals
Setup
Browser guide
Fundamentals
Security
Corporate networks
Fundamentals
Security
Emergency contact
Fundamentals
Security
Multi-factor authentication
Fundamentals
Security
Single sign-on
Fundamentals
Security
Trust and compliance
Fundamentals
Security
Lexer's Identity Resolution
Fundamentals
Identity Resolution
Troubleshooting tech issues
Fundamentals
Troubleshooting
Error code: 503 Service Unavailable
Fundamentals
Troubleshooting
Error code: 401 Unauthorized
Fundamentals
Troubleshooting
Error code: 403 Forbidden
Fundamentals
Troubleshooting
Troubleshooting Activate
Fundamentals
Troubleshooting
Troubleshooting Respond
Fundamentals
Troubleshooting
Help! My data is missing from the Hub
Fundamentals
Troubleshooting
Understanding APIs at Lexer
Data
Data Onboarding
Providing JSON data to Lexer
Data
Data Onboarding
Providing CSV data to Lexer
Data
Data Onboarding
Upload using SFTP
Data
Data Onboarding
Upload using S3
Data
Data Onboarding
Lexer data specification
Data
Lexer Data Specification
Customer data specification
Data
Lexer Data Specification
Commerce data specification
Data
Lexer Data Specification
Marketing data specification
Data
Lexer Data Specification
Compliance data specification
Data
Lexer Data Specification
Data Formatting and Validation
Data
Getting Started with APIs
Authentication and API token creation
Data
Getting Started with APIs
Rate Limits
Data
Getting Started with APIs
Response codes and common errors
Data
Getting Started with APIs
Product imagery
Data
Getting Started with APIs
Currency conversion
Data
Getting Started with APIs
Lexer’s APIs overview
Data
Lexer’s APIs
Dataset management in the Hub
Data
Dataset management
Chatbox user API
Data
Lexer’s APIs
Activity API
Data
Lexer’s APIs
Visualize API
Hidden from nav
Profile Read API
Data
Lexer’s APIs
Lexer Javascript Tag basics
Data
Lexer Javascript Tag
Lexer Javascript Tag technical guide
Data
Lexer Javascript Tag
Lexer Javascript Tag use cases
Data
Lexer Javascript Tag
dataLayer configuration: Shopify
Data
Lexer Javascript Tag
Customer segment CSV export
Data
Data off-boarding
Export to CSV
Data
Data off-boarding
Data in Lexer's CDXP
Understand
Customer Data
Lexer's attributes
Understand
Customer Data
Attribute value types
Understand
Customer Data
Data source - CRM
Understand
Customer Data
Data source - Transactions
Understand
Customer Data
Data source - Email
Understand
Customer Data
Partner data - Experian
Understand
Customer Data
Partner data - Mastercard
Understand
Customer Data
Partner data - Roy Morgan
Understand
Customer Data
GDPR and CCPA requests
Understand
Customer Data
Upload data files
Understand
Customer Data
File upload API
Understand
Customer Data
Data provision and schemas
Understand
Customer Data
Segment overview
Understand
Segment
Creating segments
Understand
Segment
Smart Search
Understand
Segment
Export attribute results
Understand
Segment
Contact a customer
Understand
Segment
Fixing a disabled segment
Understand
Segment
Profile tab
Understand
Segment
Compare segments
Understand
Compare
Compare attributes
Understand
Compare
Activate overview
Engage
Activate
Ongoing activations
Engage
Activate
Audience splits
Engage
Activate
A/B splits
Engage
Activate
Control group splits
Engage
Activate
Inbox filtering
Engage
Respond
Ignored Senders
Engage
Respond
Forms for service
Engage
Respond
Workflow states
Engage
Respond
Bulk changes
Engage
Respond
Scheduled replies
Engage
Respond
Message templates
Engage
Respond
Customer profiles
Engage
Respond
Grouped messages
Engage
Respond
Automation rules
Engage
Respond
Redact messages
Engage
Respond
Track overview
Measure
Track
Activity overview
Measure
Activity
Team report
Measure
Activity
Cases report
Measure
Activity
Listen overview
Measure
Listen
Searching in Listen
Measure
Listen
Tier filters
Measure
Listen
Boolean search
Measure
Listen
Saved dives
Measure
Listen
Email notifications
Measure
Listen
Twitter data
Measure
Listen
Facebook data
Measure
Listen
Instagram data
Measure
Listen
Visualize overview
Measure
Visualize
Curate feed
Measure
Visualize
Report overview
Measure
Report
// Rich text for code blocks and nested lists