You can use the Data Ingestion functionality if you have new data available that needs to be ingested into Optimove but isn't included in the batch data process.

Using Data Ingestion, you can independently select, map, and upload data from external sources to to register new customers or create new attributes for richer segmentation and personalization. It gives you immediate access to new data without relying on engineering teams—so you can launch personalized campaigns faster. The process also automatically identifies new customers and registers them in your instance.

This article will provide a step-by-step guide on using Data Ingestion. Before you get started, be sure to review our best practices to ensure your files are correctly formatted.

To learn more about mastering Data Ingestion, explore our online course, designed to help all clients effectively leverage this feature.

Available Data Sources

Currently, you can ingest data via:

SFTP
Amazon S3
Google Cloud - read here to learn how to connect
Snowflake - read here to learn how to connect
CSV files - you can use this sample file for practice

Please Note: CSV files are One-time ingestion only. CSV uploads must be manually re-run each time new data needs to be ingested. If re-running, the new CSV file must match the original format.

Key points to consider before you get started

You can select one file for each ingestion process.
The Data Ingestion tool supports a maximum directory depth of 2 nested folders (e.g., Folder/Subfolder). Files located deeper than two levels will not be visible in the user interface and cannot be ingested. Please ensure your files are stored within this limit.
You can create up to 10 attributes per file/table.
Each row in a file should represent an existing customer by their Customer ID.
Each column should represent an attribute. Once the data is mapped, the new customer attributes will be appended to your Single Customer View in Optimove.
Each row in your file should represent a customer, identified by their unique Customer ID, with no duplications per customer. In case of failure, check for duplicates in the Customer IDs and remove them. Each customer should appear only once in the files.
Check that there aren’t any NULL values in the Customer ID field and that all Customer IDs in the file match the IDs you have in the instance.
Make sure there aren’t any default date values in any date fields you’re trying to add. For example, 1900-01-01, 1989-01-01 and any date prior to 1753-01-01.
Remove any characters that aren’t supported, like emojis.
Only use the valid delimeter: comma
String values might be too long. The maximum string length that is supported is varchar(8000) or nvarchar(4000).
Check that the values being ingested fit the correct attribute format types (date, string, int, etc.).
Notice that in the data ingestion process, you’re able to add new attributes for existing customers but not add new customers to your instance. Therefore, new Customer IDs in the file won’t be added to the site.
Overwrite behaviour (important): Values in the file replace existing values; NULL/empty clears. Learn more.

How To Use

Step-by-step Guide

Go to the 'Data Studio' in the navigation bar and select 'Data Ingestion'.
In the top right-hand corner, select 'Ingest Data'.
Choose what ingestion process to run.
Select Register new & update existing customers if you want to ingest new customers into Optimove, which is especially useful for new tenants who want to start adding customers before the Optimove Data team completes the full setup.

Select Update existing customers only if you only want to update attributes for customers that already exist in Optimove.
Select your Data Source.
Enter Credentials to help you securely access your data. Then, click on Connect.
Review and Select Data. In the left sidebar, you can see the data available within the source and select the specific file and columns you wish to import to Optimove.
Begin Data Mapping. This allows you to map each column in your file to create new attributes.

One column must be the Customer ID.

You can then name each attribute and select the formatting of the attribute. The formatting will determine the operators shown when using this attribute in Customer Explorer and how the values are shown in Customer 360. You can add up to 10 attributes per ingestion.

- If you choose to register a new customer and add an email field, you will be asked if you want to map the email address so we can add it as an email for future use in Optimail. For any other field, select a new attribute name (some attribute names will be already in use for future ingestions through batch).
Schedule the Ingest. S3, SFTP, and Google Cloud all support automated ingestion. You can choose when the process should start and how frequently it should occur. You can update the data once every 24, 12, or 6 hours. Alternatively, you can opt for a one-time import.

Please Note:
- Only one-time ingestion is supported when selecting CSV as your data source. To update attributes ingested via CSV, re-upload a new file in the exact same format before re-running the process.
- To ensure your customer segments are updated with this data, you can use the re-evaluate functionality for campaigns using these self-serve attributes. This will remove customers who no longer fit the criteria of the Target Group.
Review the data in the summary page. If everything is correct, hit Complete. Note that the attributes cannot be edited once hitting Complete.
Go to Customer Attributes in Customer Explorer to find your new attributes.

Please note: Newly registered customers uploaded via Data Ingestion can be targeted on the same day of their registration only if the client is using Campaign Execution 2.0.

Without Campaign Execution 2.0, they’ll be available for segmentation only after the next daily batch, so “newly registered” segments (e.g., Days Since Registration = 1) may return 0 on registration day.

Managing the Ingestion Process

The Data Ingestion homepage provides full visibility into your data ingestion processes for monitoring purposes.

In the homepage, you can:

Monitor the statuses of each ingestion process
Identify failed processes. If an ingest of a recurring process fails, the campaign will go out with data from the last successful ingestion. You will receive an email notification if the ingest fails.

In the settings area, any user can sign up to the events highlighted on the image below to get a notification about data ingestion failures.
Re-run and/or update one-time ingestion processes

Edit draft processes

Updating Attributes in the Ingestion Process

Optimove offers different methods for updating the attributes created during data ingestion, depending on your use case. Here's a breakdown:

1. Updating Existing Attribute Values

One-Time Ingests:

Use the Re-Run feature to update values.
Ensure the file contains all the same columns as the original ingest.
You can include additional Customer IDs (CIDs) if needed.]

Recurring Ingests:

Upload a new file to your chosen data source in the following format:
file names must be in the format of: FileName_% For example: Customers_1, Customers_2, etc. The FileName will be the name of the initial file you chose when building the ingestion process and can contain numbers/ characters. After the underscore, we recommend using UnixTime for a unique serial number.
Optimove will automatically ingest the last modified file with the same file name in the next scheduled process.

2. Adding New Attributes

For adding new attributes to en existing file, you will always need to set up a new ingestion process.

You will be able to add the same file and map the additional columns to create new attributes

For ingestion processes in Draft status, you can simply select the 'Edit' option to make any changes

Statuses

Processing

A new process has been scheduled and is awaiting its initial run. This is a one-time status; once the process executes for the first time (at its scheduled run time), it will not return to this status.

Completed

The ingestion was completed successfully. Data was fully ingested to Optimove, and all relevant attributes were updated.
For the ‘One Time’ scheduled import, there will be a drop-down under the Completed status; this enables clients to re-run the process another time.

Running

The data is in the process of being transferred.

Partially Failed

Not all records from the file have been ingested. This means the data was transferred for some customers in the file but not all.

For Scheduled Processes: If the process is scheduled (e.g., recurring every 6, 12, or 24 hours), a partially failed process will still be re-scheduled to run at the next planned time. You cannot manually re-run a partially failed scheduled process, as this could delay the next scheduled run. Instead, the process will automatically re-run at the next scheduled time, giving you the opportunity to adjust the data (e.g., fix formatting issues) before then.

Failed

Due to an error, the data was not ingested into Optimove.

For Scheduled Processes: If the process is scheduled (e.g., recurring every 6, 12, or 24 hours), a failed process will not run again at the next scheduled time until the issue is resolved. You must fix the error (e.g., check for duplicate Customer IDs or invalid formats) and manually re-run the process using the drop-down menu in the Failed status. The process will resume its regular schedule only after a successful or partially failed run.

Overwrite & NULL behavior

Data Ingestion uses “last write wins.” For any customer rows included in your file, each mapped attribute is set to the value in the file.

NULL/empty clears: sending an empty/NULL value will set the attribute to empty.
Send the full record: even if one attribute changed, include all mapped attributes with their intended current values.
Only rows you send are affected: customers not in the file remain unchanged.
Separate by subject: keep topics in separate files/tables (e.g., promotions vs. loyalty tier) to avoid accidental clears.

Example
Before (CID 123): loyalty_tier=Silver, promo_code=SPRING10
File row: loyalty_tier=Gold, promo_code= (empty)
Result: loyalty_tier=Gold, promo_code=NULL. To keep promo_code, send it with its current value.

FAQs

What is the maximum number of attributes that can be added using this feature?

The maximum number of attributes that can be added with Data Ingestion is 200.

If we add data using the feature, will it affect the batch data process performance?

No. Data Ingestion shouldn't conflict with the Batch Data process; these are 2 completely different streams.

How much time will take to update all attribute values?

This can change based on the file size, number of records, and number of columns being ingested. The minimum time is ~20 minutes, and the maximum is ~1 hour. We recommend scheduling campaigns based on data being ingested in this feature at least an hour after scheduling the data update.

If the data ingestion process fails, will any campaigns based on this data also fail?

The campaigns will be sent either way (if the process ran successfully or failed). They will just be based on previous data. Notice the ingestion process can also partially fail- meaning not all records were ingested successfully- this will also not affect campaign execution.

For scheduled data ingestion process during the day, how can we make sure the Target Groups and campaigns will be updated based on new values?

After the data ingestion process is completed successfully and attributes are updated with new values, you will need to Reevaluate the Target Group to remove customers who no longer fit in it.

For what reasons will the data ingestion process fail?

Duplications on CID
Check that CID doesn’t have NULL values
Check that CID exists in the file
Make sure there are no default date values (such as 1900-01-01, 1989-01-01)
Remove characters that aren’t supported, such as emojis.
Cut long string values (to the max length we support)
Check field format types (int, string, date, etc.)
Optimove checks - timeout, connection issues, failure on Optimove process, etc.

Learn more here.

What do we do if the process has failed?

You can investigate any issues that may be associated with the file as listed here. If you need help, you can contact our support team.

What types of attributes can we add using this feature? Can we run calculations on the added data?

Right now, clients can only add 'flat' attributes, which are on the customer level and don't require any further calculations or manipulations. Later, data ingestion attributes can be used in the data studio to create further calculated attributes.

Can we use the feature to add new customers to the instance?

Yes. This feature enables you to add new customers, add new attributes for existing customers and update values for these attributes.

Can we update existing attributes using this feature, attributes that come into the system from the Batch Data process?

No. You can only update values for attributes that have been added using the data ingestion feature.

How does Data Ingestion handle overwrites and empty values?

Records you include in a file use “last write wins”: each mapped attribute is updated to the file’s value. Sending NULL/empty will clear that attribute. Customers not in the file aren’t changed. To avoid accidental clears, send full records (with all intended current values) or split topics into separate files/tables. See Overwrite & NULL behaviour for an example.

Can we re-run one-time processes? Can we update attribute values that have been added via a one-time ingestion process?

Yes. When clicking the drop down button on the status of the process, you'll have a re-run button. For CSV processes, you will be required to manually upload a CSV with new attribute values before re-running the process.

How much time will it take to update all attribute values?

For clients that don't have "Campaign Excecution 2.0." it will take up to 3 hours from data ingestion process completion for the attribute values to be accurate for these features:

Run Now
Re-evaluate
Run Anyway

The data will be available for targeting, personalization, and use in the instance. Just the features mentioned above that are more time-sensitive will need to take into account the up to 3-hour wait.

Why don’t new or updated attributes appear immediately in Customer360 Snapshots?

After a Data Ingestion process completes, new or updated attributes may take a few minutes to appear in Customer360 Snapshots. This is expected behavior while Optimove processes the data and refreshes the snapshot view.

If the attributes are still not visible after several minutes, try clearing your browser cache or logging out and back into Optimove to ensure you are viewing the most up to date Customer360 snapshot.