You can use the Data Ingestion functionality if you have new data available that needs to be ingested into Optimove but isn't included in the batch data process.
Using Data Ingestion, you can independently select, map, and upload data from external sources to to register new customers or create new attributes for richer segmentation and personalization. It gives you immediate access to new data without relying on engineering teams—so you can launch personalized campaigns faster. The process also automatically identifies new customers and registers them in your instance.
This article will provide a step-by-step guide on using Data Ingestion. Before you get started, be sure to review our best practices to ensure your files are correctly formatted.
To learn more about mastering Data Ingestion, explore our online course, designed to help all clients effectively leverage this feature.
Available Data Sources
Currently, you can ingest data via:
- Optimove's SFTP
- Amazon S3
- Google Cloud - read here to learn how to connect
- Snowflake - read here to learn how to connect
- CSV files - you can use this sample file for practice
Please Note: CSV files are One-time ingestion only. CSV uploads must be manually re-run each time new data needs to be ingested. If re-running, the new CSV file must match the original format.
Key points to consider before you get started
- You can select one file for each ingestion process.
- The Data Ingestion tool supports a maximum directory depth of 2 nested folders (e.g.,
Folder/Subfolder). Files located deeper than two levels will not be visible in the user interface and cannot be ingested. Please ensure your files are stored within this limit. - You can create up to 36 attributes per file/table.
- Each row in a file should represent an existing customer by their Customer ID.
- Each column should represent an attribute. Once the data is mapped, the new customer attributes will be appended to your Single Customer View in Optimove.
- Each row in your file should represent a customer, identified by their unique Customer ID, with no duplications per customer. In case of failure, check for duplicates in the Customer IDs and remove them. Each customer should appear only once in the files.
- Check that there aren’t any NULL values in the Customer ID field and that all Customer IDs in the file match the IDs you have in the instance.
- Make sure there aren’t any default date values in any date fields you’re trying to add. For example, 1900-01-01, 1989-01-01 and any date prior to 1753-01-01.
- Remove any characters that aren’t supported, like emojis.
- Only use the valid delimeter: comma
- String values might be too long. The maximum string length that is supported is varchar(8000) or nvarchar(4000).
- Check that the values being ingested fit the correct attribute format types (date, string, int, etc.).
- Notice that in the data ingestion process, you’re able to add new attributes for existing customers but not add new customers to your instance. Therefore, new Customer IDs in the file won’t be added to the site.
- Overwrite behaviour (important): Values in the file replace existing values; NULL/empty clears. Learn more.
How To Use
Step-by-step Guide
- Go to the 'Data Studio' in the navigation bar and select 'Data Ingestion'.
- In the top right-hand corner, select 'Ingest Data'.
-
Choose what ingestion process to run.
Select Register new & update existing customers if you want to ingest new customers into Optimove, which is especially useful for new tenants who want to start adding customers before the Optimove Data team completes the full setup.Select Update existing customers only if you only want to update attributes for customers that already exist in Optimove.
- Select your Data Source.
- Enter Credentials to help you securely access your data. Then, click on Connect.
- Review and Select Data. In the left sidebar, you can see the data available within the source and select the specific file and columns you wish to import to Optimove.
-
Begin Data Mapping. This allows you to map each column in your file to create new attributes.
One column must be the Customer ID.
You can then name each attribute and select the formatting of the attribute. The formatting will determine the operators shown when using this attribute in Customer Explorer and how the values are shown in Customer 360. You can add up to 10 attributes per ingestion.
- If you choose to register a new customer and add an email field, you will be asked if you want to map the email address so we can add it as an email for future use in Optimail. For any other field, select a new attribute name (some attribute names will be already in use for future ingestions through batch).
- Schedule the Ingest. S3, SFTP, and Google Cloud all support automated ingestion. You can choose when the process should start and how frequently it should occur. You can update the data once every 24, 12, or 6 hours. Alternatively, you can opt for a one-time import.
Please Note:
- Only one-time ingestion is supported when selecting CSV as your data source. To update attributes ingested via CSV, re-upload a new file in the exact same format before re-running the process.
- To ensure your customer segments are updated with this data, you can use the re-evaluate functionality for campaigns using these self-serve attributes. This will remove customers who no longer fit the criteria of the Target Group.
- Review the data in the summary page. If everything is correct, hit Complete. Note that the attributes cannot be edited once hitting Complete.
- Go to Customer Attributes in Customer Explorer to find your new attributes.
Please note: Newly registered customers uploaded via Data Ingestion can be targeted on the same day of their registration if the client is using Campaign Execution 2.0.
Without Campaign Execution 2.0, they’ll be available for segmentation only when using the "Run anyway" campaign option. If you want to target these newly registered clients specifically, you can do it only after the next daily batch, as Days Since Registration = 1 rule may return 0 on registration day.
Managing the Ingestion Process
The Data Ingestion homepage provides full visibility into your data ingestion processes for monitoring purposes.
In the homepage, you can:
- Monitor the statuses of each ingestion process
-
Identify failed processes. If an ingest of a recurring process fails, the campaign will go out with data from the last successful ingestion. You will receive an email notification if the ingest fails.
In the settings area, any user can sign up to the events highlighted on the image below to get a notification about data ingestion failures. - Re-run and/or update one-time ingestion processes