1. How to format your data
MadKudu works with 3 types of objects
- Event: what are users doing?
- Contact: who is the user?
- Account: what accounts to my users belong to?
- Opportunity: what deals do I open?
Depending on what type of segmentation you will want to build (based on behavioral activity, based on lead/account attributes or based on both), you may send us data of 1 or more objects.
- If you track behavioral activity in a system that does not integrate with MadKudu and would like to build a behavioral segmentation, you would need to send at least Events and Contacts. If you plan on having account scoring, please send us your Accounts as well.
- If you have a homemade CRM or CRM which does not integrate with MadKudu, you would need to send at least Contacts and Opportunities for MadKudu to understand who are your contacts and who converts. If you plan on having account scoring, please send us your Accounts as well.
1.1. Event
To send behavioral data (product usage, web activity, marketing activity...), create a file named event with the following attributes:
| Attribute | Format | Example | Description | |
event_key |
required | String | "abc123" | A unique key identifying the event. If you do not have one, we suggest creating a combination of event_text + contact_key + event_timestamp |
event_text |
required | String | “signup”, “login”, “invited a friend” | The action taken by the user. |
event_timestamp |
required | Unix time | “1436172703” | The time at which the event happened |
contact_key |
required | String | "abc123" or "paul@madkudu.com" | The unique identifier of the user who performed the action. This needs to be the same as the contact_key field in the identify file. |
event_* |
optional | String or Numeric | properties describing the event (e.g. event_url for the url of visited page, event_form_title for the title of form submitted...) |
Example in JSON format
{"event_key": "abcd1234", "event_text":"signed up", "event_timestamp":1234567890, "contact_key":"abc1234"}
{"event_key": "abcd2345", "event_text":"visit web page", "event_timestamp":1234567890, "contact_key":"paul@madkudu.com", "event_url":"http://www.domain.com/pricing"}
1.2. Contact
To send enrichment or CRM data on the contact level (CRM, demographic, firmographic traits ...), create a file named contact with the following attributes:
| Attribute | Format | Example | Description | |
contact_key |
required | String | “abc123”, “paul@madkudu.com” | Unique identifier for the user in your database. It can be the email. It must be the same as used in the Event file if Event file sent as well. |
|
|
required | String | "paul@madkudu.com" | Email of the user. Pass it even if the contact_key already contains the email |
created_date |
required | Unix time | 1436172703 | Creation date of the contact (required if MadKudu does not have an integration with your CRM) |
contact_* |
optional | String or Numeric | Enrichment traits you know about the user (examples: contact_title, contact_country, contact_subscription_plan...) |
Example in JSON format
{"contact_key":"abc1234", "email":"paul@madkudu.com"}
{"contact_key":"432535", "email":"paul@madkudu.com", "contact_title":"cto"}
1.3. Account
To send enrichment or CRM data on the account level (CRM, demographic, firmographic traits ...), create a file named account with the following attributes:
| Attribute | Format | Example | Description | |
account_key |
required | String | "def456", “madkudu.com” | a unique identifier for the account the user belonged to. It can be the domain of the account |
contact_key |
required if sending Contact data | String | "abc123", "paul@madkudu.com" | the unique identifier of the user who performed the action. This needs to be the same as the contact_key field in the Contact and Event files. |
domain |
required | String | "madkudu.com" | Web domain of the account |
created_date |
required | Unix time | 1436172703 | Creation date of the account (required if MadKudu does not have an integration with your CRM) |
conversion_date |
optional | Unix time | 1436172703 | Conversion of the account into paying customer. |
ARR |
optional (highly recommended) | Numeric | $20,000 | Annual Recurring Revenue of the account |
account_* |
optional | String or Numeric | Enrichment attributes you know about the account (examples: account_industry, account_ARR, account_subscription_plan...) |
Important: Our system supports one account per contact. If there are severals, we’ll use the latest one. If you have a use case for having a user belonging to several accounts, we’d love to hear about it. Please let us know at support@madkudu.com.
Example in JSON
{"contact_key":"abc1234", "account_key":"madkudu.com", "domain": "madkudu.com", "name": "madkudu"}
{"contact_key":"abc4983", "account_key":"ibm.com", "domain": "ibm.com", "account_ARR":"3000"}
1.4. Opportunities
If you send us who are your conversions (it can be the list of your customers), you'll be able to train segmentations based on different types of conversions and have access to Insights reports and Performance report of you segmentations in the MadKudu app. If you don't track opportunities, send us the accounts flagging the ones converted as described above in 1.3 Account.
Otherwise, create a file named opportunity with the following attributes:
| Attribute | Format | Example | Description | |
opp_key |
required | String | "iuh374", “madkudu.com - new biz” | a unique identifier for the account the user belonged to. It can be the name of the opportunity |
contact_key |
required if sending Contact data | String | "abc123", "paul@madkudu.com" | the unique identifier of the user who performed the action. This needs to be the same as the contact_key field in the Contact and Event files. |
account_key |
required if sending account data | String | "def456", “madkudu.com” | a unique identifier for the account the user belonged to. It can be the domain of the account |
created_date or close_date |
required | Unix time | 1436172703 | Creation date of the opportunity or closed date when the opportunity was won (prospect becomes a customer) |
opp_amount |
required | Numeric | $2,100 | the amount of the opportunity |
opp_probability |
optional | Numeric | 10% | the probability of closing the opportunity. This can be replaced by a boolean "has_converted". |
opp_type |
optional | String | "New business" | type of the opp |
opp_* |
optional | String or Numeric | Enrichment attributes you know about the opp (examples: opp_subtype, opp_ARR, opp_subscription_plan...) |
1.5 Specific format for Customer Fit training
If you are not able to send your Opportunities as described above, MadKudu still needs to understand who converts from your historical data to configure a customer fit model. You can send us a unique flat file extracted from your CRM that tells us among your leads who has converted with the following fields:
| Attribute | Format | Example | Description | |
|
|
required | String | “paul@madkudu.com” | the unique identifier of the user who performed the action |
target |
required | Boolean | 1 | indicate if the lead converted with your conversion definition (Opp created, Opp stage 2…) |
amount |
required | Numeric | 2,300 | if target =1, amount of the opportunity converted (as defined by the conversion definition). 0 otherwise. |
target_closed_won |
required | Boolean | 1 | indicate if the lead converted into a Closed Won opp (paying customer) |
amount_closed_won |
required | Numeric | 2300 | amount generated from the first closed won opp. |
created_date |
optional | Unix time | created date of the email | |
properties |
optional | String or Numeric | Numeric self-input information at time of lead creation or any other field that you’ve augmented your leads with and want MadKudu to evaluate (example: team_size, industry...) |
Example in JSON
{"email":"elon@tesla.com", "target":"1", "amount": "2499", "target_closed_won":"0", "amount_closed_won":"0", "created_date": "1234567890" }
{"email":"paul@madkudu.com", "target":"0", "amount": "0", "target_closed_won":"0", "amount_closed_won":"0", "created_date": "1234567810", "team_size":"5"}
2. How to format the files
MadKudu currently support two file formats:
- Newline-delimited JSON (preferred)
- CSV
2.1 Newline-delimited JSON
Our preferred format for upload is newline-delimited JSON, which is more standardized and less error-prone than CSV.
In this format, the different records are separate by the newline (\n) character. Each line is a valid JSON object:
{"event_text":"signed up", "event_timestamp":1234567890, "contact_key":"abc1234"}
{"event_text":"added a friend", "event_timestamp":1234567890, "contact_key":"paul@madkudu.com", "some_other_event_field":"some_value"}
{"contact_key":"abc1234", "email":"paul@madkudu.com"}
{"contact_key":"432535", "email":"paul@madkudu.com", "some_other_contact_field":"some value"}
- escape any double quote
"in your data with a\(e.g. replace"with\")
Incorrect
{"event_text":"signed up", "event_timestamp":1234567890, "contact_key":"abc1234", "key": "val"ue"}
Correct
{"event_text":"signed up", "event_timestamp":1234567890, "contact_key":"abc1234", "key": "val\"ue"}
2.2 CSV
We also support the .csv format, with the recommended format:
- column names in the first line
- separator: ~ -> separate the value with ~ (abc~def~..) Please do not use
,or-as it easily creates parsing issues - delimiter: " -> this adds quotes around the values (abc -> "abc")
- line separator: line-break (\n)
- Escape your
"characters by adding a second"character in front of it (see here for details)
Incorrect
abc,cde,efg
Correct
"abc","cde","efg"
- Remove all line break characters (for example
\n) from your fields - Make sure the number of fields is the same for each lineIncorrect
"abc","cd"e","efg"
Correct
"abc","cd""e","efg"
Using the UTF-8 encoding would be useful to avoid any issues with special characters in the files.
2.3 Data validation
JSON line and CSV are relatively easy to corrupt (for example with " or , characters in the data).
We will validate the data on your side and warn you on your corruption issue, but it helps a lot if you follow the format requested above.
2.4 Compression
To speed up the data upload part, we highly recommend that you compress your file with GZIP before uploading them to S3.
You can call your file however you want it (we recommend event, contact and account), just make sure to add the correct extension depending on your file format:
- .json.gz for compressed JSON (recommended)
- .json for uncompressed JSON
- .csv.gz for compressed CSV
- .csv for uncompressed CSV
3. How to transfer your file
You will be provided the credentials of our Amazon S3 bucket in the Madkudu app > Integrations > Amazon S3.
Upload the files to our Amazon S3 bucket in the format specified below and we will automatically import the new data.
You will also need to set up a recurring push of your data to S3 for MadKudu to score fresh data.
We highly recommend to use our Amazon S3 bucket, but we can pull the data from your own S3 bucket. Please refer to these instructions to share access.
3.1 File naming
In the S3 bucket, please upload data into separate folders by date and by objects
{object}/{timestamp}
where the objects are
- event
- contact
- account
- opportunity
If you use the S3 API, simply “prefix” your destination file name. For example, uploading to "contact/20201120-11:00:00/name_of_file.csv" will add a file name name_of_file.csv to the contact folder.
You can name the file whichever way you’d like but we recommend to use the object and the date like timestamp_object_name.extension. (e.g. 20201128_account_part_01.json.gz) We’ll automatically import any new file in the folder. You just need to make sure that the right type of files go in the right folder.
3.2. File upload
Files can be uploaded to Amazon S3 via:
- a Command Line Interface (CLI). Ideal for testing and debugging.
- SDKs in many common language (Java, JavaScript, Python…). Perfect for automating your data transfer.
3.3 Upload files via the CLI
Get started by following the instructions to install the AWS CLI.
Once installed, configure the AWS CLI with the credentials of the S3 bucket we sent you.
To upload a file with the aws s3api CLI, commands will look like this
aws s3api put-object --bucket madkudu-data-in-XXXX --key contact/destination_file_name.csv --body ./Documents/file_to_upload.csv
3.4 Compression
To speed up file transfer, you can compress files locally before transferring them to Amazon S3. If you want to compress your files, please use the GZIP compression method and use .gz or .gzip as your file extension (we currently don’t support other methods or other extensions)
3.5 Frequency - Setting up a recurring push of data to MadKudu
Depending on how your data will be used in the MadKudu platform, we recommend to send us data
- every 4 hours or at least twice a day if used in a segmentation
- in one shot if used for a punctual model training or analysis
For setting up a recurring push of data,
- if your files contains only the new records at each upload, it's very important to name each file differently (see 2.1 File naming) not to lose the historical records.
- if your file contains the accumulated the records, then it should replace the previous file otherwise we would be pulling duplicated records.
Please email support@madkudu.com if you have a doubt, we'll be happy to assist you.
Frequently Asked Questions
I'm having an issue with S3 / I don't know how to use S3
Please email support@madkudu.com or contact your Customer Success Manager and we will be happy to assist you.
Your file format doesn’t work for me. What do I do?
If you’re having any issue with the file format, please reach out to us at support@madkudu.com and we’ll be happy to help.
How often is the data refreshed?
As soon as you drop data to the S3 bucket, expect results to be updated in the Madkudu platform within 6 hours.
What would happen if I send the same event more than once - will it appear twice in MadKudu?
Our system will dedupe the events based on contact_key / event_text / timestamp. If you send the same event twice, only one will be kept:
- If sent in two separate batches: only the most recent will be kept
- If sent in the same data batch, first one in the file.
Can we add other attributes to the Contact records?
Yes, please send any attributes you have stored in your user table (except sensitive ones (password, cc number)).
In particular, it is always helpful to get the following:
created_datelead source- current plan / value of the plan
Comments
0 comments
Please sign in to leave a comment.