Hightouch is a low-code reverse ETL tool for marketers allowing to easily move data sitting in your data warehouse into other tools where you need it. 

If you'd like to send data sitting in Snowflake or BigQuery, we recommend using our direct Snowflake and BigQuery integration.

You can use Hightouch to send data from your data warehouse to MadKudu into an S3 bucket 

Pre-requisites

• You have a Hightouch account connected to your database (Redshift, Postgres...) 
• You have an AWS account to create a S3 bucket 
• You have added this S3 bucket as a destination in Hightouch 

Step 1: Create a Model in Hightouch to format your data

The Models feature in Hightouch allows you to transform the data from your database into the format expected by MadKudu with a simple SQL query. 

1. Log into your Hightouch account and go to Models and click Add model 
2. Select your database (Redshift, Postgres...) for the source
3. Select a modeling method. We recommend SQL if your database data needs transformation to match MadKudu schema 
   
4. Write your SQL query to transform into MadKudu format please refer to this article detailing the format depending on the data you are sending 
   Not a SQL expert? We've got you! Our Support team is and can do this live with you, no biggy!

   --- App events ---
   SELECT 
     CAST(id as VARCHAR(256) as event_id,
     CAST(event_text as VARCHAR(256)) as event_text, 
     CAST(timestamp as TIMESTAMP) as event_timestamp,
     NULL as event_url,
     CAST(email AS VARCHAR(256)) as email
   FROM analysis.app_events
     UNION 
   --- website page views and events
   SELECT 
     CAST(id as VARCHAR(256) as event_id,
     CAST(event_text as VARCHAR(256)) as event_text, 
     CAST(timestamp as TIMESTAMP) as event_timestamp,
     CAST(url AS VARCHAR(256)) as event_url,
     CAST(email AS VARCHAR(256)) as email
   FROM analysis.website_events

    

   
   
5. Test your code by clicking Preview results and click Continue
6. The naming is up to you but we can recommend: 
   1. App and Website Events - MadKudu format : if you are sending product events, web visits.. 
   2. Contacts - MadKudu format : if you are sending users, leads, contacts properties 
   3. Accounts - MadKudu format: if you are sending account properties 
   4. Account Signals - MadKudu format:  if you are sending account usage properties (like intent)
      
      Tip: For events, we recommend creating 2 models: 
      • one for the 9 months of historical data you'll sync in one shot, once 
      • the other one that would send on a daily basis your most recent data to the S3  

Step 2: Create a Sync in Hightouch to send this model to your S3 

In this model in Hightouch,

1. click on the Sync tab and Add a sync
2. Select as destination Amazon S3 bucket you have created and connected (see pre-requisites) If you have not created and connected a S3 bucket for MadKudu to pull from, you'll need to do that first in Destinations  
   
3. In the Configure Sync step,
   1. Select Insert behavior to add files to the bucket
   2. Format JSON

   3. Object: YYYY/MM/YY/{object}.jsonexample: YYYY/MM/YY/events.json

   4. Timestamp offset: 0

   5. Check Sync ALL columns

   6. Batching: yes

   7. Rows in each file: 10000
   8. Sequential file naming: by batch number
   9. Handle empty file: don't skip empty file
      
4. In Finalize sync,
    Tip! We recommend sending a test file manually (option 1) before setting up the recurring sync. And move to Step 3. Once we confirm everything looks good, you can set up the recurring! 
   1. Option 1: you are sending only 9 months of historical data
      • Select Manual 
   2. Option 2: you are setting up the recurring flow to MadKudu 
      • Select Interval of 4 hours, immediately, indefinitely 
5. Run the sync

Step 3: Give MadKudu access to your S3 bucket

MadKudu connects to your S3 bucket via an IAM role. Please follow the instruction starting step 2 in this documentation: Giving MadKudu access to your S3 bucket

Once you're all set, let us know via a support ticket and we'll start pulling the data from S3