In the following tutorial we will explain how to setup an S3 bucket and give us access to it securely.
S3 buckets can be used to exchange data with MadKudu
- Send to MadKudu the data from your Data Warehouse (Snowflake, Redshift, BigQuery...) or from other integrations
- Receive MadKudu data (scores, predictions, segmentations...) in S3 to then export it to where you need it (data warehouse or other integrations)
If you already export Kissmetrics, Segment, or your own data to a S3 bucket, we can access data directly from that bucket.
If the bucket you want to share already exists, please skip the following section (Create an S3 Bucket).
Create an S3 Bucket
- Go to AWS Management Console
- Go to S3 service from Services > Storage > S3
- Create Bucket Amazon S3 > Create bucket
- Fill in the form and make note of your bucket
- You can use any bucket name, example: my-madkudu-shared-bucket
Setting up the correct access to your S3 bucket
For MadKudu to access your S3 bucket, our preferred option is for you to grant access to your S3 bucket to a MadKudu IAM role.
To do this, you'll need MadKudu's AWS account ID and External ID to Create an IAM policy and Create an IAM role (see AWS documentation).
Step 1. Get MadKudu AWS account ID and external ID
Visit app.madkudu.com > Integrations > Amazon S3 > Configuration to find MadKudu's account id and external id or use:
- Account ID: 203796963081
- External ID: your MadKudu API key (see below where to find it in app.madkudu.com)
Step 2. Create an IAM Policy for MadKudu
For MadKudu to pull data from your bucket, you'll need to grant read permissions (ListBucket, GetObject)
For MadKudu to push data back into your bucket, you'll need to grant write (and delete) permissions on top of that (PutObject, DeleteObject)
- Go to your AWS Management Console.
- Go to IAM Identity and Access Management from: Services > Security, Identity & Compliance > IAM.
- Go to Policies and click on Create Policy.
- If you use the Visual editor (AWS documentation) to select the permissions,
-
- Chose the Service: S3
- You can give all permissions or select only the following Actions
List: ListBucket
Read: HeadObject
Read:GetObject -- this is needed for a pull from S3
Write:PutObject -- this is needed for a push to S3
Write: DeleteObject -- this is needed for a push to S3 (to delete test files)
If you use the JSON tab instead
Copy the following policy and paste it in the JSON tab, replacing
bucket_namewith the name of your S3 bucket.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::203796963081:root"
},
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::bucket_name"
]
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::203796963081:root"
},
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::bucket_name/*"
]
}
]
}
-
- Click Review Policy
-
Name the policy "MadKudu-S3-Access".
- Click Create Policy.
Step 3. Create an IAM Role for MadKudu
- Got to Roles and Click on Create role.
- Choose the Another AWS account role type.
- For Account ID, type the MadKudu AWS account ID to which you want to grant access to your resources.
- As you are granting permissions to users from an account that you do not control, and the users will assume this role programmatically, select Require external ID. (see more info)
- Enter the External ID you got at Step 1.
- Confirm that Require MFA is not selected.
- Click Next: Permissions.
- Select the "MadKudu-S3-Access" policy that you created in Step 2.
- Click Next:Tags.
Add any tag and description to this role to identify it.
- Click Next:Review.
Use any role name you want but we suggest to include "madkudu" in it, for example "integration-madkudu-s3-read" .
- Click Create role.
-
Select the MadKudu role you just created.
- Find the Role ARN and make note of it.
- Change the maximum session duration to 12 hours (This is to ensure that MadKudu will be able to extract data from your bucket without the credentials expiring in the middle).
- Go the Amazon S3 page in the MadKudu app: app.madkudu.com > Integrations > Amazon S3 > Configuration and input & save:
- the Role ARN
- region of your Bucket
- bucket directory (with a folder path(s) if MadKudu can only access specific folders): this specifies the portion of the bucket in which you'd like MadKudu to pull data from. Any files under the specified folder and all of its nested subfolders will be examined for files we can upload. If no prefix is supplied, we'll look through the entire bucket for files to sync.
Encryption
We recommend that you encrypt your data in the S3 bucket for increased protection. If your data in S3 is encrypted server-side, you would need to add a policy to let us use the encryption key.
Step 1. Update the KMS key policy
1. Open the AWS KMS console, and then view the key's policy document using the policy view. Modify the key's policy to grant the IAM user permissions for the kms:GenerateDataKey and kms:Decrypt actions at minimum. You can add a statement that's similar to the following:
{
"Sid": "ExampleStmt",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:user/Jane"
},
"Resource": "*"
}Note: This example policy includes only the minimum permissions required for an individual IAM user to download and upload to an encrypted S3 bucket. You can modify or expand the permissions based on your use case.
Step 2. Update the IAM policy
2. Open the IAM console, update the IAM policy (created in Step 2 above) that grants the permissions to read from the bucket to work with the AWS KMS key that's associated with the bucket.
For the Resource value, enter the AWS KMS key's ARN.
{
"Sid": "KMSAccess",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Effect": "Allow",
"Resource": "arn:aws:kms:example-region-1:123456789098:key/111aa2bb-333c-4d44-5555-a111bb2c33dd"
}
Please open a ticket here if you are facing difficulties, or consult the F.A.Q.
NOTE: If all of this sounds like gibberish, please forward it directly to your favorite developer :)