Uploading Segments and Bid Models to S3

If a segment file is over 10 Mb in size, the file needs to be uploaded to a Beeswax S3 bucket. To upload your segment files or bid model files to S3, you need to complete the following steps:

1. Provide your Beeswax account representative with your AWS Identity & Access Management (IAM) Amazon Resource Name (ARN). ( Link to AWS documentation). Ensure the ARN isn't for a "group", or for "root", only "user" and "role" ARNs will work.

2. Beeswax will notify you once your permissions have been added to the Beeswax bucket policy.

NOTE: Please do not try uploading the files before receiving notification from your Beeswax account representative as Beeswax will have to update permissions as well. Once notified, you can start writing to your designated S3 folder.

3. Beeswax will provide an ACL policy to apply to your user. It will look like:

{
    "Sid": "AllowAssumeBeeswaxRole",
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::164891057361:role/customer-s3-<user-list or bid-models>-<customer_buzz_key>"
}
	

Note: The placeholder <user-list or bid-models> must be replaced depending on what file type you are uploading. The placeholder <customer_buzz_key> must be replaced with your actual buzz key.

4. You will assume the role from your ACL policy in order to access the Beeswax S3 bucket. ( Link to AWS documentation). We recommend that you create a new profile in ~/.aws/config that uses your AWS user provided in step 1 to assume the specified role. You will use this profile when executing steps 5 and 6.

5. Upload the segment file to the Beeswax S3 path

Example s3 path:  s3://beeswax-data-us-east-1/<user-list or bid_models>/<customer_buzz_key>/

Note: Please be sure your file is formatted per the instructions in the Segment Uploads documentation 

Note: The placeholder <user-list or bid_models> must be replaced depending on what file type you are uploading. The placeholder <customer_buzz_key> must be replaced with your actual buzz key. If you are uploading files to EMEA or APAC please use the corresponding bucket <beeswax-data-eu-west-1> or <beeswax-data-ap-northeast-1>. 

Note: You should be using the --profile option with the name of the profile you created in step 4.

Note: Ensure you have not forgotten the forward-slash at the end when specifying the bucket in your command/code. Omitting this may result in the Access Denied error.

6. Perform the following command via AWS Command Line Interface (CLI) to provide proper permissions to Beeswax for the file: 

	 aws s3api put-object-acl \

	    --profile <assumeBeeswaxRoleRrofile> \
	    --bucket beeswax-data-us-east-1 \
	    --key <user-list or bid_models>/<customer_buzz_key>/file_name.txt \
	    --acl bucket-owner-full-control \
	    --region us-east-1

Note: The placeholder <user-list or bid_models> must be replaced depending on what file type you are uploading. The placeholder <customer_buzz_key> must be replaced with your actual buzz key. Please ensure the bucket and region values are replaced when uploading files to EMEA or APAC. The placeholder <assumeBeeswaxRoleRrofile> should be replaced with whatever you named your profile you created in step 4.

7. [SEGMENT UPLOAD ONLY] Execute a segment_upload POST call via Buzz API that refers to the file location on S3. An example call can be found below:It should look something like:

curl -X POST "https://<customer_buzz_key>.api.beeswax.com/rest/segment_upload" -b cookies.txt -d '{"segment_file_list":["s3://beeswax-data-us-east-1/user-list/<customer_buzz_key>/file_name.txt"],"account_id":2,"file_format":"DELIMITED","user_id_type":"OTHER_MOBILE_ID","segment_key_type":"DEFAULT"}'

Note: The placeholder <customer_buzz_key> must be replaced with your actual buzz key. The "account_id" must be replaced with the account ID of the account you would like the segments to live in (details on accounts can be found here). The default account ID is 2, however, if your buzz key has more than one account, this may be a different ID. 

Note: If uploading users to the EMEA region, you will need to include a "continent":"EMEA" field and value in your payload.

8. [SEGMENT UPLOAD ONLY] You will be able to track the progress of the processing of the file in the Segment Uploads screen of the UI.

Find more detail on the API calls here, and details on the segment_upload post call specifically here.

FAQ

Q: Can bid models uploaded to a regional AWS bucket be utilized globally?

A: Yes, when a bid model is uploaded Beeswax will replicate that bid model data and load into every region where the customer has a bidder active.