Setting up S3 delivery of Custom Bid Router bid request/response logs

The Custom Bid Router bid request/response logs can be configured for delivery to an S3 bucket in your account.

IAM roles are the AWS recommended practice for managing secure access to AWS resources.

The role created in the below steps is used to establish a trusted relationship between your AWS account and the AWS account belonging to MediaMath. An external ID will be provided for you to use during this setup.

1. Create an S3 bucket in your account for Custom Bid Router bid request/response logs to be delivered

If you're just interested in data from the US, you can create your bucket in a single US AWS region. Otherwise, you'll need to set up buckets in multiple regions (see Custom Bid Router Endpoints Regions).

2. Create an IAM role in your AWS account for the Custom Bid Router system to use

Create a new IAM role that the Custom Bid Router system will assume when delivering your data.

3. Assign the role an access policy to the created buckets

The new IAM role will need PUT/READ access to the S3 buckets you created in step 1. You can grant these permissions by assigning the role an access policy that looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "*"
                    ]
                }
            }
        }
    ]
}

4. Create a trust relationship with MediaMath AWS account ID

The IAM role needs to have a trust relationship established with the MediaMath AWS account ID.

Editing the trust relationship of the IAM role:

• Sign in to your AWS portal and go to Identity & Access Management
• Using the navigation menu, open Roles and select the role that you created
• Find the Trust Relationships tab and edit Trust Relationship
• Attach the below policy

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {"AWS": "arn:aws:iam::mediamath-account-id:user/service_bidvaluator-to-s3"},
        "Action": "sts:AssumeRole",
        "Condition": {"StringEquals": {"sts:ExternalId": "mediamath-provided-customer-unique-external-ID"}}
    }]
}

Data Format

File Names and Prefixes

Once the Custom Bid Router service is configured to deliver data to S3, it will start writing files into your bucket(s) using the following prefixes and naming conventions:

• {shard-index}/{yyyy-mm-dd}/{pop}/{utc-nano-time}.jsonl

All times and dates are UTC

Objects are written and delivered to S3 once either of these conditions is satisfied:

• size is > 100000
• log data spans > 10 seconds

File Structure

Custom Bid Router bid request/response logs delivered to S3 are newline-delimited JSON files.

When are the logs being written to the S3 bucket?

The logs with the above stucture can be logged on two different occasions:

1. When we detect timeout (request time based on the X-Timeout header) has been reached and we can't wait any longer to return our response to the MediaMath Bidder.

{  
   "headers":{  
        "X-Timeout":"25"
   },
   "body":{},
   "response":{},
   "endpoint":"http://bid-handler-prod-us-east-1.acme.co/bid/valuate",
   "requestTime":25.504193,
   "timestamp":1568329266994945222
}

2. If we got back the response on time, and the consumer sends back the X-Log-Request header, the entry will be logged to the consumer's S3 bucket.

{  
   "headers":{  
        "X-Mm-Response-Format": "json",
        "X-Mm-Cid": "93e75a86-036a-4000-9679-8eca835cd757",
        "X-Mm-Auction-Id": "024ed76f-4b36-43a9-8217-3ef2c193cfea",
        "X-Mm-Exchange-Id": "4",
        "X-Mm-Uuid": "93e75a86-036a-4000-9679-8eca835cd757",
        "X-Timeout": "23"
   },
   "body":{},
   "response":{  
        "CampaignID":"123456",
        "StrategyID":"1234567",
        "CPM":2.0,
        "CreativeID":"1234567",
        "PmpDealID":"",
        "ModelID":"Custom-Bid-Router-Test-Endpoint:us-east-1"
   },
   "endpoint":"http://bid-handler-prod-us-east-1.acme.co/bid/valuate",
   "requestTime":11.068123,
   "timestamp":1569270083738107340
}