# Setting up S3 delivery of Custom Bid Router bid request/response logs

The Custom Bid Router bid request/response logs can be configured for delivery to an `S3 bucket` in your account.

`IAM` roles are the `AWS` recommended practice for managing secure access to `AWS resources`.

The role created in the below steps is used to establish a trusted relationship between your AWS account and the AWS account belonging to MediaMath. An external ID will be provided for you to use during this setup.

## 1. Create an S3 bucket in your account for Custom Bid Router bid request/response logs to be delivered

If you're just interested in data from the US, you can create your bucket in a single US AWS region. Otherwise, you'll need to set up buckets in multiple regions (see Custom Bid Router Endpoints Regions).

## 2. Create an IAM role in your AWS account for the Custom Bid Router system to use

Create a new IAM role that the Custom Bid Router system will assume when delivering your data.

## 3. Assign the role an access policy to the created buckets

The new IAM role will need `PUT/READ` access to the S3 buckets you created in step 1. You can grant these permissions by assigning the role an access policy that looks like this:


```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "*"
                    ]
                }
            }
        }
    ]
}
```

## 4. Create a trust relationship with MediaMath AWS account ID

The IAM role needs to have a trust relationship established with the MediaMath AWS account ID.

Editing the trust relationship of the IAM role:

* Sign in to your `AWS portal` and go to `Identity & Access Management`
* Using the navigation menu, open `Roles` and select the role that you created
* Find the `Trust Relationships` tab and edit `Trust Relationship`
* Attach the below policy



```json
{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {"AWS": "arn:aws:iam::mediamath-account-id:user/service_bidvaluator-to-s3"},
        "Action": "sts:AssumeRole",
        "Condition": {"StringEquals": {"sts:ExternalId": "mediamath-provided-customer-unique-external-ID"}}
    }]
}
```

## Data Format

### File Names and Prefixes

Once the Custom Bid Router service is configured to deliver data to S3, it will start writing files into your `bucket(s)` using the following prefixes and naming conventions:

* `{shard-index}/{yyyy-mm-dd}/{pop}/{utc-nano-time}.jsonl`


**All times and dates are UTC**

Objects are written and delivered to `S3` once either of these conditions is satisfied:

* size is > 100000
* log data spans > 10 seconds


### File Structure

Custom Bid Router bid request/response logs delivered to `S3` are newline-delimited JSON files.

| Section | Description |
|  --- | --- |
| Headers | Headers Custom Bid Router sent in the request to the consumer |
| Body | Body Custom Bid Router sent the consumer |
| Response | Response Custom Bid Router received from the consumer |
| Endpoint | Consumer endpoint where Custom Bid Router sent the request |
| RequestTime | Roundtrip time in ms |
| TimeStamp | Timestamp(nano) Custom Bid Router started processing request |


### When are the logs being written to the S3 bucket?

The logs with the above stucture can be logged on two different occasions:

1. When we detect timeout (request time based on the `X-Timeout` header) has been reached and we can't wait any longer to return our response to the MediaMath Bidder.



```
{  
   "headers":{  
        "X-Timeout":"25"
   },
   "body":{},
   "response":{},
   "endpoint":"http://bid-handler-prod-us-east-1.acme.co/bid/valuate",
   "requestTime":25.504193,
   "timestamp":1568329266994945222
}
```

1. If we got back the response on time, and the consumer sends back the `X-Log-Request` header, the entry will be logged to the consumer's S3 bucket.



```
{  
   "headers":{  
        "X-Mm-Response-Format": "json",
        "X-Mm-Cid": "93e75a86-036a-4000-9679-8eca835cd757",
        "X-Mm-Auction-Id": "024ed76f-4b36-43a9-8217-3ef2c193cfea",
        "X-Mm-Exchange-Id": "4",
        "X-Mm-Uuid": "93e75a86-036a-4000-9679-8eca835cd757",
        "X-Timeout": "23"
   },
   "body":{},
   "response":{  
        "CampaignID":"123456",
        "StrategyID":"1234567",
        "CPM":2.0,
        "CreativeID":"1234567",
        "PmpDealID":"",
        "ModelID":"Custom-Bid-Router-Test-Endpoint:us-east-1"
   },
   "endpoint":"http://bid-handler-prod-us-east-1.acme.co/bid/valuate",
   "requestTime":11.068123,
   "timestamp":1569270083738107340
}
```