- What is BOF?
- How is BOF data accessed?
- How do I get started with BOF?
- How do I get self-service access to my BOF config file?
- List of Filters
- Example Filter Implementation
- List of Tier 1 Exchanges (X-Mm-Exch-Id)
- List of Channels (X-Mm-Channel-Type)
- List of Geos (X-Mm-Geo)
- Tell me more about Role ARN and buckets
- Partition Macros for S3/Google cloud users
- JSON Object Capabilities
BOF Config API (2.0)
The Bid Request Firehose (BOF), is a high performance system designed to ingest bid opportunities from MediaMath bidders and filter and batch these requests to third parties based on configurable parameters.
In order to grant access to the BOF, a config file, called a consumer, must be set-up for the internal/external client. There are 2 types of consumers that are available:
Web endpoint - This can significantly reduce client data response time but requires the client to operate a web cluster capable of ingesting the traffic.
S3 endpoint - This is for clients who are less worried about the delivery time of the data, and prefer to receive the data in larger batch sizes (30,000). This is typically the lowest cost consumer for clients.
Compression Types: LZOP, GZIP
Format Types: Json Array, JSONL
Setting up a web or s3 consumer takes minimal effort from the MM side. To get set up, please contact your Account representative and provide the following information:
- If S3 consumer: Role ARN (see below for more details)
- If S3 consumer: S3 bucket name(s). You will need to create and provide access to a bucket with a role for BOF to deliver files to it. See below for more details.
- Note: BOF V2 offers cross-region data replication meaning you do not have to worry about creating multiple buckets e.g all your regional data can be sent to us-east-1
- Email Contact
- Which regions you are interested in capturing traffic from i.e US, EMEA, or APAC
- Sample rate (0-100%): If you plan on receiving data from all of the US you should start with a 1% sample. For reference, as of early March 2020, the United States sees approximately 4 million requests a second during the day, so with a 1% filter and a deployment in ewr/ord/pao (United States) you will see ~40k requests per second. This is plenty of data for a sample.
- At least one of the following filters (see below for more details):
- X-Mm-Geo
- X-Mm-Deal-Id
- X-Mm-Exch-Id
- X-Mm-Channel-Type
As part of the release of BOF v2.0, clients will now have self-service access to their BOF config files. Specifically, they will be able to dynamically update their sample rate, and/or filters without having to engage the MM team. In order to gain self-service access all consumers will need an auth0 account with MediaMath. Please contact developers@mediamath.com to recieve your auth0 client id and secret and then proceed with the following steps:
Curl auth.mediamath.com with your client ID and secret
POST https://auth.mediamath.com/oauth/token
Example Request:
curl -X POST --header 'content-type: application/json' 'https://auth.mediamath.com/oauth/token' -d '{"audience": "https://api.mediamath.com/opportunity_firehose/", "grant_type":"client_credentials", "client_id":"{CLIENT_ID}", "client_secret":"{CLIENT_SECRET}"}'Example Response:
{"access_token":"ACCESS_TOKEN_HERE","expires_in":86400,"token_type":"Bearer"}Use this token here when sending requests to https://api.mediamath.com/opportunity-firehose/v2.0/
curl -X GET -H "Authorization:Bearer ACCESS_TOKEN_HERE" 'https://api.mediamath.com/opportunity-firehose/v2.0/'
In addition to the API's shown in the left column of your display, the following information will help you with updates to your BOF configurations.
"filters": {
"X-Mm-Channel-Type": ["DISPLAY","VIDEO","SOCIAL","MOBILE_DISPLAY_MOBILE_WEB","MOBILE_VIDEO_MOBILE_WEB","SEARCH","EMAIL","NEWSFEED"],
"X-Mm-Exch-Id": ["3","4","5","9","13","15","30"],
"X-Mm-Geo": ["60231"]
}The filters are additive (they are AND'ed together), so the more filters, the smaller the data set. The above filters will result in seeing request of only those channel types AND only those exchange ids AND only that geo.
{
"15": "Index Exchange",
"42": "Sonobi",
"62": "TripleLift Wizard",
"63": "LoopMe",
"65": "Kargo",
"69": "AdColony",
"70": "Triton Audio",
"71": "GumGum",
"72": "Teads",
"73": "33 Across",
"74": "Unruly",
"75": "OneTag",
"76": "YieldMo",
"83": "ShareThrough",
"87": "Smaato",
"89": "TrustX",
"91": "ADman Media",
"92": "Taboola",
"94": "AdsWizz",
"96": "Unity"
}{
"DISPLAY",
"VIDEO",
"SOCIAL",
"MOBILE_DISPLAY_MOBILE_WEB",
"MOBILE_VIDEO_MOBILE_WEB",
"SEARCH",
"EMAIL",
"MOBILE_DISPLAY_IN_APP",
"MOBILE_VIDEO_IN_APP",
"NEWSFEED"
}{
"60000": "Unknown",
"60001": "Other",
"60002": "Aruba",
"60003": "Afghanistan",
"60004": "Angola",
"60005": "Anguilla",
"60006": "Aland Islands",
"60007": "Albania",
"60008": "Andorra",
"60009": "Netherlands Antilles",
"60010": "United Arab Emirates",
"60011": "Argentina",
"60012": "Armenia",
"60013": "American Samoa",
"60014": "Antarctica",
"60015": "French Southern Territories",
"60016": "Antigua And Barbuda",
"60017": "Australia",
"60019": "Azerbaijan",
"60020": "Burundi",
"60022": "Benin",
"60023": "Burkina Faso",
"60024": "Bangladesh",
"60026": "Bahrain",
"60027": "Bahamas",
"60028": "Bosnia And Herzegowina",
"60029": "Belarus",
"60030": "Belize",
"60031": "Bermuda",
"60032": "Bolivia",
"60033": "Brazil",
"60034": "Barbados",
"60035": "Brunei Darussalam",
"60036": "Bhutan",
"60037": "Bouvet Island",
"60038": "Botswana",
"60039": "Central African Republic",
"60040": "Canada",
"60041": "Cocos (Keeling) Islands",
"60042": "Switzerland",
"60043": "Chile",
"60044": "China",
"60045": "Cote D Ivoire",
"60046": "Cameroon",
"60047": "Congo - The Democratic Rep Of",
"60048": "Congo",
"60049": "Cook Islands",
"60050": "Colombia",
"60051": "Comoros",
"60052": "Cape Verde",
"60053": "Costa Rica",
"60054": "Cuba",
"60055": "Christmas Island",
"60056": "Cayman Islands",
"60060": "Djibouti",
"60061": "Dominica",
"60063": "Dominican Republic",
"60064": "Algeria",
"60065": "Ecuador",
"60066": "Egypt",
"60067": "Eritrea",
"60068": "Western Sahara",
"60071": "Ethiopia",
"60073": "Fiji",
"60074": "Falkland Islands (Malvinas)",
"60076": "Faroe Islands",
"60077": "Micronesia - Federated States Of",
"60078": "Gabon",
"60080": "Georgia",
"60081": "Guernsey",
"60082": "Ghana",
"60083": "Gibraltar",
"60084": "Guinea",
"60085": "Guadeloupe",
"60086": "Gambia",
"60087": "Guinea-Bissau",
"60088": "Equatorial Guinea",
"60090": "Grenada",
"60091": "Greenland",
"60092": "Guatemala",
"60093": "French Guiana",
"60094": "Guam",
"60095": "Guyana",
"60096": "Hong Kong",
"60097": "Heard And Mc Donald Islands",
"60098": "Honduras",
"60100": "Haiti",
"60102": "Indonesia",
"60103": "Isle Of Man",
"60104": "India",
"60105": "British Indian Ocean Territory",
"60107": "Iran (Islamic Republic Of)",
"60108": "Iraq",
"60110": "Israel",
"60112": "Jamaica",
"60113": "Jersey",
"60114": "Jordan",
"60115": "Japan",
"60116": "Kazakhstan",
"60117": "Kenya",
"60118": "Kyrgyzstan",
"60119": "Cambodia",
"60120": "Kiribati",
"60121": "Saint Kitts And Nevis",
"60122": "Korea - South",
"60123": "Kuwait",
"60124": "Lao Peoples Democratic Republic",
"60125": "Lebanon",
"60126": "Liberia",
"60127": "Libyan Arab Jamahiriya",
"60128": "Saint Lucia",
"60130": "Sri Lanka",
"60131": "Lesotho",
"60135": "Macau",
"60136": "Morocco",
"60137": "Monaco",
"60138": "Moldova - Republic Of",
"60139": "Madagascar",
"60140": "Maldives",
"60141": "Mexico",
"60142": "Marshall Islands",
"60143": "Macedonia - The Frm Yugoslav Rep Of",
"60144": "Mali",
"60146": "Myanmar",
"60147": "Montenegro",
"60148": "Mongolia",
"60149": "Northern Mariana Islands",
"60150": "Mozambique",
"60151": "Mauritania",
"60152": "Montserrat",
"60153": "Martinique",
"60154": "Mauritius",
"60155": "Malawi",
"60156": "Malaysia",
"60157": "Mayotte",
"60158": "Namibia",
"60159": "New Caledonia",
"60160": "Niger",
"60161": "Norfolk Island",
"60162": "Nigeria",
"60163": "Nicaragua",
"60164": "Niue",
"60167": "Nepal",
"60168": "Nauru",
"60169": "New Zealand",
"60170": "Oman",
"60171": "Pakistan",
"60172": "Panama",
"60173": "Pitcairn",
"60174": "Peru",
"60175": "Philippines",
"60176": "Palau",
"60177": "Papua New Guinea",
"60179": "Puerto Rico",
"60180": "Korea - North",
"60182": "Paraguay",
"60183": "Palestinian Territories",
"60184": "French Polynesia",
"60185": "Qatar",
"60186": "Reunion",
"60188": "Russian Federation",
"60189": "Rwanda",
"60190": "Saudi Arabia",
"60191": "Sudan",
"60192": "Senegal",
"60193": "Singapore",
"60194": "South Georgia / South Sandwich Isl",
"60195": "St. Helena",
"60196": "Svalbard And Jan Mayen Islands",
"60197": "Solomon Islands",
"60198": "Sierra Leone",
"60199": "El Salvador",
"60200": "San Marino",
"60201": "Somalia",
"60202": "St. Pierre And Miquelon",
"60203": "Serbia",
"60204": "Sao Tome And Principe",
"60205": "Suriname",
"60209": "Swaziland",
"60210": "Seychelles",
"60211": "Syrian Arab Republic",
"60212": "Turks And Caicos Islands",
"60213": "Chad",
"60214": "Togo",
"60215": "Thailand",
"60216": "Tajikistan",
"60217": "Tokelau",
"60218": "Turkmenistan",
"60219": "Timor-Leste",
"60220": "Tonga",
"60221": "Trinidad And Tobago",
"60222": "Tunisia",
"60223": "Turkey",
"60224": "Tuvalu",
"60225": "Taiwan",
"60226": "Tanzania - United Republic Of",
"60227": "Uganda",
"60228": "Ukraine",
"60229": "Us Minor Outlying Islands",
"60230": "Uruguay",
"60231": "United States",
"60232": "Uzbekistan",
"60233": "Holy See (Vatican City State)",
"60234": "Saint Vincent And The Grenadines",
"60235": "Venezuela",
"60236": "British Virgin Islands",
"60237": "Us Virgin Islands",
"60238": "Viet Nam",
"60239": "Vanuatu",
"60240": "Wallis And Futuna Islands",
"60241": "Samoa",
"60242": "Yemen",
"60243": "South Africa",
"60244": "Zambia",
"60245": "Zimbabwe",
"60246": "Asia Pacific",
"60247": "Europe",
"72279": "Bonaire/Sint Eustatius/Saba",
"72280": "South Sudan"
}Create an AWS role with at trust relationship of the following
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::888665229551:root",
"arn:aws:iam::654223338280:root"
]
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}And then give that role Put access to the bucket and/or bucket path
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:Put*"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*"
}
]
}Once you have done that, provide us with the role:arn, bucket name, and region you put the bucket in and we can start delivering you files.
If you want to deliver files into a bucket with a prefix such as s3://mediamath/fileshere tell us the prefix and we can deliver the files there.
The delivery partitions are as follows.
prefix/HexValue (0-F)/Year-Month-Day/Hour/appUUID_Year-Month-Day-Hour-Minute_opportunities_UTCMS.json.lzo
The numbered prefixes are there for high volume consumers based on the best practices for s3 delivery on aws.
I've created some terraform code to reflect this relationship.
resource "aws_s3_bucket" "bof_ingest" {
bucket = "bof-ingest"
region = "us-east-1"
acl = "private"
}
resource "aws_iam_role" "bof_ingest_service_role" {
name = "bof_ingest_service_role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::888665229551:root",
"arn:aws:iam::654223338280:root"
]
},
"Action": "sts:AssumeRole",
"Condition": {
}
}
]
}
EOF
}
resource "aws_iam_policy" "bof_ingest_service_role_policy" {
name = "bof_ingest_service_role_policy"
path = "/"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [{
"Action": [
"s3:Put*"
],
"Effect": "Allow",
"Resource": "${aws_s3_bucket.bof_ingest.arn}/*"
}]
}
EOF
}
resource "aws_iam_role_policy_attachment" "bof_ingest_service_role_policy_attachment" {
role = "${aws_iam_role.bof_ingest_service_role.name}"
policy_arn = "${aws_iam_policy.bof_ingest_service_role_policy.arn}"
}Partition Macros can be set inside of AwsSettings/GoogleSettings. You can change it by posting a macro to the following: POST /consumers/{name}/awssettings or POST /consumers/{name}/googlesettings
Example post data: {"PartitionMacro": "%hex%/%yyyy%-%mm%-%dd%/%hh%/%appuuid%_%yyyy%-%mm%-%dd%-%hh%-%mm%opportunities%unixms%.jsonl.lzo"}
%yyyy% = year %mm% = month %dd% = day %hh% = hour %mi% = minute %ss% = seconds
%hex% = hex value 0-F %unix% = unix time stamp %unixms% = unix millisecond stamp %nano% = unix nanosecond stamp
%appuuid% = our internal app UUID which is just a generated UUID
You can change the entire delivery of your path with these macros.
The default delivery partition macro is
"%hex%/%yyyy%-%mm%-%dd%/%hh%/%appuuid%_%yyyy%-%mm%-%dd%-%hh%-%mm%opportunities%unixms%.jsonl.lzo"
If you were to create a partition macro with the following value
"%yyyy%%mm%%dd%/%hh%/%appuuid%_%nano%.jsonl.lzop
It would expand to
20200402/14/b63aeb5b-9afa-4a42-9d33-b024fd96743f_1585853925137647046.jsonl.lzop
We highly suggest keeping %appuuid%_%nano% in the file name. The files are generated from each server and there is no de-duplication, so if you inadvertently overwrite your own files by choosing a path that is not unique enough there's nothing we can do. This will mean you are being delivered files, charged for that delivery but losing that data.
Also remember that these need to be valid s3 path names. If you create an invalid s3 path with a macro your file delivery will be interrupted until it is resolved.
type struct ConsumerSettings {
DropZeroUID bool
DropExchangeUID bool
DropBody bool
Filters map[string][]string
SampleSettings *sampleSettings
AwsSettings *awsSettings
WebSettings *webSettings
GoogleSettings *googleSettings
}type sampleSettings struct {
Disabled bool
SampleRate int
}type awsSettings struct {
Regions map[string]string
Buckets map[string]string
ASE256ServerSideEncryption bool
PartitionMacro string
}type webSettings struct {
Endpoints map[string]string
}type googleSettings struct {
Buckets map[string]string
PartitionMacro string
}Download OpenAPI description
Overview
URL
developers@mediamath.com
License
Languages
Servers
https://api.mediamath.com/opportunity-firehose/v2.0