BOF Config API

  • Version: 2.0
  • Host: api.mediamath.com/opportunity-firehose/v2.0
  • Protocols: https
  • Accepts: application/x-www-form-urlencoded
  • Responds With: application/json
  1. What is BOF?
  2. How is BOF data accessed?
  3. How do I get started with BOF?
  4. How do I get self-service access to my BOF config file?
  5. List of Filters
  6. Example Filter Implementation
  7. List of Tier 1 Exchanges (X-Mm-Exch-Id)
  8. List of Channels (X-Mm-Channel-Type)
  9. List of Geos (X-Mm-Geo)
  10. Tell me more about Role ARN and buckets
  11. Partition Macros for S3/Google cloud users
  12. JSON Object Capabilities

What is BOF?

The Bid Request Firehose (BOF), is a high performance system designed to ingest bid opportunities from MediaMath bidders and filter and batch these requests to third parties based on configurable parameters.

How is BOF data accessed?

In order to grant access to the BOF, a config file, called a consumer, must be set-up for the internal/external client. There are 2 types of consumers that are available:

  1. Web endpoint - This can significantly reduce client data response time but requires the client to operate a web cluster capable of ingesting the traffic.

  2. S3 endpoint - This is for clients who are less worried about the delivery time of the data, and prefer to receive the data in larger batch sizes (30,000). This is typically the lowest cost consumer for clients.

Compression Types: LZOP, GZIP

Format Types: Json Array, JSONL

How do I get started with BOF?

Setting up a web or s3 consumer takes minimal effort from the MM side. To get set up, please contact your Account representative and provide the following information:

  • If S3 consumer: Role ARN (see below for more details)
  • If S3 consumer: S3 bucket name(s). You will need to create and provide access to a bucket with a role for BOF to deliver files to it. See below for more details.
    • Note: BOF V2 offers cross-region data replication meaning you do not have to worry about creating multiple buckets e.g all your regional data can be sent to us-east-1
  • Email Contact
  • Which regions you are interested in capturing traffic from i.e US, EMEA, or APAC
  • Sample rate (0-100%): If you plan on receiving data from all of the US you should start with a 1% sample. For reference, as of early March 2020, the United States sees approximately 4 million requests a second during the day, so with a 1% filter and a deployment in ewr/ord/pao (United States) you will see ~40k requests per second. This is plenty of data for a sample.
  • At least one of the following filters (see below for more details):
    • X-Mm-Geo
    • X-Mm-Deal-Id
    • X-Mm-Exch-Id
    • X-Mm-Channel-Type

How do I get self-service access to my BOF config file?

As part of the release of BOF v2.0, clients will now have self-service access to their BOF config files. Specifically, they will be able to dynamically update their sample rate, and/or filters without having to engage the MM team. In order to gain self-service access all consumers will need an auth0 account with MediaMath. Please contact developers@mediamath.com to recieve your auth0 client id and secret and then proceed with the following steps:

Curl auth.mediamath.com with your client ID and secret

POST https://auth.mediamath.com/oauth/token

Example Request:

curl -X POST --header  'content-type: application/json' 'https://auth.mediamath.com/oauth/token' -d '{"audience": "https://api.mediamath.com/opportunity_firehose/", "grant_type":"client_credentials", "client_id":"{CLIENT_ID}", "client_secret":"{CLIENT_SECRET}"}'

Example Response:

{"access_token":"ACCESS_TOKEN_HERE","expires_in":86400,"token_type":"Bearer"}

Use this token here when sending requests to https://api.mediamath.com/opportunity-firehose/v2.0/

curl -X GET -H "Authorization:Bearer ACCESS_TOKEN_HERE" 'https://api.mediamath.com/opportunity-firehose/v2.0/'

Additional Information

In addition to the API’s shown in the left column of your display, the following information will help you with updates to your BOF configurations.

List of Filters

[
  "X-Mm-Exch-Id",
  "X-Mm-Deal-Id",
  "X-Mm-Channel-Type",
  "X-Mm-Geo"
]

Example Filter Implementation

"filters": {
"X-Mm-Channel-Type": ["DISPLAY","VIDEO","SOCIAL","MOBILE_DISPLAY_MOBILE_WEB","MOBILE_VIDEO_MOBILE_WEB","SEARCH","EMAIL","NEWSFEED"],
"X-Mm-Exch-Id": ["3","4","5","9","13","15","30"],
"X-Mm-Geo": ["60231"]
  }

The filters are additive (they are AND’ed together), so the more filters, the smaller the data set. The above filters will result in seeing request of only those channel types AND only those exchange ids AND only that geo.

List of Tier 1 Exchanges (X-Mm-Exch-Id)

{
  "15": "Index Exchange",
  "42": "Sonobi",
  "62": "TripleLift Wizard",
  "63": "LoopMe",
  "65": "Kargo",
  "69": "AdColony",
  "70": "Triton Audio",
  "71": "GumGum",
  "72": "Teads",
  "73": "33 Across",
  "74": "Unruly",
  "75": "OneTag",
  "76": "YieldMo",
  "83": "ShareThrough",
  "87": "Smaato",
  "89": "TrustX",
  "91": "ADman Media",
  "92": "Taboola",
  "94": "AdsWizz",
  "96": "Unity"
}

List of Channels (X-Mm-Channel-Type)

{
  "DISPLAY",
  "VIDEO",
  "SOCIAL",
  "MOBILE_DISPLAY_MOBILE_WEB",
  "MOBILE_VIDEO_MOBILE_WEB",
  "SEARCH",
  "EMAIL",
  "MOBILE_DISPLAY_IN_APP",
  "MOBILE_VIDEO_IN_APP",
  "NEWSFEED"
}

List of Geos (X-Mm-Geo)

{
  "60000": "Unknown",
  "60001": "Other",
  "60002": "Aruba",
  "60003": "Afghanistan",
  "60004": "Angola",
  "60005": "Anguilla",
  "60006": "Aland Islands",
  "60007": "Albania",
  "60008": "Andorra",
  "60009": "Netherlands Antilles",
  "60010": "United Arab Emirates",
  "60011": "Argentina",
  "60012": "Armenia",
  "60013": "American Samoa",
  "60014": "Antarctica",
  "60015": "French Southern Territories",
  "60016": "Antigua And Barbuda",
  "60017": "Australia",
  "60019": "Azerbaijan",
  "60020": "Burundi",
  "60022": "Benin",
  "60023": "Burkina Faso",
  "60024": "Bangladesh",
  "60026": "Bahrain",
  "60027": "Bahamas",
  "60028": "Bosnia And Herzegowina",
  "60029": "Belarus",
  "60030": "Belize",
  "60031": "Bermuda",
  "60032": "Bolivia",
  "60033": "Brazil",
  "60034": "Barbados",
  "60035": "Brunei Darussalam",
  "60036": "Bhutan",
  "60037": "Bouvet Island",
  "60038": "Botswana",
  "60039": "Central African Republic",
  "60040": "Canada",
  "60041": "Cocos (Keeling) Islands",
  "60042": "Switzerland",
  "60043": "Chile",
  "60044": "China",
  "60045": "Cote D Ivoire",
  "60046": "Cameroon",
  "60047": "Congo - The Democratic Rep Of",
  "60048": "Congo",
  "60049": "Cook Islands",
  "60050": "Colombia",
  "60051": "Comoros",
  "60052": "Cape Verde",
  "60053": "Costa Rica",
  "60054": "Cuba",
  "60055": "Christmas Island",
  "60056": "Cayman Islands",
  "60060": "Djibouti",
  "60061": "Dominica",
  "60063": "Dominican Republic",
  "60064": "Algeria",
  "60065": "Ecuador",
  "60066": "Egypt",
  "60067": "Eritrea",
  "60068": "Western Sahara",
  "60071": "Ethiopia",
  "60073": "Fiji",
  "60074": "Falkland Islands (Malvinas)",
  "60076": "Faroe Islands",
  "60077": "Micronesia - Federated States Of",
  "60078": "Gabon",
  "60080": "Georgia",
  "60081": "Guernsey",
  "60082": "Ghana",
  "60083": "Gibraltar",
  "60084": "Guinea",
  "60085": "Guadeloupe",
  "60086": "Gambia",
  "60087": "Guinea-Bissau",
  "60088": "Equatorial Guinea",
  "60090": "Grenada",
  "60091": "Greenland",
  "60092": "Guatemala",
  "60093": "French Guiana",
  "60094": "Guam",
  "60095": "Guyana",
  "60096": "Hong Kong",
  "60097": "Heard And Mc Donald Islands",
  "60098": "Honduras",
  "60100": "Haiti",
  "60102": "Indonesia",
  "60103": "Isle Of Man",
  "60104": "India",
  "60105": "British Indian Ocean Territory",
  "60107": "Iran (Islamic Republic Of)",
  "60108": "Iraq",
  "60110": "Israel",
  "60112": "Jamaica",
  "60113": "Jersey",
  "60114": "Jordan",
  "60115": "Japan",
  "60116": "Kazakhstan",
  "60117": "Kenya",
  "60118": "Kyrgyzstan",
  "60119": "Cambodia",
  "60120": "Kiribati",
  "60121": "Saint Kitts And Nevis",
  "60122": "Korea - South",
  "60123": "Kuwait",
  "60124": "Lao Peoples Democratic Republic",
  "60125": "Lebanon",
  "60126": "Liberia",
  "60127": "Libyan Arab Jamahiriya",
  "60128": "Saint Lucia",
  "60130": "Sri Lanka",
  "60131": "Lesotho",
  "60135": "Macau",
  "60136": "Morocco",
  "60137": "Monaco",
  "60138": "Moldova - Republic Of",
  "60139": "Madagascar",
  "60140": "Maldives",
  "60141": "Mexico",
  "60142": "Marshall Islands",
  "60143": "Macedonia - The Frm Yugoslav Rep Of",
  "60144": "Mali",
  "60146": "Myanmar",
  "60147": "Montenegro",
  "60148": "Mongolia",
  "60149": "Northern Mariana Islands",
  "60150": "Mozambique",
  "60151": "Mauritania",
  "60152": "Montserrat",
  "60153": "Martinique",
  "60154": "Mauritius",
  "60155": "Malawi",
  "60156": "Malaysia",
  "60157": "Mayotte",
  "60158": "Namibia",
  "60159": "New Caledonia",
  "60160": "Niger",
  "60161": "Norfolk Island",
  "60162": "Nigeria",
  "60163": "Nicaragua",
  "60164": "Niue",
  "60167": "Nepal",
  "60168": "Nauru",
  "60169": "New Zealand",
  "60170": "Oman",
  "60171": "Pakistan",
  "60172": "Panama",
  "60173": "Pitcairn",
  "60174": "Peru",
  "60175": "Philippines",
  "60176": "Palau",
  "60177": "Papua New Guinea",
  "60179": "Puerto Rico",
  "60180": "Korea - North",
  "60182": "Paraguay",
  "60183": "Palestinian Territories",
  "60184": "French Polynesia",
  "60185": "Qatar",
  "60186": "Reunion",
  "60188": "Russian Federation",
  "60189": "Rwanda",
  "60190": "Saudi Arabia",
  "60191": "Sudan",
  "60192": "Senegal",
  "60193": "Singapore",
  "60194": "South Georgia / South Sandwich Isl",
  "60195": "St. Helena",
  "60196": "Svalbard And Jan Mayen Islands",
  "60197": "Solomon Islands",
  "60198": "Sierra Leone",
  "60199": "El Salvador",
  "60200": "San Marino",
  "60201": "Somalia",
  "60202": "St. Pierre And Miquelon",
  "60203": "Serbia",
  "60204": "Sao Tome And Principe",
  "60205": "Suriname",
  "60209": "Swaziland",
  "60210": "Seychelles",
  "60211": "Syrian Arab Republic",
  "60212": "Turks And Caicos Islands",
  "60213": "Chad",
  "60214": "Togo",
  "60215": "Thailand",
  "60216": "Tajikistan",
  "60217": "Tokelau",
  "60218": "Turkmenistan",
  "60219": "Timor-Leste",
  "60220": "Tonga",
  "60221": "Trinidad And Tobago",
  "60222": "Tunisia",
  "60223": "Turkey",
  "60224": "Tuvalu",
  "60225": "Taiwan",
  "60226": "Tanzania - United Republic Of",
  "60227": "Uganda",
  "60228": "Ukraine",
  "60229": "Us Minor Outlying Islands",
  "60230": "Uruguay",
  "60231": "United States",
  "60232": "Uzbekistan",
  "60233": "Holy See (Vatican City State)",
  "60234": "Saint Vincent And The Grenadines",
  "60235": "Venezuela",
  "60236": "British Virgin Islands",
  "60237": "Us Virgin Islands",
  "60238": "Viet Nam",
  "60239": "Vanuatu",
  "60240": "Wallis And Futuna Islands",
  "60241": "Samoa",
  "60242": "Yemen",
  "60243": "South Africa",
  "60244": "Zambia",
  "60245": "Zimbabwe",
  "60246": "Asia Pacific",
  "60247": "Europe",
  "72279": "Bonaire/Sint Eustatius/Saba",
  "72280": "South Sudan"
}

Tell me more about Role ARN and buckets

Create an AWS role with at trust relationship of the following

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::888665229551:root",
          "arn:aws:iam::654223338280:root"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

And then give that role Put access to the bucket and/or bucket path

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Put*"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*"
        }
    ]
}

Once you have done that, provide us with the role:arn, bucket name, and region you put the bucket in and we can start delivering you files.

If you want to deliver files into a bucket with a prefix such as s3://mediamath/fileshere tell us the prefix and we can deliver the files there.

The delivery partitions are as follows.

prefix/HexValue (0-F)/Year-Month-Day/Hour/appUUID_Year-Month-Day-Hour-Minute_opportunities_UTCMS.json.lzo

The numbered prefixes are there for high volume consumers based on the best practices for s3 delivery on aws.

I’ve created some terraform code to reflect this relationship.

resource "aws_s3_bucket" "bof_ingest" {
  bucket = "bof-ingest"
  region = "us-east-1"

  acl = "private"
}

resource "aws_iam_role" "bof_ingest_service_role" {
  name = "bof_ingest_service_role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::888665229551:root",
          "arn:aws:iam::654223338280:root"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
      }
    }
  ]
}
EOF
}

resource "aws_iam_policy" "bof_ingest_service_role_policy" {
  name = "bof_ingest_service_role_policy"
  path = "/"

  policy = <<EOF
{
	"Version": "2012-10-17",
	"Statement": [{
		"Action": [
			"s3:Put*"
		],
		"Effect": "Allow",
		"Resource": "${aws_s3_bucket.bof_ingest.arn}/*"
	}]
}
EOF
}

resource "aws_iam_role_policy_attachment" "bof_ingest_service_role_policy_attachment" {
  role       = "${aws_iam_role.bof_ingest_service_role.name}"
  policy_arn = "${aws_iam_policy.bof_ingest_service_role_policy.arn}"
}

Partition Macros for S3/Google cloud users

Partition Macros can be set inside of your AwsSettings. You can change it by posting a macro to the following POST /consumers/{name}/awssettings

Example post data: {"PartitionMacro": "%hex%/%yyyy%-%mm%-%dd%/%hh%/%appuuid%_%yyyy%-%mm%-%dd%-%hh%-%mm%opportunities%unixms%.jsonl.lzo"}

%yyyy% = year %mm% = month %dd% = day %hh% = hour %mi% = minute %ss% = seconds

%hex% = hex value 0-F %unix% = unix time stamp %unixms% = unix millisecond stamp %nano% = unix nanosecond stamp

%appuuid% = our internal app UUID which is just a generated UUID

You can change the entire delivery of your path with these macros.

The default delivery partition macro is

"%hex%/%yyyy%-%mm%-%dd%/%hh%/%appuuid%_%yyyy%-%mm%-%dd%-%hh%-%mm%opportunities%unixms%.jsonl.lzo"

If you were to create a partition macro with the following value

"%yyyy%%mm%%dd%/%hh%/%appuuid%_%nano%.jsonl.lzop

It would expand to

20200402/14/b63aeb5b-9afa-4a42-9d33-b024fd96743f_1585853925137647046.jsonl.lzop

We highly suggest keeping %appuuid%_%nano% in the file name. The files are generated from each server and there is no de-duplication, so if you inadvertently overwrite your own files by choosing a path that is not unique enough there’s nothing we can do. This will mean you are being delivered files, charged for that delivery but losing that data.

Also remember that these need to be valid s3 path names. If you create an invalid s3 path with a macro your file delivery will be interrupted until it is resolved.

JSON Object Capabilities

type struct ConsumerSettings {
    DropZeroUID     bool
    DropExchangeUID bool
    DropBody        bool

    Filters map[string][]string

    SampleSettings *sampleSettings

    AwsSettings    *awsSettings
    WebSettings    *webSettings
    GoogleSettings *googleSettings
}
type sampleSettings struct {
	Disabled   bool
	SampleRate int
}
type awsSettings struct {
	Regions                    map[string]string
	Buckets                    map[string]string
	ASE256ServerSideEncryption bool
	PartitionMacro             string
}
type webSettings struct {
	Endpoints map[string]string
}
type googleSettings struct {
	Buckets        map[string]string
	PartitionMacro string
}