A/B Split Testing with Custom Brain

A/B split testing is useful for comparing model performance across the same campaign or strategy. This is accomplished by applying multiple models to a given campaign or strategy thereby ensuring all targeting and external settings are consistent between the test groups. All bid opportunities will be divided up randomly in the proportions selected by the user on the basis of the MediaMath UUID or CID.

A/B Split Targeting

The MediaMath UUID or CID present on the incoming bid opportunity is used to generate a hash (range: [0, 99]) and then route the bid request to the corresponding Model. The models are targeting by configuring a low/high target area in the campaign settings. The model for which the hash result falls in the target area will be executed. It is required to have 100% coverage of the entire hash range using one or more models. The following example shows a 50/50 split for two models.

Using the BYOA API to Configure an A/B Split for a Campaign

Once all the models that will participate in the A/B split testing are uploaded, the campaign settings can be configured to utilize those models.

The following example for a hypothetical Campaign 999999 illustrates the Campaign Level Split in which 80% of UUIDs will be evaluated using myorg_999999_1569332672_a and the remaining 20% will use myorg_999999_1569332672_b. In this case, there is no Strategy ID present, so all bid opportunities that match the Campaign ID will be executed using either of these two models. The model chosen will be determined by the calculated UUID hash.

A/B Split Campaign Example

curl -X PUT "https://api.byoa.mediamath.com/campaign_settings/999999" \
 -d '{
    "settings": [
      {
        "executor_id": 2,
        "high": 79,
        "low": 0,
        "model_id": "myorg_999999_1569332672_a",
        "namespace": "<OrganizationID>"
      },
      {
        "executor_id": 2,
        "high": 99,
        "low": 80,
        "model_id": "myorg_999999_1569332672_b",
        "namespace": "<OrganizationID>"
      }
    ],
    "uuid_type": "UUID"
  }'
  -H 'authorization: Bearer <oauth_access_token>'

• If the UUID hash falls in the range low=0, high=79 the bid request will be evaluated using model myorg_999999_1569332672_a.
• If the UUID hash falls in the range low=80, high=99 the bid request will be evaluated using model myorg_999999_1569332672_b.

If a campaign has only one model, the UUID hash will have a default range of low=0 to high=99. If a campaign has unique models at the strategy level then the A/B split is as explained in the next section.

Using the BYOA API to Configure an A/B Split for a Strategy

The following example for a hypothetical Campaign 999999 and Strategy 888888 illustrates the Strategy Level Split in which 50% of UUIDs will be evaluated using myorg_999999_888888_1569332672_a and 50% will use myorg_999999_888888_1569332672_b. In this case, since there is a Strategy ID present, only bid opportunities that match the Campaign ID and Strategy ID will be executed using either of these two models. The model chosen will be determined by the calculated UUID hash.

A/B Split Strategy Example

curl -X PUT "https://api.byoa.mediamath.com/campaign_settings/999999/strategies/888888" \
  -d '{
    "settings": [
      {
        "executor_id": 2,
        "high": 49,
        "low": 0,
        "model_id": "myorg_999999_888888_1569332672_a",
        "namespace": "<OrganizationID>"
      },
      {
        "executor_id": 2,
        "high": 99,
        "low": 50,
        "model_id": "myorg_999999_888888_1569332672_b",
        "namespace": "<OrganizationID>"
      }
    ],
    "uuid_type": "UUID"
  }'
  -H 'authorization: Bearer <oauth_access_token>'

• If the UUID hash falls in the range low=0, high=49 the bid request will be evaluated using model myorg_999999_888888_1569332672_a.
• If the UUID hash falls in the range low=50, high=99 the bid request will be evaluated using model myorg_999999_888888_1569332672_b.