# A/B Split Testing with Custom Brain

A/B split testing is useful for comparing model performance across the same campaign or strategy.  This is accomplished by applying multiple models to a given campaign or strategy thereby ensuring all targeting and external settings are consistent between the test groups. All bid opportunities will be divided up randomly in the proportions selected by the user on the basis of the MediaMath UUID or CID.

## A/B Split Targeting

The MediaMath UUID or CID present on the incoming bid opportunity is used to generate a hash (range: [0, 99]) and then route the bid request to the corresponding Model. The models are targeting by configuring a low/high target area in the campaign settings. The model for which the hash result falls in the target area will be executed. It is required to have 100% coverage of the entire hash range using one or more models. The following example shows a 50/50 split for two models.

| Model | campaign | low | high |
|  --- | --- | --- | --- |
| myorg_999999_1569332672_a | 999999 | 0 | 49 |
| myorg_999999_1569332672_b | 999999 | 50 | 99 |


## Using the BYOA API to Configure an A/B Split for a Campaign

Once all the models that will participate in the A/B split testing are uploaded, the campaign settings can be configured to utilize those models.

The following example for a hypothetical Campaign `999999` illustrates the Campaign Level Split in which 80% of UUIDs will be evaluated using `myorg_999999_1569332672_a` and the remaining 20% will use `myorg_999999_1569332672_b`.
In this case, there is no `Strategy ID` present, so all bid opportunities that match the `Campaign ID` will be executed using either of these two models. The model chosen will be determined by the calculated `UUID` hash.

### A/B Split Campaign Example


```bash
curl -X PUT "https://api.byoa.mediamath.com/campaign_settings/999999" \
 -d '{
    "settings": [
      {
        "executor_id": 2,
        "high": 79,
        "low": 0,
        "model_id": "myorg_999999_1569332672_a",
        "namespace": "<OrganizationID>"
      },
      {
        "executor_id": 2,
        "high": 99,
        "low": 80,
        "model_id": "myorg_999999_1569332672_b",
        "namespace": "<OrganizationID>"
      }
    ],
    "uuid_type": "UUID"
  }'
  -H 'authorization: Bearer <oauth_access_token>'
```

- If the UUID hash falls in the range low=0, high=79 the bid request will be evaluated using model `myorg_999999_1569332672_a`.
- If the UUID hash falls in the range low=80, high=99 the bid request will be evaluated using model `myorg_999999_1569332672_b`.


> If a campaign has only one model, the UUID hash will have a default range of low=0 to high=99. If a campaign has unique models at the strategy level then the A/B split is as explained in the next section.


## Using the BYOA API to Configure an A/B Split for a Strategy

The following example for a hypothetical Campaign `999999` and Strategy `888888` illustrates the Strategy Level Split in which 50% of UUIDs will be evaluated using `myorg_999999_888888_1569332672_a` and 50% will use `myorg_999999_888888_1569332672_b`. In this case, since there is a `Strategy ID` present, only bid opportunities that match the `Campaign ID` and `Strategy ID` will be executed using either of these two models. The model chosen will be determined by the calculated `UUID` hash.

### A/B Split Strategy Example


```bash
curl -X PUT "https://api.byoa.mediamath.com/campaign_settings/999999/strategies/888888" \
  -d '{
    "settings": [
      {
        "executor_id": 2,
        "high": 49,
        "low": 0,
        "model_id": "myorg_999999_888888_1569332672_a",
        "namespace": "<OrganizationID>"
      },
      {
        "executor_id": 2,
        "high": 99,
        "low": 50,
        "model_id": "myorg_999999_888888_1569332672_b",
        "namespace": "<OrganizationID>"
      }
    ],
    "uuid_type": "UUID"
  }'
  -H 'authorization: Bearer <oauth_access_token>'
```

- If the UUID hash falls in the range low=0, high=49 the bid request will be evaluated using model `myorg_999999_888888_1569332672_a`.
- If the UUID hash falls in the range low=50, high=99 the bid request will be evaluated using model `myorg_999999_888888_1569332672_b`.