BYOA

  • Host: api.byoa.mediamath.com
  • Protocols: https

Bring Your Own Algorithm

Bring Our Own Algorithm (BYOA) allows advertisers to apply their own bidding algorithms within MediaMath using the Custom Brain service

Custom Brain

In Custom Brain the client uses the BYOA API to upload a set of logistic coefficients corresponding to any of the variables currently in use by the MediaMath Brain. These coefficients will then be used by participating strategies to calculate the predicted response rate for each impression. The bidder will calculate bid price by multipling the predicted response rate by the strategy’s goal value. Goal values can be modified using the T1 campaign management API.

BYOM

Custom Brain setup & data flows (numbers correspond to diagram above)

  1. [Setup] Client uploads a set of logistic coefficients via the BYOA API. See Custom Brain page for sample workflow.
  2. [Setup] Client uses MediaMath’s BYOA API to specify which strategies should use BYOA and whether any A/B testing split should be applied.
  3. New models and any changes to BYOA settings are reflected within MediaMath’s bidder within 10 minutes of the API call.
  4. MediaMath receives a bid request from one of our supply partners.
  5. The MediaMath bidder identifies which strategies are eligible to particpate in the auction for that bid opportunity. In order to be eligible, the bid must match the strategy’s targeting, the strategy must be eligible to spend based on pacing & budgeting, and there must be capacity under the user’s frequency cap for that strategy, campaign, and advertiser. The bid opportunities that have been filtered to meet these criteria will be evaluated using the client’s model to calculate the bid opportunity’s predicted response rate.
  6. Predicted response rate is multiplied by the strategy’s goal value to calculate bid price.
  7. MediaMath includes the this bid price in our internal auction similar to any other strategy.

Custom Brain Sample Workflow

1.1. Login to Adama

 $ curl --request POST \
  --url https://auth.mediamath.com/oauth/token \
  --header 'content-type: application/json' \
  --data '{
  "grant_type":"password",
  "username":"email@example.com",
  "password":"myPassword",
  "audience":"https://api.mediamath.com",
  "client_id":"BDEXb9Pv5GZy55tcogmwz1cnx6qhxJ6l",
  "client_secret":"46v7RCsBjCkjItS2iP1DByhkf97v3vjegQCoLkRKnKq5GfEDSZFj25ddY4mNU9L-"
}'
{
    "access_token": "eyJ0eX...KACzrBhNEg",
    "expires_in": 86400,
    "token_type": "Bearer"
}
$ curl --request GET \
  --url https://api.mediamath.com/api/v2.0/session \
  --header 'Accept:application/vnd.mediamath.v1+json' \
  --header 'Authorization:"Bearer eyJ0eX...KACzrBhNEg"'

1.2. Response from Login to Adama

{
   "data" : {
      "session" : {
         "current_time" : "2017-06-29T14:51:35",
         "sessionid" : "3126caabcd296a2eece9118c84c14093f82f0ec8",
         "expires" : "2017-06-30T14:57:18"
      },
      "entity_type" : "user",
      "name" : "email@example.com",
      "id" : 21370
   },
   "meta" : {
      "status" : "ok"
   }
}
  1. Upload Model
curl -D- -XPUT -d@./model_data.json https://api.byoa.mediamath.com/data/mm/models/model_name \
  -H 'adama_session: 3126caabcd296a2eece9118c84c14093f82f0ec8' \
  -H 'adama_session_exp: 2017-06-30T14:57:18'
  1. Configure Campaign, Select Model for Campaign. Note that executor_id should be set to 2.
$ curl -X PUT "https://api.byoa.mediamath.com/campaign_settings/10000" \
  -d '{
    "settings": [
      {
        "executor_id": 2,
        "high": 99,
        "low": 0,
        "model_id": "model_name",
        "namespace": "mm"
      }
    ],
    "uuid_type": "UUID"
  }' \
  -H 'adama_session: 3126caabcd296a2eece9118c84c14093f82f0ec8' \
  -H 'adama_session_exp: 2017-06-30T14:57:18'

Model Creation

After the user has derived a logistic model, it needs to be encoded using the flatbuffer schema.

Example: Model Data Before and After Flatbuffer Encoding

Goal Type:

CPA

Features:

793788:2.0490670204162598 1017657:-0.0021825090516358614 427957:-0.0069612981751561165 604453:0.5561302900314331 11495:-0.009623249061405659 425339:-2.543164014816284 838922:1.0690099000930786 706879:-0.16286365687847137 164577:-2.832087993621826 649911:-0.02144586481153965 906442:3.492929220199585 1008034:-0.06976734101772308 736219:-2.4745678901672363 158390:-0.00087798445019871 497983:-0.05527660995721817 851244:3.304011344909668 753431:-1.715367317199707 1037817:-4.932645320892334 24640:-8.085592269897461 76624:-0.06591115891933441 30636:-3.0566017627716064 70024:1.9495718479156494 603888:-0.0021209141705185175 191524:-0.007224101573228836 289474:-0.002307330025359988 982865:-1.7661681175231934 1046837:-1.791756510734558 781872:-3.3209569454193115 306246:2.0065507888793945 671773:-0.0008994271047413349 204228:-0.00980205275118351 202880:2.2398664951324463 608803:0.9965876340866089 227315:-0.007938901893794537 1020736:-0.17039285600185394 157842:-0.006363738793879747 477532:-0.008428595960140228 141374:-0.1755569875240326 383359:1.8971534967422485 226538:-2.0928549766540527 310758:-5.498961925506592 981533:-0.0014491173205897212 789842:-3.550835609436035 837563:-2.785093307495117 705453:-0.0003143609210383147 52055:2.9752323627471924 662329:0.1868445724248886 430828:-0.242512509226799 749816:-0.004770004190504551 775489:2.163778305053711 1013390:1.2257739305496216 739284:1.0348016023635864 1033081:-0.0007728830678388476 1013722:1.459721326828003 202416:-0.007225042209029198 516310:5.609863758087158 538355:-3.7682852745056152 553904:-0.13856282830238342 810835:-2.8358609676361084 439959:3.1520450115203857 742879:-0.8310482501983643 705268:-0.02078358829021454 415580:-0.0028995168395340443 470748:-0.011204746551811695 44373:-1.8673909902572632 825376:-0.14783848822116852 448905:-0.006860028486698866 411528:-0.0015557286096736789 824860:2.4473116397857666 696746:1.554240107536316 685609:1.708579421043396 58337:-0.00951989833265543 450279:-3.4480514526367188 276173:-1.7700515985488892 402787:-2.500797986984253 479226:-0.0025257100351154804 235471:-3.962188482284546 647595:3.7923452854156494 781826:-3.119818925857544 323360:0.1943395584821701 730787:-3.5488545894622803 1035232:-3.734067678451538 376846:-1.2135591506958008 849184:-1.149871826171875 742963:-0.0032067440915852785 693288:-0.009427626617252827 201408:1.2002086639404297 368025:-4.881236553192139 803917:-1.2574892044067383 14860:-1.8726115226745605 719585:2.5541913509368896 826494:-0.10751663148403168 89726:1.2589201927185059 382456:-0.007109648548066616 35066:-1.6523818969726562 485412:-2.492065668106079 749004:-2.7057011127471924 1005387:-3.22613263130188 803807:-0.09217073768377304 533382:-0.757358193397522 807524:4.173923015594482 132602:-0.0014492202317342162 151176:-0.11999741196632385 1037439:-3.3707330226898193 322190:-2.1575429439544678 1024074:-0.6038103103637695 383942:-2.587520248198416e-05 993242:-0.0025314544327557087 371873:0.46447739005088806 736159:8.97714900970459 563461:-0.022241288796067238 540412:-0.008914331905543804 359190:2.1467339992523193 71014:-2.2391984462738037 184321:-0.0004651463241316378 821187:2.592735767364502 177416:-3.4919393062591553 919091:-2.962733268737793 25955:2.4335412979125977 900444:-0.15173961222171783 630201:-0.0012233864981681108 744709:-0.013869797810912132 321619:-3.7221338748931885 572811:-5.377411365509033 522263:2.894685745239258 718569:-5.095548152923584 499013:3.231168746948242 264404:-1.7640866041183472 1002068:-0.06658326089382172 596343:-4.661330223083496 405744:-1.8135327100753784 469234:2.4513490200042725 164947:-5.416643142700195 567040:-0.1068623811006546 320720:-0.7120553851127625 154990:0.667290210723877 352328:-0.0028881749603897333 839378:-3.758525848388672 887999:-2.244004011154175 314236:-3.4070193767547607 736013:-0.920697808265686 785222:-4.169840335845947 459160:-5.5368804931640625 54441:-0.0011177302803844213 112082:-4.1835150718688965 214818:-0.0022511885035783052 211980:-0.009937641210854053 321984:-0.9943729639053345 625632:-0.06515834480524063 1043427:2.427412509918213 957252:-0.6287228465080261 689811:2.3210854530334473 846938:-0.7706658244132996 236955:-0.0026100310496985912 774400:-0.013604754582047462 582548:4.129323482513428 967904:1.657650351524353 259007:1.4880952835083008 259980:2.8317525386810303 824764:2.516631841659546 973918:-0.3222847580909729 12585:-0.00039776082849130034 310816:-0.010587526485323906 104739:-3.7002031803131104 672446:-2.3934991359710693 435137:-0.023854056373238564 315367:-0.008320478722453117 165770:-4.354322910308838 381632:-0.0024718833155930042 61304:-6.1946702003479 873781:-1.7179856300354004 373546:-1.1140437126159668 928131:-2.5884088245220482e-05 58889:-0.7320888042449951 932514:1.868207335472107 874388:-0.008679015561938286 888587:-0.0011152613442391157 31276:-0.006020830478519201 322949:1.5043410062789917 875308:2.593449115753174 305201:-0.014687829650938511 1016616:-1.6896647214889526 266316:-0.7724344730377197 782437:-0.030695363879203796 199020:3.6294524669647217 80391:0.9802243709564209 977161:2.9569849967956543 798377:-0.0033411229960620403 642894:-0.012592882849276066 861115:-4.015702247619629 749133:5.301662921905518 563339:-0.4068736433982849 646593:-2.1293206373229623e-05 585534:0.1190125048160553 358322:-1.962149977684021 178492:-4.527097702026367 619906:-3.9470326900482178 29152:0.7312189340591431 665833:-0.020769966766238213 186491:-0.0022713392972946167 29840:-0.0011832626769319177 388105:5.218237400054932 785336:7.3453545570373535 167517:3.176812171936035 860644:-1.0374189615249634 818156:-0.10296870023012161 55025:1.3083971738815308 591932:-2.0647950172424316 378925:-0.022885210812091827 717208:-0.263315886259079 752136:-3.9185259342193604 606304:-0.003351172199472785 258645:-0.9158366322517395 914279:-0.001811492838896811 963647:-0.002815872896462679 581326:-1.5565578937530518 511514:-3.6174070835113525 638683:-0.003548029577359557 842248:-0.006158021278679371 576669:-2.1862313747406006 769567:-4.149776935577393 25518:-0.08242963999509811 166986:0.9255013465881348 40607:3.5665884017944336 370868:-2.0497498512268066 582556:-6.1424455642700195 683492:-0.0024928131606429815 175948:-3.9235775470733643 156166:-7.867849349975586 566690:2.877570629119873 757225:-0.2928065359592438 736506:0.7061265707015991 696370:-0.4404628276824951 830352:-4.936284065246582 752115:-4.2918500900268555 702458:-3.686044692993164 968213:-3.211169481277466 417057:-2.722015142440796 879814:1.7570884227752686 7588:-0.9306135773658752 64393:-0.021845722571015358 617807:-0.003765170695260167 256674:-0.0038862063083797693 149062:-1.1547662019729614 219903:-0.06453467905521393 182202:-0.00031561273499391973 748038:-0.05969460308551788 929172:-0.5052452683448792 661263:5.929258346557617 282526:-3.0383965969085693 519660:-0.746164858341217 577445:-2.4236137866973877 1033585:-0.05227728933095932 832700:-0.002531226957216859 260839:-0.02298763021826744 8271:-0.16972211003303528 1043851:-0.007726801093667746 686281:1.4129809141159058 697303:-0.0006840641726739705 79897:-0.0029270388185977936 128234:6.5300421714782715 672458:2.1980578899383545 1029747:-2.473872184753418 744501:-0.008836443535983562 839399:6.019924163818359 635188:-2.876889228820801 133816:0.2332848459482193 782125:1.6458415985107422 875565:-0.0013482581125572324 742470:-1.1985541582107544 1045878:11.227041244506836 566998:0.5229340195655823 253999:-0.2645968496799469 163237:-0.000131303007947281 670387:-0.025310730561614037 649554:1.4535691738128662 339009:-1.2854511737823486 601557:-0.010552099905908108 942143:-0.0033125472255051136 407677:-3.1272292137145996 603967:-5.185605049133301 541702:-4.558934688568115 742087:-0.30610865354537964 24070:-1.9537278413772583 406022:-8.473625183105469 841317:-3.242177963256836 1035605:2.578131914138794 858386:-0.17704574763774872 984846:-0.0057215564884245396 695051:-3.1152472496032715 289965:-0.07320985198020935 879230:-3.0772857666015625 984372:1.4279719591140747 845407:2.776535987854004 665891:4.95247220993042 905947:2.4297335147857666 610458:4.622753143310547 552644:-0.0031528202816843987 321019:-0.6633411645889282 485782:1.6903525590896606 980529:6.240449905395508 482742:-0.006581943016499281 941039:-0.030571268871426582 599340:1.6603080034255981 702607:-1.2541992664337158 849828:-0.21410101652145386 477683:1.0697791576385498 1020435:4.959871292114258 285142:-0.015317522920668125 619599:0.9658787250518799 423435:-0.0009000392747111619 287647:-3.0146636962890625 904090:-4.883293151855469 268376:0.8100829720497131 93660:-2.767340660095215 645080:1.1014988422393799 633508:-0.005426022689789534 358691:-4.565420150756836 604946:-2.4820871353149414 430514:-0.20840106904506683 898614:-0.017127560451626778 962252:-0.04317281395196915 337951:1.4773811101913452 201883:-0.03877020999789238 633793:-1.5044909715652466 1036358:-0.20280484855175018 467427:-6.697716798953479e-06 693842:-0.6468619108200073 596120:3.7356064319610596 321871:-2.3692097663879395 376259:-0.1162206158041954 514122:-0.00027025729650631547 178206:-4.29463543696329e-05 318858:-0.002192581305280328 899913:-0.007774543482810259 372914:0.3372972905635834 959136:1.0891896486282349 179826:-0.0021704942919313908 234123:-1.19321870803833 813286:-3.5519497394561768 340086:-0.003605559701099992 150240:2.7253730297088623 828151:-3.800271511077881 449168:-2.630423069000244 553240:-3.188481092453003 499206:4.477156639099121 777256:-0.024707041680812836 880534:-0.07881394028663635 333808:-0.20931687951087952 445557:-0.062260951846838 380518:-0.17531484365463257 138589:-0.003323475131765008 264764:-0.055447906255722046 586556:-0.06357882171869278 276457:-3.312981605529785 463653:-4.62150764465332 645293:-1.4429360628128052 672175:-3.2845215797424316 46210:-0.2508317530155182 1018202:-0.011790101416409016 984841:-0.03535202518105507 475323:4.9164886474609375 804003:-0.3801780343055725 519089:-0.010952257551252842 627181:1.9892710447311401 463128:1.411637306213379 337422:2.3054051399230957 979458:-0.3442067503929138 515712:-6.984004974365234 973822:-0.000710653024725616 977730:-3.203017695341259e-05 327123:2.3236098289489746 42497:-1.679479956626892 563816:-2.095945358276367 268328:-3.72514441551175e-05 148526:-0.008797659538686275 922446:-2.133270025253296 778004:-3.8293068408966064 5784:-3.9397459030151367 863506:-0.006100233178585768 591517:-0.00045754192979075015 163555:3.135331869125366 851016:-1.5446714162826538 86112:-0.17742089927196503 618528:-0.003593269968405366 186023:-0.0035580454859882593 361908:-0.13983485102653503 692852:-1.3135814666748047 231522:-2.3044815063476562 545932:-0.23519346117973328 184162:-0.009027881547808647 610410:-0.02320343255996704 441573:3.293436288833618 768807:-0.006950204726308584 183166:-0.0015509799122810364 309561:-0.008695517666637897 934786:-0.027135640382766724 833482:-0.2741893529891968 236327:-0.003207895904779434 108856:-6.931659698486328 50931:6.140589714050293 563072:-0.6983262300491333 12082:-0.000990896369330585 331635:-5.6607584953308105 820050:-0.4372427463531494 656523:-1.1816900968551636 1044149:-1.0908610820770264 809773:-3.0819873809814453 157264:-0.1485559642314911 13999:7.5556559562683105 126917:3.14150071144104 597998:6.735396862030029 188264:-0.00773733202368021 271255:-0.003106403863057494 542266:2.0035643577575684 697020:-0.5479828119277954 437784:-0.00022991804871708155 162239:-0.11781596392393112 942526:-0.026621025055646896 742887:-0.18759286403656006 521084:-4.652835845947266 1035496:-1.0889004468917847 254961:-0.3763171434402466 477189:0.4076422154903412 761964:-3.7201333045959473 409584:-0.03634537756443024 4974:-0.12012577056884766 841038:1.4707939624786377 273625:-3.9770145416259766 1012641:-0.004110152367502451 335953:-3.4429614543914795 144397:-0.00011612702655838802 149561:-0.059698790311813354 183033:-0.6882627606391907 462938:-0.0033848511520773172 391663:-0.3537296950817108 190696:-1.664947271347046 395341:-0.013210452161729336 786964:-0.0026221114676445723 367665:-1.8124502897262573 245412:1.378013253211975 223587:-0.9504601359367371 398520:-1.9399858713150024 990304:-2.4093430042266846 785462:-0.7256032824516296 654278:-0.010284122079610825 722852:-0.1305830180644989 476125:-0.4648801386356354 599768:-0.028924157842993736 456766:-1.5132344961166382 524954:-0.7229013442993164 119583:-3.797675132751465 995761:-0.24995438754558563 1042210:-0.01856144331395626 475590:-2.5473062123637646e-05 132869:-0.006414779461920261 249611:1.1766856908798218 17619:0.6740084886550903 95403:-2.5262253284454346 318652:-0.0014886532444506884 677262:-0.0008052160847000778 419106:-0.24129734933376312 521704:-3.745185136795044 330183:-1.6043071746826172 956842:2.642775774002075 497873:-3.4366791248321533 1036619:-0.08397473394870758 706655:-0.9690324664115906 273989:-0.0035283470060676336 924907:-0.0005189107614569366 257675:-0.03870728611946106 790374:-3.6922719478607178 541561:-0.6234930753707886 206732:4.229461669921875 244814:-0.4786629378795624 878793:2.489076852798462 804807:-3.288648843765259 293646:-0.5044172406196594 636703:4.645219802856445 806828:-0.024151798337697983 992226:1.5932087898254395 365232:-0.41003912687301636 738916:3.034201145172119 407185:-0.0028762260917574167 984077:-0.0014182397862896323 362693:-0.021469485014677048 850331:1.2945506572723389 568073:-3.055021047592163 526040:-0.006455026566982269 782193:-0.06799738109111786 758058:-0.008210480213165283 280980:-0.36258429288864136 858843:-0.003870332147926092 142813:-4.23362398147583 353459:-0.09618021547794342 348375:-0.0003953056293539703 307420:-0.9586520791053772 800240:-0.24493791162967682 1047151:-2.2884021291247336e-06 483057:-0.06124894320964813 919814:-0.0015984117053449154 288552:0.7545209527015686 598654:4.208958625793457 915422:-3.4169979095458984 693313:-0.0005351585568860173 965885:-1.2492090463638306 198682:-0.46152567863464355 186618:-0.5781188011169434 296890:-2.0032036304473877 282919:-0.04574904963374138 358089:-0.06543676555156708 569136:1.3881675004959106 171723:0.9389171004295349 598017:7.614250183105469 259169:6.1187005043029785 257455:-0.051606129854917526 82295:-4.637382507324219 565420:3.300854206085205 397425:-3.7191519737243652 19455:-3.166167736053467 10080:0.9431564211845398 129167:-0.01811959035694599 668136:0.9995802044868469 617211:4.743837356567383 98793:-0.03065473400056362 98421:-2.332261562347412 491634:-0.6605109572410583 424430:-3.821803569793701 478308:-2.5689001083374023 368234:-0.003328819526359439 70560:-0.0010997903300449252 833601:-0.2887202799320221 382930:5.021555423736572 822776:-0.15616069734096527 690023:-2.804410457611084 442169:-0.0011799208587035537 198732:-1.5676250457763672 243410:-0.06491780281066895 399671:3.950568675994873 869180:-0.06154037266969681 456972:0.2184310108423233 717477:-3.7741858959198 591269:-0.000802051683422178 576723:-0.0008446146966889501 614162:-0.43677818775177 238634:2.00960636138916 107342:-0.41643521189689636 24400:-8.67347526550293 380041:-0.35511863231658936 25640:0.8357896208763123 114100:-0.0016481089405715466 794971:-2.1696770191192627 893928:1.6281213760375977 880383:2.5963127613067627 253421:-2.2444541454315186 662587:-0.06275316327810287 425204:1.3736538887023926 767917:-0.009813893586397171 484045:0.8902035355567932 292058:-0.0006561094778589904 187470:-0.0024660115595906973 673026:-0.11730525642633438 239660:-0.0014397635823115706 165660:-0.003985301591455936 664331:-3.2057361602783203 478820:-1.4717154044774361e-05 710406:-0.006862441543489695 458958:-0.0010482881916686893 514411:1.3680474758148193 678526:-0.00361796747893095 293475:-3.155144214630127 658642:2.5636329650878906 399878:-0.0008346932008862495 939582:0.6119475364685059 859063:1.8135530948638916 939068:-0.0044678510166704655 629694:-0.015825672075152397 464607:-0.1429588347673416 262271:-0.7208114862442017 531560:-2.506133794784546 311591:-2.636962652206421 943670:-0.015914520248770714 483245:-4.121762275695801 821352:-3.335796594619751 782413:-0.006409614812582731 489378:-0.0035166197922080755 869537:-3.125789953628555e-05 1002241:-0.039361048489809036 503272:-2.2264699935913086 263274:-0.015937242656946182 711615:-0.21024201810359955 18088:-0.0014377785846590996 75542:4.324630260467529 509866:1.2823762893676758 388432:5.818491458892822 828207:-0.7237341403961182 100426:-0.07782351970672607 731190:-0.06086989864706993 904803:-0.07966125011444092 886036:-3.616307020187378 706104:-0.017677024006843567 518452:-3.251999855041504 732396:-0.0012359779793769121 515592:-5.742593765258789 343901:-0.23273341357707977 943791:-0.0615205392241478 392603:-4.451334476470947 61671:3.077059268951416 1029250:-0.6098930239677429 502050:8.855966567993164 853749:-0.4999532997608185 630250:-1.5018278360366821 749181:-0.7751759886741638 273183:2.8833556175231934 499503:0.7782667875289917 206284:2.6205174922943115 668642:-1.9339505434036255 931439:-1.668373465538025 847408:1.0563610792160034 335511:-4.835413932800293 807834:-0.02933323010802269 668930:1.575769305229187 595400:-2.4536845684051514 378205:-2.5841193199157715 203037:-0.8221661448478699 366598:-4.321277141571045 117928:10.669608116149902 296566:-0.7262303829193115 226810:-0.0008938448154367507 44975:-0.010223857127130032 988690:-2.0442285537719727 599072:-1.361709713935852 523441:-0.8708069920539856 471870:2.01637864112854 558748:-3.8251209259033203 285834:-0.021435651928186417 979337:-0.006885109934955835 576711:-0.0014601156581193209 1020905:3.132425546646118 156797:-0.0003365958691574633 614640:-5.7162604331970215 982603:-3.8550946712493896 19560:-0.3653649091720581 27489:-0.00441391346976161 849727:-1.187961459159851 602413:-2.017336368560791 879112:-2.584052085876465 763918:-3.787480115890503 20068:-0.0011503343703225255 1005532:-0.041151486337184906 106959:-3.0489630699157715 461001:1.703960657119751 121651:-3.4788219928741455 1024995:-2.900547742843628 87634:-0.03948487341403961 72546:1.042080044746399 847949:1.147854208946228 365923:-4.589203834533691 871302:2.1741628646850586 100960:-2.6499156951904297 498462:-1.1760601997375488 1006589:-6.195132255554199 106522:-0.16232824325561523 974365:1.492828607559204 472843:-4.164554119110107 1011103:-1.1209383010864258 487886:4.0076584815979 928753:-0.06464133411645889 64051:-0.0006450607907027006 1019946:-2.6911680698394775 3328:-0.062050022184848785 602818:-0.3798813223838806 717322:-0.034027349203825 25138:-2.803506851196289 679354:-0.9891505837440491 627711:-0.001270478474907577 10035:-2.0663704872131348 887929:0.8478248715400696 194581:-1.47919499874115 722131:-0.068385548889637 858584:-0.6998291611671448 287922:-0.011785240843892097 733187:-0.004968813620507717 313921:3.073532819747925 56577:-5.1714768409729 176173:-1.4387357234954834 194900:1.630476713180542 615403:3.081087827682495 986515:-0.19433565437793732 35606:-2.800652265548706 105629:2.461562395095825 674460:-0.1856389343738556 446188:-0.004677240736782551 466246:-0.051735714077949524 112386:-0.0004336154379416257 692440:-0.0028199609369039536 781473:2.1653034687042236 496367:-0.004503390286117792 152137:-0.001360880327410996 237391:-0.14377854764461517 70937:-0.040636125952005386 120496:-0.00980197824537754 414690:-0.004256764426827431 847829:-0.9053429961204529 102449:1.6914416551589966 721334:6.264616966247559 261840:-3.314887762069702 61001:-0.0006214388413354754 234339:-0.0017035617493093014 602757:-4.803659915924072 757507:-5.530953407287598 219640:-1.5852513313293457 679784:-0.5907926559448242 890843:2.094722032546997 617783:-3.924588680267334 668648:-1.2833913564682007 542671:-6.366264343261719 754106:-0.052781447768211365 457254:-1.7738721370697021 228349:-0.48043933510780334 761839:-0.30610865354537964 189710:-1.4196830987930298 466126:1.2682876586914062 302773:-1.6287517547607422 562738:-0.008540605194866657 516408:-0.019847633317112923 374411:-0.0005460392567329109 819780:2.65535569190979 877167:-0.03765048086643219 1048440:-3.0350494384765625 553873:-0.021587295457720757 590665:-0.001086701638996601 717538:-1.4647518396377563 417915:-0.0005508006433956325 657562:-3.57989665644709e-05 312675:-0.9026594758033752 183348:-3.9443140029907227 311570:-0.0049557751044631 204118:-0.352860689163208 705973:-5.1338419914245605 479766:-0.0011466976720839739 399956:-0.41319242119789124 607303:-0.14649361371994019 806810:-3.261040687561035 680457:-5.913156032562256 849487:6.980285167694092 562831:-2.270430326461792 475330:-0.010461365804076195 696321:-0.38733282685279846 242409:-1.8601231575012207 862898:-0.1987740695476532 787021:-0.0034978182520717382 255301:-0.02883598953485489 873565:-0.0977904349565506 495516:-0.12554360926151276 998960:-0.0048628258518874645 938943:-4.4091644287109375 1014802:-3.922248601913452 309950:-2.0087392330169678 661587:-0.07131151854991913 637406:-0.00220854370854795 548656:-0.0007399031892418861 296146:1.752500295639038 843748:-0.018017394468188286 302012:-3.7316741943359375 500125:1.6323388814926147 966195:-2.4095048904418945 816719:-0.05649278685450554 976702:-0.002138062845915556 544119:-0.31918367743492126 791028:4.560723304748535 754636:3.2681148052215576 460067:0.7572649121284485 443204:-2.3370766639709473 73002:-0.007231709081679583 224527:-0.0059996494092047215 427875:-0.0007438071770593524 270314:-0.00359780783765018 56325:3.3440253734588623 859860:-0.07536645233631134 859670:-0.013387818820774555 353043:-1.7603638172149658 363968:-0.00199680682271719 901556:-0.001601495430804789 54621:0.6976400017738342 467803:-3.824267864227295 762282:-4.034711837768555 273102:-0.13756944239139557 163524:1.1496983766555786 72932:-0.03182028979063034 562451:-0.10527566820383072 609098:-0.004138213116675615 679341:-0.006881841458380222 228174:-1.4738447666168213 391857:-1.867754340171814 433404:5.027834892272949 682227:-4.379093170166016 558510:-0.040340036153793335 132668:-0.008096951059997082 329955:2.0356619358062744 581281:-1.5154831409454346 737636:-0.0006799315451644361 917116:1.596236228942871 67797:-1.3863004446029663 634954:3.222548723220825 569019:-1.496812105178833 193632:-0.2030983716249466 8226:-1.5590925216674805 466031:-1.0945250988006592 519519:-0.021949434652924538 959224:3.1897332668304443 678239:-3.3204216957092285 837021:-0.8652384877204895 461109:-0.3185209333896637 398546:-0.08245216310024261 230974:-0.10079736262559891 208968:0.8154450058937073 749892:0.3427736759185791 551512:-0.00010570314043434337 522335:-0.006681952625513077 223085:-0.0030655439477413893 976643:-0.4527604281902313 682514:-1.2135517597198486 325461:-2.4799697399139404 956935:-0.5044172406196594 934414:-0.04337121918797493 40073:-0.2675604522228241 608793:-3.6743030548095703 936704:-2.7144615650177 685203:-0.5908105969429016 582963:-0.014976341277360916 26222:-0.006320594809949398 950165:-0.03632161021232605 875929:-0.011353662237524986 576173:-3.034708261489868 968360:-0.7697311639785767 703986:-3.4086432456970215 906824:2.300140142440796 905130:-2.6678082942962646 896773:-2.412766456604004 459584:-0.12204380333423615 1039834:-0.005337120965123177 97250:-0.029811112210154533 136365:-0.018550453707575798 760836:-0.07809764891862869 866889:-0.0008103090221993625 644663:-0.005839538294821978 594309:-0.005497521720826626 507834:3.138357400894165 749050:-3.4193248748779297 1032028:-6.9341654777526855 765360:-4.827066421508789 904326:2.411379337310791 672647:0.14448758959770203 489307:-2.4183106422424316 943709:-0.7601034045219421 673762:-4.132874011993408 88425:-8.150726318359375 822938:-0.2435142695903778 507155:-0.00472437497228384 552263:-0.0013341607991605997 -1:-5.358976364135742 506176:-4.3185715675354 84507:-4.61585807800293 506522:4.201238632202148

Calibration Predictions:

0 0 1.7225935152964666e-05 1.7225935152964666e-05 6.977992597967386e-05 6.977992597967386e-05 0.00015441630966961384 0.00015441630966961384 0.00024679172202013433 0.00024679172202013433 0.0004843620117753744 0.0004843620117753744 0.0005029181484133005 0.0005029181484133005 0.0015060240402817726 0.0015060240402817726 0.0019488794496282935 0.0019488794496282935 0.002908832859247923 0.002908832859247923 0.0029140410479158163 0.0029140410479158163 0.0032581454142928123 0.0032581454142928123 0.003816793905571103 0.003816793905571103 0.0057092406786978245 0.0057092406786978245 0.007735377177596092 0.007735377177596092 0.00947948731482029 0.00947948731482029 0.013888888992369175 0.013888888992369175 0.016949152573943138 0.016949152573943138 0.017571885138750076 0.017571885138750076 0.017985612154006958 0.017985612154006958 0.018987340852618217 0.018987340852618217

Calibration Boundaries:

9.154067040711985e-18 0.12575063109397888 0.1257523000240326 0.20837895572185516 0.20838172733783722 0.319793701171875 0.3198087513446808 0.33767929673194885 0.33769869804382324 0.4457332491874695 0.44573450088500977 0.49919742345809937 0.4992121756076813 0.6428586840629578 0.6428791284561157 0.6556175351142883 0.6556223630905151 0.7362712621688843 0.7362892627716064 0.7830820083618164 0.7831047773361206 0.8438205718994141 0.84382563829422 0.851881742477417 0.8518823385238647 0.8553000092506409 0.8553085327148438 0.8747504949569702 0.8747777342796326 0.9207541346549988 0.9207577109336853 0.9989897608757019 0.9989904761314392 0.9990796446800232 0.999079704284668 0.9992200136184692 0.9992218613624573 0.9995685815811157 0.9995701313018799 0.9998603463172913 0.9998642206192017 0.9999995827674866

Flatbuffer encoded Model (the flatbuffer data (binary format) needs to be converted into a base64 encoded string):

{
  "data":"DAAAAAAABgAIAAQABgAAAAQAAAABAAAAFAAAAAAADgAYABQAEAAMAAQACAAOAAAAwAAAABAAAABkAQAA6A4AAGwcAAAqAAAA3NwoI8bEAD42xQA+S2FVPgViVT4AvKM++b2jPk3krD7Y5qw+JjfkPlA35D7Olv8+vZj/PmOSJD+6kyQ/jdYnP97WJz9GfDw/dH08PxB4SD+OeUg/oARYP/UEWD/sFFo/9hRaP/H0Wj+A9Vo/pu9fP2/xXz+Ltms/x7ZrP8u9fz/XvX8/r8N/P7DDfz/izH8/Ac1/P7rjfz/U438/2fZ/Pxr3fz/5/38/KgAAAAAAAAAAAAAAaoCQN2qAkDfQVpI40FaSONDqITnQ6iE502OBOdNjgTn48f05+PH9OUTWAzpE1gM6yGXFOshlxTqLcf86i3H/Oh6iPjseoj47f/k+O3/5PjuchlU7nIZVOy0jejstI3o7lRS7O5UUuzsMef07DHn9O9pPGzzaTxs8OY5jPDmOYzzz2Io889iKPOryjzzq8o88kFaTPJBWkzxXi5s8V4ubPGEDAAA36wvA7FMewO5ANcDSiyw/3juwvh2xb8DUvPM9Z6IPQT15h74vVD8+31xwPwm4sLpYhOG+mO2TQKoWbsC9YrA/fTQAwG8gY8A2xTO+l83hv2Q3Gr3Lt8W/PT5OQKSllEArRqS/DZHiv3AjqL/TrTW+T/C9uiGyuL9Gupy+EXImvRdnVb65DYy9u4SAvSXTk76mpLm+C2yjvntLNb6DzR69/lUCvfCn80Ciyy2+A8S8vaMf+7yQS968Pa+vvAv0f76q2tq9EwXuvcwR4r9q4w2+r0lXvnTKILwi+sS9izBFQPyYjr+Kl7+/epggvDmlcLyYOyW9H/eXvCsOmLwGQc+5ggKlvr5R0L5bYzu9whF8vRqHOL9wZZTA78fxQDfQ575mGYy/2SYWuHOSGbZFK0G8LRRvQBlxM7wF+U++VHyLuxoNRb9X0Ya+8yEcv28+HLgkWAa4Yq4JuUwhNLjOXIi9qCHZt/yu1bey6Xa33kSPQMIqhL1DSTa/UJMav1tRc7+aiaC/6iMDQH9kIMDFmte9sRRnv2iHfsAFpjG9vXGFvV0/F7+xfVXAVW+FwJolBMCtrSHA8qa8vzA+F7/wp8i/gUyuv5Ffsrroeo+/mjVSwLyFM76KAKK6Kih+vTriQb9UX3W8fLy9QHrwpcBku5i/R1jlvxXmn79APkLAaPWgvy+PAMAioRfAzsB0wK0yGsDKgVTAMkFTQCsJHcAlU2PAlgZuwLsOo8CPilTA73tfwPiTB8GSi/k/J4xfQJL9sMA2YiXA+cqEwLa7msBsxfvA/N2QwC9ObMDL4pHAQ4xxwCAuscBcHIHAVMO3wJZeAcH4fN/AkxLvv29IpMDfdyNAM/nWv7CLcMCgfBPA1zaHPx4qOEBZDklAXiB/P/xCZEDt1fI/ulobQJASJEAG5KBAm+u2wMwkfMDGQxXAr+TdwGXMw0AK9p3AvTGKwL18pcAI5JTA2H9PwN+5dsCVt5nANe1evySJlr8tuLW/Pp0fv8m1ckC9PoBAlWnDvKKAgMCkb3zAqlWbvxNmcsDAV4dA5EBjwCHJesD9MTC/Ul8ewMOdD8AXdULAfH4OwI3yW8DwWc2/2rJrv3t5Ur+7ThHAHLWTwJ4plcCZg2fAHatHwIFAhMBUd5rAGE3svm6YdMCrscG/RYBdv5WwoEA5apm/Pg7jv6qJpL+9PfC9SN7fvvk4fb+rBu+/+/Mgv4xwhkC8se+/n0GXvxdhi7+aSAy/I8E5v64x3L0FbYC+qQQ/v/821b5WoYu/LGo7v+hVm7886jm/xLHHQLpZmr13hUPAk6I5wG78hr1gaQLBhj7GwCjQ3cDC2pLAcLjLwJMO2bdW8QXAtn/EQOoaA7im8NG+jtZwvgEoM79FdGq/d3suvlr7wb+kRjm/70bYvw4fGL7f0Ki9DTkhvTmJ87jzvOC2gxJ4vxwRu75sAha+M1V4vjlqdb+rUW6+PkzwvPL6q71YQqq8GzH9uwn/CrutiBO7ClqgutToU71SsY25L4amugAvk7+ncFi81yaQurYgVr0ouOy78GGfvTvkubooFnG5mcH+uw+GaLtzWM+83NwCu78H0rtZ9nq8kzYXu+XtC7yOxgrB+2RnvSJpXbqwhNO7Uv8hu03Ps7xui0u+5N4UvdKOgrz5Yhe+NkKLvS2Rk7vRxpa6u8YQvNKZr7ypzlm74t7zua/9iL4u4o69k6eaukCB079cBdi6jGKMvq7pE7xYHWO9RL5FvxlPjLxsZsq8+MUUvZ3wa7rKTJa6YdRdu20ID7s4o8BAN5mTvD0oUrvSBT678eIlu7x+KLziFkG8tgOGvelQarqJRz278azduFet17up8Z+9rWKEvZ01gr0VMbpAveqVvlXe4Lt0Oey8gZeivPWN074c0rW+Jfz1vvb8e72OitC5LOB6vVdYn7sB3wy+kDtSu463Bb7TeKW5d+THu4lvlLx2jii9nlIIvLppYr1Qz1q6YoYlu9luzr1Xn1u7pztluyUN60C71yu73jAPvjRpob06j36/SPTau1pGyL22BDq8Hks6unKJ/bszyWu7KldWvsdAUrqZWb+7gsUyv24yDryY/xO/7f9GvndeI7t90SK85TvAv4jHa7q3Unm9nW/tupob5LvhVuO5/hzPu8fMsbvOW3m+JmYrvP/DKMAPJBC8Ihg+vnO2vLoKVBpA1aodvEwZKbqBjgC+HZQ3vG7RortTKFq7da6Gu9c7Z7twJaO9ypggvIu+47sa8vm9zfWyvIAE9r1f/ue/QErLuthzvLq+9lq7wuYlu8VSM7oTqQS8aM+QvH2c4bukrMC+ikOZu31Qxr6ZcPq862pUuiJ5sLnXpIG8w/4rulcFf70jFVO6GS5pu2mxD7sO+Oi72ZmHu6DzhL2DBwi6EoHhu39J8b0E466719ewvMbgr7yMn067ua9+u+oew7rGC5K9TptKurvFJr4rX4K8z5whu5eikLsMwOy7lVhbvO7OmrthUk685Xu7uw4NK7uldw68sk2cu/n37Lv3gSe8EBA5v45QvLzHJaq8qG+Ous6AkrpAhQa8S+dIu/ahM0F65l68EjPSu1YU2rxZ+Ru8F8rgu7iGdL06ycm7N70QuysSArw0wfW9At+uuo4ktLsBfh/AtZjEu7fiLLz9fLu/DWeSu/YtkrrOf8K+EPZBuumG0LsOZKK79+CButo+DrvA2hS7z5RLu4ooZro94u+5aw0SvIiLHr3NYxK+GBgKvFphv7pOdy28GM84u0FLbLtcMVi9t2aJum89MrqCiji7Q2ALvSKXgrsXd2a7xHm7vA78QrqvOhO+Jj5jvP4c1b++Hgy7VIJ0vcyB0brD87262HT7vFV2GrzzYFO9RenRuhF9a7v+2cW8FcF2u2829Ly1F5u6EX88u67cqL1mpX27UhdZu5fpy7rkuiG9kNM/u0nNEL0TJA+6bBttu1vy7LxjSsW7l+gfvg7oIrrP0KS5DErfuiAVvrybYxC6pxZ3vkHh0r1rI4RAf4cIwAzvlb29OsbA9B44wITaHsBGupy+XTO2vO9xRr+GJEjAutApv8UE7r761TC9KZPAv/gjBsAokdu/ZKHfvsFXAb/05tu/hOnKv4/EZ78oqrS+jvQTPkbo4D+DBFZAhBjuv19jikB+PEc/o52WP1WUCkCPeac/SShBPymB2D9mOgBA7yS1wKnHUkDN+6ZA4z0zwMSyMUDhYoU/AhW/P4NsLkCOXg4/4BeNwH3kfz/A027A6CSkP8KLE0DkB1TAsTxuv7NycT8B3wU//SkmQJ1Wi8A/jsA/WjBCQES3nkA1aj5AG7tJQDuyrD5dE3XAKOhrwA0LWT8BhLNAWHsKQElIAkBVcY7AzYNNwH9e30CqjBRAtsQ0P+0PBkBhz5O/iNMqP5ioHD9YkMe/yCpNwMgna8DXIei/bJ09wHLxkUAzwyLAqGwzwCS/G0Bi7yVA8wBHPj2KHUBqPwTAz7LJP08T9b4fD5i/IdBswGHkYz/92wrAZOOTwIIi6D/7z+0+b6D+P6k4QsC+d8hAlL9UvxovA8CRaos/dnszwKTUAsD4PjLAQPBAwOS0UMBZ8SlANidawBv20EBxN27ACrINQVRrAEDlG3vAV/HGP4TNl0C7sto/HQAlQHhRzD/jLdQ//O96P8qvhkDLKFFAp3qeQB7WfEDZ2khAX4jXQB8Ge8Dj7JI/Sj3HvxL7JUA+Pz1Ax5CFQHjLTkCWK3HA+VkPQMET+r8JTR9A1EN3P0hm0D9ZDAFA7HRTQH6iSsDcaCTAT/ZVP/f6bsDmiDhA4KxcwNMavT+mN3PAQPJEwEBXoj/uUeA/HGElwOLtScCP3LQ/cKCZP3vw0D+peUhAfzUTQBiwWsBb34XA1laJwEepSEA9IylAxCBQwGSdAEB7WVzA8EOcwC+cfMDIznTAyce2P4iwtD9IP0XAB08PwHmvsT/Ttx7AeuWDwOBTnUAcDXPAdix7wDUqLcDaWCjAvbktwFtKRb82IkPAayHvP0ON1b+bDFrAdVH4v30cG8Ad3EE/OaepQCnmnD9dn0PAeV3YPwa2FEAXZAlAv341wPV4TsBUNRrAk3FnwKqSFcDwqtI/fCULQI4Ouj9/NS7AF7pXwMGgHEDWs6U/sYv3v0Tuyz/wetC/fii4v4ruREDBE6zAB0SFwJlhTz/7rAxAdrPQPy4crz85eVLA5+IcQCOlD8BhrF8+msUawNl5h8BhdIQ/OJgpwBcznMAcHDHAONZawImYMj/kUEtAwYAbQBMQTMC3tipBPxcpvybYuj9iG9o/6v2MP0wkoT+o7Ww/iEI5QCOWQr+TOL3AEw0gwCoxOz872J3A80hoQAHBUD854m4+w7REQKFhG77oeb4/JcqEvx8nVMC+mCW/JFWtwAWlXsDEahrAX70qwBk8LMAKHAHAYDkmvtCesrctFaO+WBhAvm87NUBMPVu+Fy8ZwOJJDLoB0Xq+Dxy1vjZgR8B7ttA+Q1a9v+TTrz9LcrG/USmTP3/3r8CXJExA4fn/vobuiD/nR4rAuyf7v38QIUAvFQrAfSEBv+qOxMD5hNQ/+kK8P5pT4b8IgK8+7BeSwLKmwr59IQG/UdWIP4+2J0CIIYzAf5R9wBOsT768fKvAYQMAAJ3MCABztg8A4YICANNEAAAC8g4A6PUHAD7vCACfOwsAL+ADADkbCgDLngIALVwNADKgCgCaUAkAbKALAKS+AwC6hwQAoyYLAD4oAgDUCAQAb2INAEj8DABKsAkAH7cJAOgzCgDNNgQAdJIKAGBQAQAd+g4ArdgJAO+fCwAZFQEAspEGANMECwA7HAoAQbgMAJRJBAB3TQgAEhkNAJsUAwDkHAEAASAJAE8gAADfQwwA6YEBAIJDDgC36gkAsTEPAACnCADDvQUAUf8OALBzCAC/2woArbcLALNkBQDrYwkAKrMFALuuCACw1gEAMagEAK6FCACtFAIAIucPANdQBQBe3A4Ai5gIACdRBAA8Qw0AfwAEAHdBAQCvNgAAA+cOAG8cBwCaCAoAb/oPAFqJDwCYGAkAsesHAGD0AgDiUwYAqMYOAJjxCgCCtA8AKBgEAELrDgClfQIAHrgCAFRKDwCDKQ4AxkEHAGROBwAGngcA/1oDANDkBABKoA8AY2kDAI+4CgC8HAwAaBwIABOVCABjxQQA2SwEAA5CDgDgiwkAk3QKAGiIDABG+wsAPAgJAKt0AQBOewMAaF8KAEwIAwAgJAkASVICAJ9tDwCvQQoAZs4FAOwsCwAADQAAhiMIADPlCAAPFwoAPzcJAIuSAwA1+Q8A/bwOAHj/DwBNRAwAvroEAE/pBABbIwcAYBwPAF9ZCgCsoAgAyBUJAOZoDABxEAYA6fYKADDuCwAItQIABjIGAIgRAQDK1A0AA48LAF3FBQAfvgsAlx4FAAZiAgA8uQIAZg8MAAZECACl8goAmAEHAKqhCwAI3gcAQGAAAIDeBwCx+gUAtcUKAOH6CgABpgAA0s4MAGKIAwAw7gwAoqUIAMXvAQAjSgkAn54AAH/ZBQDj6w8A0gwKAPycBgDwYAkAmBYAAHWAAQBcvw8AYfQDAJCrDABAuQcAAd0AAHzzBwBl1gwAS/4OAIUyCQCx/AcAHpsHAA7lAgB5QwgAq+EJAM5xBwDBowYAuyMNADTMAgASagoADqgLAIwnAwBSDQwACHoLAPnKAgDbOwsAv4wNAJ5PBADorQcA0ZgHAMcJBQANOwsAHRkDAI+WCAAbSgEAdxkJABrOBwAC7gsA4kcKALCtCwAaCAMA7nkGAD74BgCdxQwA0tcFAEZUCwAm+gYAQSwFAAJFCgBSgwwAul0KAFWtAABEmw4AmroHAAw6AACLBAoA6MwPALyiCgA2/AsAfpwMAIK0AADs7QcATqMBALXuDwAJ5gAADsAFAHaGBAAx9g4A1B4NAAmrCADjow8AUCsBAGlZAQD9Ww8AOKkBAGOVBQDPRwgAxtsFAOp0AwDzxgAAoUQNALCSBQCMVAgA2BkNAFXyAwBAkw8Aod4IAC+jDAAogw8AUGYCAK5jAAABSw8ADTQCAOMhBwBfyAoAaEwAAEdECQDskgYA3LAEAF0/BQCaUwwAS9EPAPTCCgCL7Q8A8DYJACJHAwC5nQkARh0HAErYBwD/kwkAIPUMAE0IBgCgEwEAccUPACTsAgBKiAEADQQPABiuBgBJuw0A274JALM6CgDAjQUATfALANZZBADCagQAMpYIAFBfAABPdgwA08wIANgGCADA0gUAX+0HALIqDQDwPwYAagQEACCYDABx7wsA75IHAGROAAA1XAsAilwEAF0dAgAB0AIAiZwAAKJhDwA5vwYA+ogAALS9AQDKtwwAYs8CADwKBABMEAQANrYNACjcCwCVfw4AC3YGABZSBwBaEAcAOYcPAOfODADk3wwAM1YLAFxXBgC8tAwAxvsJALJkBADJdgUA+nUDAEhgBQBYaggAtl0HAAScCwDxKw4APPMIAFDtBQDpjQsABtcKAEXlAwA44QcAVBoGAInMBQD9ewMAr2YOACkxAADxXgcAMD4PAM4qBAAnmwMApAcLALrHAgASLQ0Aj/gBANxXDwDnzwQAP5kHAAYaBgD6TwcAPoYDAGBACQBNAgwAuPsLABQCDAC0hQUAlm8NAMDpBABf+AcAXVQNAJldDQD+2w4AaN8CAOofBADwFwUApQUJADfWCQCAlwgAlFcNAPrYAgCTDQ8A5G0KAAw8AwDqnQkAHUAKADYoCwBn8w0AtYcGAAK3AQBuZgAApKoJAJqODADCQAcAJ8EEAC5EAgCcSgoALKgDAIbMDQDnLAAAM/oAAJyPBwDcLgcAAzALAGqeBQChcw8ARS4EAGPODQDEHQMAJ7sLAEADBwCJ+wAAbhMAADGcBQB+ywIAqEYAAKkuDADaJw8A16MKADwGAgA4xgoAifEOAPHjAwDszgYAAaAKAO9bDgBJOg0AfWQCAL6bCQDadAQAdcwGAI5VCgCn1gIAit0EAPjVBQBKSwkA0rYDAOscDgCtXQoAv3kCANrdDwCRcwgAxYgFAMRuCACi6gMAvNwEAFMYCgB5ww8AP8kKADZmDgBO3AIAYWsAALAWAwAWHg0AE70HAE7PCQAOBw8Am50DADm5BAD4cAsAKh0BAK+vAACaAggA5/oDAOkoCgBJAwkAqdQAACqRCwBtZwMAdvUPAADRCwAFBwIAvmEOAOHjAACJ2QYAOUgCAAjaDADeuQkA83cDAIhOAgBHbQgAhREJACRoBwAPbQMA1S0JAOLyCgA8VA4AC48NAMIyCQAwXwgAkmgCABLBBAAyLwAAcr4CAHvYAgCXIwQAtmoCAJ0GCQD8PggAi+4DAN8WBwBcSQcAx8wIACC+BADYkAoAdjAFALqBCwDOAAcAZEELAD+0DgAK8goAHIcCAKJ3BwAtyAUAY4cGAE+fAwAFXQsA6OgCAD7nDgAGagsABgkOAPoFAgBl8AsAKJQKAK/tAwC0wQ0AIHAJAKxPDABPbQkA4nsBAJB0AACRNgYA0hQGANsaDQA/YA4AiEcGAFJWAQAZOAEACQcPAIu2BQB+WgoA2CYJACx6AAD4jQwASe4AAK3DCgBjkwMAalAJAHtgBgAiZQYA7HsMAJTjCABOEw4ArWwEAHjvAAA0sQkAEjsJAMdSCwAFmQgAfW4LAH04BgD75QQA3UMHAMyuDgDBqwkAaJoIABd/CwASXwkAlC0OADVVDQD4WQMA1e8MAFYdAwCHQwoAxmwNAAXcAADpsgMAFicBAC+fBwALzwMAoewLAPHWAAAoZwQAMZABADpGCABzDwUA5bwGAAnsBQAWiwAAX+YMAGIbAQAd3g4A4EoCACU5CQC/Uw4A6DEKALybBACqxwcADiYFAOk3BACkHQAAYCcAANamCAD/bg0AiocCAIXtBABkRgsAE5IPAFfLAACXtgYAsrAFABTfCwD6twoAeYwNANbgBwBB1QsA4wgFAJv9BQAVxg4AT/YMAJOGCgD6PAsA25cNAEZGAgBuXQIAPlYOACIgAAALIwoAGUoJAPAwBgAzBg4A9BEMAHt9BgAyYgAAY2UAAMOHDAAg7wQAnZwBADMnAAACNQoATrwDAD/3DAAjmQEAzWIHAFshDAAlEwcAtxsNAKGsBQDtkQkArcoIALYBCwDfVQsAtKgFAKCiDgBnhwoAEhYPALvHDACfYwQAmk8MAESCDADyvQoA6vQBAFPoBAAiqQcARqwEAEyvAgCqoQoA+2oJACl2CgBVzQ8AfP4NAODEDgAHOgEAfiIJAMyDCwAjKQoANxkGALq/BwDuHwkAEnwPAE3wDADO3ggALFsNAAnpDgBkUgwARZ0HAPM2CACAGAMABl4AAMloDQBPdAkA6KMNAD4zBwAs/QwA/0sAAGRMBwAoZAAA4MsPAB8rBADn3gYAHygFAPeiDAB+ag0AzhwHANKEBAAIag0AY3oEAMl4CgDAEgMAnaEHAOmTDwBI1g0A3vcNANK1AQDzeQsA434CAKqZDgA06QcAKqQDAFEgBQCayw0AgnUJAJyGCAA0BQ8AGBEHAC1bDABmFQEAMK8IAFX3BACtXwcAu0AHAB/TAQA3bQkAzG0LAJDaBgAASw4AWuwMAM+hAQCiOg4AbzYOAHzLBAC4FAYApc8IACMFBwBNbgsAjnYPAKx3AACWaQcA0/0EABZ7BQBTXwwAS1cPADO+DgAUhQ0ARMMGAC3vCwCGSw0AUukJACFdBgB/1A8AHJYMAJv5DADiMwoA4iMPALWeBAAtsAIA5/AAAIu9CAALNwcAWBgEAMpCCgBU+QIAa9kHAMdHDADyKAcA7d0DAAz5BgBbdwcA3S0CANRHCwBgigEAmZ0FANxtAQD6bQsAXdUAAF2OAgDb0g0AGHEIAKjMAQBygAcA2ncPAMkIBwDY1wkAfl4BAEqMAgAX+AcAXWYOAAliCgBjJQYA4HEAAPnVDwBsCQMASDADALgKAgBBygQAXL0NAL/zAwDkIQ0A0P4DAFKWCgBThAIAM9sBAAWvDQCqzw0AKpAPAC0xCQAaoAEAwd0JADUJBwDnVQsAjPcDAKT3DAC+QgoAQZQKAPA1DADv+QUAC5sKAAVIBwAV+AIA9HwGANUIAQDEfgIA5r0EAPiiDgD1Bg0A80kHAAaYBQCydwUAvJUMAI7qBAAHmg4AnOMIACwlCQBO1QwAE2MFAERxCwAjeQUAo0QMAA57BAAKzQwAzCUDAPNoCgDPlwMARtAPAP////8DAAAAY3BhAA=="
}

A/B Testing

A/B testing is useful for comparing model performance. Users can assign multiple models within a given strategy to ensure all targeting and external settings are consistent between the test groups. Bid opportunities will be divided up randomly in the proportions in the proportions selected by the user on the basis of MM UUID or CID.

The following image shows the A/B Testing is implemented in the byoa-price-engine:

ABTest

Workflow

BYOA receives a bid request containing a UUID/CID from Bidder. The hash function generates a hash (range: [0, 99]) for this UUID/CID and routes the bid request to the corresponding model_id. (UUID/CID hash, CampaignID(SelectedEntity), StrategyID(SelectedEntity), campaign_settings) -> Executor. We currently support both campaign and strategy level splits.

Campaign Level Split

The following example (for hypothetical campaign_id=123456) illustrates the Campaign Level Split in which 80% of UUIDs will be evaluated using model1 and the remaining 20% will use model2. In this case, there is no strategy_id present, so T1 will simply check whether the campaign_id in the Bid Request matches the campaign_id in the CampaignSetting and then calculate the UUID hash. In the next step, we find the executor and model_id given the UUID/CCID hash.

Example

curl -X PUT "https://api.byoa.mediamath.com/campaign_settings/123456" \
 -d '{
    "settings": [
      {
        "executor_id": 2,
        "high": 79,
        "low": 0,
        "model_id": "model1",
        "namespace": "mm"
      },
      {
        "executor_id": 2,
        "high": 99,
        "low": 80,
        "model_id": "model2",
        "namespace": "mm"
      }
    ],
    "uuid_type": "UUID"
  }'
  • If the UUID hash falls in the range low=0, high=79 we will forward the bid request to Executor 2 (= MediaMaths TF LogBrain) and use model_id=model1.
  • If the UUID hash falls in the range low=80, high=99 we will forward the bid request to Executor 2 (= MediaMaths TF LogBrain) and use model_id=model2.

If a campaign has only one setting, the UUID hash will have a default range of low=0 to high=99. If a campaign has a strategy defined (using the following endpoint: #endpoint:AhmqszpgFYiLDn3wq) we will do a strategy level split as explained in the next section.

To configure the campaign level split use the following endpoint: #endpoint:DsEv95wxD3YJbR8cg.

Strategy Level Split

For the following Campaign (campaign_id=1234567) there is a Strategy defined (strategy_id=11111) in which 50% of UUIDs should be evaluated using model3 and the remaining 50% will be evaluated using model4. That means the split is on the strategy level and the low and high range within the Strategy will be used and the Campaign low and high range will be ignored.

Example

curl -X PUT "https://api.byoa.mediamath.com/campaign_settings/1234567/strategies/11111" \
  -d '{
    "settings": [
      {
        "executor_id": 2,
        "high": 49,
        "low": 0,
        "model_id": "model3",
        "namespace": "mm"
      },
      {
        "executor_id": 2,
        "high": 99,
        "low": 50,
        "model_id": "model4",
        "namespace": "mm"
      }
    ],
    "uuid_type": "CID"
  }'
  • If the CID hash falls in the range low=0, high=49 we will forward the bid request to Executor 2 (= MediaMaths TF LogBrain) and use model_id=model3.
  • If the CID hash falls in the range low=50, high=99 we will forward the bid request to Executor 2 (= MediaMaths TF LogBrain) and use model_id=model4.

To configure the strategy level split use the following endpoint: #endpoint:AhmqszpgFYiLDn3wq.

Available Features

We accommodate the features listed below. Please refer to the T1 Knowledge Base for an explanation of what these features represent.

Feature Name Type Corresponding field(s) in mm_impressions Notes
dma_id Simple categorical dma_id
region_id Simple categorical region_id
category_id Simple categorical category_id
os_id Simple categorical os_id We recommend using the ‘os’ and ‘os_id’ fields instead of this one due to better granularity and accuracy.
exchange_id Simple categorical exchange_id
isp_id Simple categorical isp_id
fold_position Simple categorical fold_position
browser_id Simple categorical browser_id We recommend using the ‘browser’ and ‘browser version’ features instead of this one due to better granularity and accuracy.
conn_speed_id Simple categorical conn_speed_id
hashed_app_id Simple categorical app_id -> calculate hashed_app_id See App ID notes.
interstitial Simple categorical interstitial
device_id Simple categorical device_id Refers to ‘Device’ on T1 Knowledge Base page and represents form factor broken out by OS. We recommend using ‘device_manufacturer’, ‘device_model’ and ‘device_type’ features instead of this one due to better granularity and accuracy.
user_frequency Simple categorical user_frequency Refers to ‘Session Frequency’ on T1 Knowledge Base page
day_of_week Simple categorical timestamp_gmt 0 = Sunday … 6 = Saturday
day_part Simple categorical timestamp_gmt 0 = 12AM to 5:59AM; 1 = 6AM to 11:59AM; 2 = 12PM to 5:59PM; 3 = 6PM to 11:59PM
week_part Simple categorical timestamp_gmt 1 = weekday; 0 = weekend
deal_id Simple categorical deal_id
creative_id Simple categorical creative_id
size Simple categorical size
browser Simple categorical contextual_data See ‘Wurfl Feature’ below
browser_version Simple categorical contextual_data See ‘Wurfl Feature’ below
os Simple categorical contextual_data See ‘Device Information’ below
os_version Simple categorical contextual_data See ‘Wurfl Feature’ below
device_manufacturer Simple categorical contextual_data See ‘Wurfl Feature’ below
device_model Simple categorical contextual_data See ‘Wurfl Feature’ below
device_type Simple categorical contextual_data See ‘Wurfl Feature’ below
channel_type Simple categorical channel_type
browser_language_id Simple categorical browswer_language_id BrowserLanguageID: If browserLanguage is set to 0 in mm_impression, browserLanguage id was not send to BYOA.
country_id Simple categorical country_id
pixel Simple categorical overlapped_brain_pixel_selections See ‘Audience Data’ section below.
cookieless Simple categorical cross_device_flag cookieless can be derived from the cross_device_flag field in mm_impressions using the following logic: If cross_device_flag = (2 or 3) then cross_device=TRUE else FALSE
cross_device Simple categorical cross_device_flag cross_device can be derived from the cross_device_flag field in mm_impressions using the following logic: If cross_device_flag = (1 or 3) then cross_device=TRUE else FALSE
exchange_id_cs_vcr Hardcoded interaction Combination of exchange_id and prebid_video_completion We round prebid_video_completion to one decimal and keep a 0 before the decimal if the value is less than 1. If there no record for this section, it will record -1. For example, 1.34553 becomes 1.3 and .3549 becomes 0.4. See ‘Hardcoded Interactions’ below for more information.
exchange_id_cs_ctr Hardcoded interaction Combination of exchange_id and prebid_historical_ctr We round prebid_historical_ctr using the same rounding convention as prebid_video_completion above.If there no record for this section, it will record -1. See ‘Hardcoded Interactions’ below for more information
exchange_id_cs_vrate Hardcoded interaction Combination of exchange_id and prebid_viewability We round prebid_viewability down to the nearest multiple of 10. For example, 120, 121, and 129 all become 120. If there no record for this section, it will record -1. See ‘Hardcoded Interactions’ below for more information.
id_vintage Numerical id_vintage If incoming request has id_vintage >= 999, the calculation of the response rate will use id_vintage = 0. mm_impression will still log id_vintage >= 999.
bidder_pixel_frequency Mapped numerical overlapped_brain_pixel_selections See ‘Audience Data’ section below.
bidder_pixel_recency Mapped numerical overlapped_brain_pixel_selections See ‘Audience Data’ section below.
exchange_id_cs_category_id Hardcoded interaction Combination of exchange_id and category_id
exchange_id_cs_site_id Simple Catergorical Combination of exchange_id and site_id
site_id Simple Catergorical site_id

Audience Data

There are three audience-based features in our model that are derived from the overlapped_brain_pixel_selections field in mm_impressions are separated by | char. format is: pixel, bidder_pixel_frequency, and bidder_pixel_recency

For context, overlapped_brain_pixel_selections is a pipe-delimited list of tuples that contain segment membership information. Each tuple is of the format mm:px1:r1:f1; the components of the tuple are separated by colons and can be interpreted as follows:

“mm” - the namespace of the pixel

“px1” - the pixel_id of the audience segment. In the logistic model, this is converted into a binary field indicating whether or not the impressed user is in this segment.

“r1” - the recency, or amount of time that has elapsed, since the user was added to px1. In the logistic model, this is converted into a mapped numerical field whose value is equal to 1440.0/r1. By way of background, 1440.0/recency is simply converting the recency value, which is denominated in minutes, to its inverse, measured in days—there are 1,440 minutes in a day.

Input: recency_minutes

we calculate recencyDays := math.Max(recencyMinutes/1440.0, 1.0)

and limit recencyDays as follows:
recencyDaysFn = 1.0 / math.Min(200, recencyDays) // math.Min(200, recencyDays) will limit recencyDaysFn in the range 0.005...1

If the recency is zero (perhaps because recency data is not available for that audience segment), the corresponding map-entry for recency would not exist. I.e. we do not allow division by zero.

“f1” - the frequency, or amount of time that has elapsed, since the user was added to px1. In the logistic model, this is converted into a mapped numerical field whose value is simply equal to f1.

If frequency > 200 frequency will be set to 200.

Device Data

Model features related to a device, including browser, browser_version, os, os_version, device_manufacturer, device_model, device_type are derived from the contextual_data field in mm_impressions. For example

{
  "24": {
    "1": {
      "targeted": [],
      "untargeted": [
        "br_Chrome:ve_60.0.3112"
      ]
    }
  },
  "25": {
    "1": {
      "targeted": [],
      "untargeted": [
        "os_Windows:ve_10.0.0"
      ]
    }
  },
  "26": {
    "1": {
      "targeted": [],
      "untargeted": [
        "fo_Desktop"
      ]
    }
  },
  "27": {
    "1": {
      "targeted": [],
      "untargeted": [
        "ma_Desktop Make:mo_Desktop Model"
      ]
    }
  },
  "28": {
    "1": {},
    "2": {},
    "3": {}
  }
}

Would be read as

browser = "br_Chrome"
browser_version = "br_Chrome:ve_60.0.3112"
os = "os_Windows"
os_version = "os_Windows:ve_10.0.0"
device_model = "ma_Desktop Make:mo_Desktop Model"
device_manufacturer = "ma_Desktop Make"
device_type = “fo_Desktop"

We include browser name in browser version (i.e. we prepend “br” in “br_Chrome:vs60.0.3112”) because two different browsers could have the same version. The same logic holds for os_version and device_model.

Hardcoded Interactions

Formed by appending exchange id with other half of feature value, e.g. ExchangeID = 4 and site_id = 100. We will lookup exchange_id_cs_site_id^4-100 in features -> weights.

Numeric values less than 1 should have a “0” prepended to them, so that the feature-value exchange_id_cs_vrate = .543 would be "exchange_id_cs_vrate^0.5.

We round prebid_viewability down to thenearest multiple of 10. For example, 120, 121, and 129 all become 120.

func exchange_id_cs_vrate() {
	exchangeID := "10"
	vr := 19 // prebid_viewability
	
int64(math.Floor(float64(vr)/10))*10
	
sha}
	
	fmt.Println("exchange_id_cs_vrate^" + exchangeID + "-" +  strconv.FormatInt(finalVal, 10))
}

AppID

The raw Bid request to BYOA Price Engine has AppID: “0” butthe current impression_data only logsapp_id: “N/A”. The Bidder currently sends hashed_app_id to BYOA Price Engine, but impression_data only stores app_id. The hashed_app_id needs to be calculated manually as follow.

Use Boost Library 1.58

uint32_t m_HashedAppId = 0;

void setHashedAppId(const char* appid)
{
    if (appid) {
        m_HashedAppId = atoi(appid);
        if (m_HashedAppId == 0) {
            m_HashedAppId = MM::Utils::pstr_ihash()(appid) % INT_MAX;
        }
    }
}

struct pstr_ihash
    : std::unary_function<const char*, std::size_t>
{
    std::size_t operator()(const char* x) const
    {
        std::size_t seed = 0;

        while (*x) {
            boost::hash_combine(seed, ::toupper(*x++));
        }
        return seed;
    }
};

Please use the following code to test the calcHashedAppId. It takes app_id from mm_impressions and calcs the hashed_app_id to be used in the model and these examples will make sure the implementation is correct

#include <iostream>
#include <string>
#include <boost/functional/hash.hpp>
#include <climits>
#include <cassert>
#include <cstring>

struct pstr_ihash
    : std::unary_function<const char*, std::size_t>
{
    std::size_t operator()(const char* x) const
    {
        std::size_t seed = 0;

        while (*x) {
            boost::hash_combine(seed, ::toupper(*x++));
        }
        return seed;
    }
};

// pass appId from mm_impressions
// TODO: add handling for special case: 
// If App ID is absent, mm_impressions logs N/A for app_id
// if app_id = N/A -> hashed_app_id = 0
unsigned int calcHashedAppId(const char* appid)
{
    unsigned int m_HashedAppId = 0;
	
    if (appid) {
        if (std::strcmp(appid, "N/A") == 0) {
            return 0;
        }
        
        m_HashedAppId = atoi(appid);
        if (m_HashedAppId == 0) {
            m_HashedAppId = pstr_ihash()(appid) % INT_MAX;
        }
    }
    
    return m_HashedAppId;
}

struct testcase {
    const char* input;
    unsigned int expected_output;
};
  
int main() {
	// your code goes here
	
    std::vector<testcase> tests {
      {"com.fivemobile.thescore", 1453566594},
      {"605581486", 605581486},
      {"tunein.player", 1173358324},
      {"com.aws.android", 1903276095},
      {"com.document.pdf.scanner.docscan", 1812585910},
      {"com.apalon.weatherlive.free", 591449217},
      {"com.pandora.android", 1387399900},
      {"com.weather.weather", 447752198},
      {"de.wetteronline.wetterapp", 1107246225},
      {"439873467", 439873467},
      {"N/A", 0},
    };
    
    for (unsigned int i = 0; i < tests.size(); i++) {
	   assert(calcHashedAppId(tests[i].input) == tests[i].expected_output);
	}
    
	return 0;
}

ExecutorID 2

Putting together the model: FlatBuffers

namespace linearmodel;

table GroupModels {
models:[CalibLogModel];
}

table CalibLogModel {
group:string; // this is the goal type
beta:[float];
features:[string];
weights:[float];
preds:[float];
bounds:[float];
}

root_type GroupModels;

Note

1. All goal types in group:string; need to be lowercase: e.g. "cpa", "roi"
2. The intercept field in Features needs to be named: "__const"
3. The Preds and Bounds array need to be empty (size = 0)

Once the above schema is defined, the flatbuffer utilities are used to auto-generate (java or go) code to encode/decode a model into this format. For your purposes, you only need to know how to encode the models into this format. Here is the auto-generated GO code we use to encode our models:

// automatically generated by the FlatBuffers compiler, do not modify

package linearmodel

import (
	flatbuffers "github.com/google/flatbuffers/go"
)

type GroupModels struct {
	_tab flatbuffers.Table
}

func GetRootAsGroupModels(buf []byte, offset flatbuffers.UOffsetT) *GroupModels {
	n := flatbuffers.GetUOffsetT(buf[offset:])
	x := &GroupModels{}
	x.Init(buf, n+offset)
	return x
}

func (rcv *GroupModels) Init(buf []byte, i flatbuffers.UOffsetT) {
	rcv._tab.Bytes = buf
	rcv._tab.Pos = i
}

func (rcv *GroupModels) Table() flatbuffers.Table {
	return rcv._tab
}

func (rcv *GroupModels) Models(obj *CalibLogModel, j int) bool {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(4))
	if o != 0 {
		x := rcv._tab.Vector(o)
		x += flatbuffers.UOffsetT(j) * 4
		x = rcv._tab.Indirect(x)
		obj.Init(rcv._tab.Bytes, x)
		return true
	}
	return false
}

func (rcv *GroupModels) ModelsLength() int {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(4))
	if o != 0 {
		return rcv._tab.VectorLen(o)
	}
	return 0
}

func GroupModelsStart(builder *flatbuffers.Builder) {
	builder.StartObject(1)
}
func GroupModelsAddModels(builder *flatbuffers.Builder, models flatbuffers.UOffsetT) {
	builder.PrependUOffsetTSlot(0, flatbuffers.UOffsetT(models), 0)
}
func GroupModelsStartModelsVector(builder *flatbuffers.Builder, numElems int) flatbuffers.UOffsetT {
	return builder.StartVector(4, numElems, 4)
}
func GroupModelsEnd(builder *flatbuffers.Builder) flatbuffers.UOffsetT {
	return builder.EndObject()
}

Downsampling Correction and Platt calibration We will move TF calibration at the end after downsampling to retain follow the current flow of down sampling correction followed by calibration.

b = beta[0]
platt_alpha = beta[1]
platt_beta = beta [2]

We follow the "p'" above with
ROI: goal : z' = log(p')
all other goals: z' = log(p'/(1-p'))

and finally:
p_calib = sigmoid(platt_alpha + platt_beta*z') for non-roi
p_calib = exp(platt_alpha + platt_beta*z') for roi

Code Example as follow:

// automatically generated by the FlatBuffers compiler, do not modify

package linearmodel

import (
	flatbuffers "github.com/google/flatbuffers/go"
)

type CalibLogModel struct {
	_tab flatbuffers.Table
}

func GetRootAsCalibLogModel(buf []byte, offset flatbuffers.UOffsetT) *CalibLogModel {
	n := flatbuffers.GetUOffsetT(buf[offset:])
	x := &CalibLogModel{}
	x.Init(buf, n+offset)
	return x
}

func (rcv *CalibLogModel) Init(buf []byte, i flatbuffers.UOffsetT) {
	rcv._tab.Bytes = buf
	rcv._tab.Pos = i
}

func (rcv *CalibLogModel) Table() flatbuffers.Table {
	return rcv._tab
}

func (rcv *CalibLogModel) Group() []byte {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(4))
	if o != 0 {
		return rcv._tab.ByteVector(o + rcv._tab.Pos)
	}
	return nil
}

func (rcv *CalibLogModel) Beta(j int) float32 {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(6))
	if o != 0 {
		a := rcv._tab.Vector(o)
		return rcv._tab.GetFloat32(a + flatbuffers.UOffsetT(j*4))
	}
	return 0
}

func (rcv *CalibLogModel) BetaLength() int {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(6))
	if o != 0 {
		return rcv._tab.VectorLen(o)
	}
	return 0
}

func (rcv *CalibLogModel) Features(j int) []byte {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(8))
	if o != 0 {
		a := rcv._tab.Vector(o)
		return rcv._tab.ByteVector(a + flatbuffers.UOffsetT(j*4))
	}
	return nil
}

func (rcv *CalibLogModel) FeaturesLength() int {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(8))
	if o != 0 {
		return rcv._tab.VectorLen(o)
	}
	return 0
}

func (rcv *CalibLogModel) Weights(j int) float32 {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(10))
	if o != 0 {
		a := rcv._tab.Vector(o)
		return rcv._tab.GetFloat32(a + flatbuffers.UOffsetT(j*4))
	}
	return 0
}

func (rcv *CalibLogModel) WeightsLength() int {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(10))
	if o != 0 {
		return rcv._tab.VectorLen(o)
	}
	return 0
}

func (rcv *CalibLogModel) Preds(j int) float32 {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(12))
	if o != 0 {
		a := rcv._tab.Vector(o)
		return rcv._tab.GetFloat32(a + flatbuffers.UOffsetT(j*4))
	}
	return 0
}

func (rcv *CalibLogModel) PredsLength() int {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(12))
	if o != 0 {
		return rcv._tab.VectorLen(o)
	}
	return 0
}

func (rcv *CalibLogModel) Bounds(j int) float32 {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(14))
	if o != 0 {
		a := rcv._tab.Vector(o)
		return rcv._tab.GetFloat32(a + flatbuffers.UOffsetT(j*4))
	}
	return 0
}

func (rcv *CalibLogModel) BoundsLength() int {
	o := flatbuffers.UOffsetT(rcv._tab.Offset(14))
	if o != 0 {
		return rcv._tab.VectorLen(o)
	}
	return 0
}

func CalibLogModelStart(builder *flatbuffers.Builder) {
	builder.StartObject(6)
}
func CalibLogModelAddGroup(builder *flatbuffers.Builder, group flatbuffers.UOffsetT) {
	builder.PrependUOffsetTSlot(0, flatbuffers.UOffsetT(group), 0)
}
func CalibLogModelAddBeta(builder *flatbuffers.Builder, beta flatbuffers.UOffsetT) {
	builder.PrependUOffsetTSlot(1, flatbuffers.UOffsetT(beta), 0)
}
func CalibLogModelStartBetaVector(builder *flatbuffers.Builder, numElems int) flatbuffers.UOffsetT {
	return builder.StartVector(4, numElems, 4)
}
func CalibLogModelAddFeatures(builder *flatbuffers.Builder, features flatbuffers.UOffsetT) {
	builder.PrependUOffsetTSlot(2, flatbuffers.UOffsetT(features), 0)
}
func CalibLogModelStartFeaturesVector(builder *flatbuffers.Builder, numElems int) flatbuffers.UOffsetT {
	return builder.StartVector(4, numElems, 4)
}
func CalibLogModelAddWeights(builder *flatbuffers.Builder, weights flatbuffers.UOffsetT) {
	builder.PrependUOffsetTSlot(3, flatbuffers.UOffsetT(weights), 0)
}
func CalibLogModelStartWeightsVector(builder *flatbuffers.Builder, numElems int) flatbuffers.UOffsetT {
	return builder.StartVector(4, numElems, 4)
}
func CalibLogModelAddPreds(builder *flatbuffers.Builder, preds flatbuffers.UOffsetT) {
	builder.PrependUOffsetTSlot(4, flatbuffers.UOffsetT(preds), 0)
}
func CalibLogModelStartPredsVector(builder *flatbuffers.Builder, numElems int) flatbuffers.UOffsetT {
	return builder.StartVector(4, numElems, 4)
}
func CalibLogModelAddBounds(builder *flatbuffers.Builder, bounds flatbuffers.UOffsetT) {
	builder.PrependUOffsetTSlot(5, flatbuffers.UOffsetT(bounds), 0)
}
func CalibLogModelStartBoundsVector(builder *flatbuffers.Builder, numElems int) flatbuffers.UOffsetT {
	return builder.StartVector(4, numElems, 4)
}
func CalibLogModelEnd(builder *flatbuffers.Builder) flatbuffers.UOffsetT {
	return builder.EndObject()
}

WURFL Parsing

  1. Combine targeted and untargeted fields from the contextual data:

• Combine targeted and untargeted array to one array.

• Sort resulting array by string length and pick the largest string for each dim.

{"24":{"1":{"targeted":["br_Firefox"],"untargeted":["br_Chrome Mobile:ve_67.0.3396"]}}}
targeted_and_untarted = ["br_Chrome Mobile:ve_67.0.3396", "br_Firefox"]
-> for dimension "24" we will pick the largest string as follows "br_Chrome Mobile:ve_67.0.3396"

entire output:
browser="br_ChromeMobile"
browser_version="br_ChromeMobile:ve_67.0.3396"
os=""
os_version=""
device_type=""
device_manufacturer=""
device_model=""
  1. Wurfl features extraction

• All wurfl feature have default value if empty string (""). It will always generate 7 features as below:

2.1 browser: Sets the name of the browser. It looks for idenfifier br_ and reads the browser name until the first “:” char. Will fallback to empty string in case if there are no more chars after br_.

2.2 browser_version: Sets the version of the browser. It looks for the identifier br_ and ve_ and reads the entire string. Will fallback to empty string in case if there are no more chars after ve_.

2.3 os:sets the name of the operating system. It looks for the identifier os_ and reads the os name until the first “:” char. It will fallback to empty string in case if there are no more chars afteros_.

2.4 os_version:sets the version of the operating system.It looks for the identifier os_ and ve_ and reads the entire string. Will fallback to empty string in case if there are no more chars after ve_.

2.5 device_manufacturer:sets the manufacturer.It looks for the identifier ma_ and reads the os name until the first “:” char. Will fallback to empty string in case if there are no more chars after ma_.

2.6 device_model: sets the manufacturer and model. It looks for the identifier ma_ and mo_ and reads the entire string. Will fallback to empty string in case if there are no more chars after mo_.

2.7 device_type: sets the type of the device and may return any of these values: Desktop, App, Tablet, Smartphone, Feature Phone, Smart-TV, Robot, Other non-Mobile, Other Mobile.It look for the identifier fo_ and reads the entire string. Will fallback to empty string in case if there are no more chars after fo_.

  1. Example:
{"24":{"1":{"targeted":[],"untargeted":["br_Chrome Mobile:ve_67.0.3396"]}},"25":{"1":{"targeted":[],"untargeted":["os_Android:ve_8.1.0"]}},"29":{"1":{"targeted":[],"untargeted":["ma_Generic:mo_Android 2.0"]}},"26":{"1":{"targeted":[],"untargeted":["fo_Feature Phone"]}}}
browser="br_ChromeMobile" 
browser_version="br_ChromeMobile:ve_67.0.3396"
os="os_Android"
os_version="os_Android:ve_8.1.0"
device_type="fo_FeaturePhone"
device_manufacturer="ma_Generic"
device_model="ma_Generic:mo_Android2.0"
{"24":{"1":{"targeted":[],"untargeted":["br_"]}},"25":{"1":{"targeted":[],"untargeted":["os_Android:ve_"]}},"29":{"1":{"targeted":[],"untargeted":["ma_:mo_"]}},"26":{"1":{"targeted":[],"untargeted":["fo_Feature Phone"]}}}
browser=""
browser_version=""
os="os_Android"
os_version=""
device_type="fo_FeaturePhone"
device_manufacturer=""
device_model=""

4.Source Code

package main

import (
	"fmt"
	"sort"
	"strings"
	"encoding/json"
)

type Num1 struct {
	Targeted   []string `json:"targeted"`
	Untargeted []string `json:"untargeted"`
}

type wurfulData struct {
	Num24 struct {
		Dim Num1 `json:"1"`
	} `json:"24"`
	Num25 struct {
		Dim Num1 `json:"1"`
	} `json:"25"`
	Num26 struct {
		Dim Num1 `json:"1"`
	} `json:"26"`
	Num29 struct {
		Dim Num1 `json:"1"`
	} `json:"29"`
}

func extractDim(dim *Num1, dimID string, appendTo *[]string) {
	type conf struct {
		lookup        []string
		logbrainNames []string
	}
	var dimLookUpMap = map[string]conf{
		"24": conf{[]string{"br_", "ve_"}, []string{"browser", "browser_version"}},
		"25": conf{[]string{"os_", "ve_"}, []string{"os", "os_version"}},
		"29": conf{[]string{"ma_", "mo_"}, []string{"device_manufacturer", "device_model"}},
		"26": conf{[]string{"fo_"}, []string{"device_type"}},
	}
	dim.Targeted = append(dim.Targeted, dim.Untargeted...)
	sort.Slice(dim.Targeted, func(i, j int) bool {
		return len(dim.Targeted[i]) > len(dim.Targeted[j])
	})
	cf := dimLookUpMap[dimID]
	wurflFeatures := make(map[string]string)
	for _, featureName := range cf.logbrainNames {
		wurflFeatures[featureName] = ""
	}
	// at the moment we only support dim_id = 24,25,26,29
	if len(cf.lookup) <= 0 {
		return
	}
	if len(dim.Targeted) <= 0 {
		for featureName, featureVal := range wurflFeatures {
			*appendTo = append(*appendTo,featureName+"^"+featureVal)
		}
		return
	}
	stringWithValue := strings.Replace(dim.Targeted[0], " ", "", -1)

	// handles device_type
	if len(cf.lookup) == 1 &&
		strings.Contains(dim.Targeted[0], cf.lookup[0]) &&
		!strings.HasSuffix(dim.Targeted[0], "_") {

		wurflFeatures[cf.logbrainNames[0]] = stringWithValue
	}
	// handles the browser, os, device_manufacturer
	if len(cf.lookup) == 2 &&
		strings.Contains(stringWithValue, cf.lookup[0]) {

		splitted := strings.Split(stringWithValue, ":")
		if !strings.HasSuffix(splitted[0], "_") {
			wurflFeatures[cf.logbrainNames[0]] = splitted[0]
		}
		if len(splitted) == 2 && !strings.HasSuffix(stringWithValue, "_") {
			wurflFeatures[cf.logbrainNames[1]] = stringWithValue
		}
	}

	for featureName, featureVal := range wurflFeatures {
		*appendTo = append(*appendTo,featureName+"^"+featureVal)
	}
}

func getWurfulFeatureValues(raw []byte) []string {
	var wd wurfulData
	json.Unmarshal(raw, &wd)
	var res []string
	extractDim(&wd.Num24.Dim, "24", &res)
	extractDim(&wd.Num25.Dim, "25", &res)
	extractDim(&wd.Num26.Dim, "26", &res)
	extractDim(&wd.Num29.Dim, "29", &res)
	return res
}

func testEq(a, b []string) bool {
	if a == nil && b == nil {
		return true
	}

	if a == nil || b == nil {
		return false
	}

	if len(a) != len(b) {
		return false
	}

	for i := range a {
		found := false

		for j := range b {
			if a[i] == b[j] {
				found = true
				break
			}
		}

		if !found {
			return false
		}
	}

	return true
}

func example1() {
	rawWurflData := `{"24":{"1":{"targeted":["br_Firefox"],"untargeted":["br_Chrome Mobile:ve_67.0.3396"]}}}`
	wurflFeatureValues := getWurfulFeatureValues([]byte(rawWurflData))
	fmt.Println(wurflFeatureValues)
}

func example2() {
	rawWurflData := `{}`
	wurflFeatureValues := getWurfulFeatureValues([]byte(rawWurflData))
	fmt.Println(wurflFeatureValues)
}

func example3() {
	rawWurflData := `{"24":{"1":{"targeted":[],"untargeted":["br_Chrome Mobile:ve_67.0.3396"]}},"25":{"1":{"targeted":[],"untargeted":["os_Android:ve_8.1.0"]}},"29":{"1":{"targeted":[],"untargeted":["ma_Generic:mo_Android 2.0"]}},"26":{"1":{"targeted":[],"untargeted":["fo_Feature Phone"]}}}`
	wurflFeatureValues := getWurfulFeatureValues([]byte(rawWurflData))
	fmt.Println(wurflFeatureValues)
}

func example4() {
	rawWurflData := `{"24":{"1":{"targeted":[],"untargeted":["br_"]}},"25":{"1":{"targeted":[],"untargeted":["os_Android:ve_"]}},"29":{"1":{"targeted":[],"untargeted":["ma_:mo_"]}},"26":{"1":{"targeted":[],"untargeted":["fo_Feature Phone"]}}}`
	wurflFeatureValues := getWurfulFeatureValues([]byte(rawWurflData))
	fmt.Println(wurflFeatureValues)
}

func main() {
	example1()
	example2()

	example3()
	example4()
}

ExecutorID 1

Feature Hashing

Once a feature is encoded as a string for hashing (as in the above section), the string is then hashed to an integer. We use Google’s murmurhash3 implementation, specifically the last N bits of the hash value, where N = 20 Here is a Scala code fragment to do this:

import scala.collection.mutable
import com.google.common.hash.Hashing

val bits = 20

val hasher = Hashing.murmur3_32()

def hash(featureValueCode: String) : Int = {
    hasher.hashBytes(featureValueCode.getBytes).asInt & ((1 << bits) - 1)
}

For example, say the categorical feature day_of_week has value 3. We first string-encode it as the stringday_of_week^3, and then hash it using the above hash function as follows:

val hashInt = hash("day_of_week^3")

Model Calibrator

The output p from a logistic regression model represents the predicted probability of a certain future action (e.g. click or conversion), and may not always be perfectly calibrated to the actual, empirical action probabilities in the training data-set. Thus a calibration step is often required to bring these into alignment. We support the specification of a calibrator as a pair of float vectors called boundaries and corresponding predictions, which effectively represent a piece-wise linear calibration function. In other words, to calibrate the output p of the logistic regression model to the “final” probability prediction, we find which pair of values in the boundaries vector bracket the value p , and then do a linear interpolation to get the corresponding calibrated prediction.

Putting together the model: FlatBuffers

Conceptually a logistic model is a collection of feature-hashes and their respective weights (or coefficients), and a pair of arrays for calibration. The feature-hashes are computed as in the above two sections. We, in fact, support the notion of a Model Group, e.g. this could be a group of model-variants related to a given campaign (e.g. for different goal-types).

Now once you have such a Model-Group, you need to put it together in a format that will be usable by MediaMath bidders. We use the binary [flatbuffer](To calibrate a logistic score, find the “boundary” values that the score falls between and get the linear interpolation of the corresponding predictions.) schema from Google, which is a modern version of protocol-buffers. A flatbuffer representation is specified by a schema, and in our case, the schema we expect for Model Groups is this:

// schema for the calibrated linear model
namespace slider.spark.linearmodel;

table GroupModels{
  models:[CalibLogModel];  // vector of Calibrated Models
}

table CalibLogModel {
  group:string; // key identifies which specific model this is
  hashes:[int]; // hash values
  weights:[float]; // weights, or coefficients
  preds:[float];   // predictions, for calibration.
  bounds:[float]; // boundaries, for calibration
}

root_type GroupModels;

Once the above schema is defined, the flatbuffer utilities are used to auto-generate (java or go) code to encode/decode a model into this format. For your purposes, you only need to know how to encode the models into this format. Here is the auto-generated Java code we use to encode our models:

// automatically generated by the FlatBuffers compiler, do not modify

package slider.spark.linearmodel;

import java.nio.*;
import java.lang.*;
import java.util.*;
import com.google.flatbuffers.*;

@SuppressWarnings("unused")
public final class GroupModels extends Table {
  public static GroupModels getRootAsGroupModels(ByteBuffer _bb) { return getRootAsGroupModels(_bb, new GroupModels()); }
  public static GroupModels getRootAsGroupModels(ByteBuffer _bb, GroupModels obj) { _bb.order(ByteOrder.LITTLE_ENDIAN); return (obj.__assign(_bb.getInt(_bb.position()) + _bb.position(), _bb)); }
  public void __init(int _i, ByteBuffer _bb) { bb_pos = _i; bb = _bb; }
  public GroupModels __assign(int _i, ByteBuffer _bb) { __init(_i, _bb); return this; }

  public CalibLogModel models(int j) { return models(new CalibLogModel(), j); }
  public CalibLogModel models(CalibLogModel obj, int j) { int o = __offset(4); return o != 0 ? obj.__assign(__indirect(__vector(o) + j * 4), bb) : null; }
  public int modelsLength() { int o = __offset(4); return o != 0 ? __vector_len(o) : 0; }

  public static int createGroupModels(FlatBufferBuilder builder,
      int modelsOffset) {
    builder.startObject(1);
    GroupModels.addModels(builder, modelsOffset);
    return GroupModels.endGroupModels(builder);
  }

  public static void startGroupModels(FlatBufferBuilder builder) { builder.startObject(1); }
  public static void addModels(FlatBufferBuilder builder, int modelsOffset) { builder.addOffset(0, modelsOffset, 0); }
  public static int createModelsVector(FlatBufferBuilder builder, int[] data) { builder.startVector(4, data.length, 4); for (int i = data.length - 1; i >= 0; i--) builder.addOffset(data[i]); return builder.endVector(); }
  public static void startModelsVector(FlatBufferBuilder builder, int numElems) { builder.startVector(4, numElems, 4); }
  public static int endGroupModels(FlatBufferBuilder builder) {
    int o = builder.endObject();
    return o;
  }
  public static void finishGroupModelsBuffer(FlatBufferBuilder builder, int offset) { builder.finish(offset); }
}


The above java code is imported into Scala, and we use the following code to encode a model using the above class:

// serialize ALL models into GroupModels schema
def toBytes(): Array[Byte] = {
  val groups = // collection of keys (Strings) for the model-group
  val builder = new FlatBufferBuilder(1024)
  var modelOffsets = Array[Int]()
  groups.foreach { g =>
    val mdl = groupModel(g)  // some function to retrieve model for group g
    val calib = groupCalibrator(g) // calib has two members: predictions, boundaries

    val hashWts = mdl.featureWeights().toArray
    val hashes = hashWts.map(_._1)
    val weights = hashWts.map(_._2)
    val preds = calib.predictions
    val bounds = calib.boundaries
    val iGoal = builder.createString(g)
    val iHash = CalibLogModel.createHashesVector(builder, hashes)
    val iWt = CalibLogModel.createWeightsVector(builder, weights.map(_.toFloat))
    val iPred = CalibLogModel.createPredsVector(builder, preds.map(_.toFloat))
    val iBound = CalibLogModel.createBoundsVector(builder, bounds.map(_.toFloat))
    CalibLogModel.startCalibLogModel(builder)
    CalibLogModel.addGroup(builder, iGoal)
    CalibLogModel.addHashes(builder, iHash)
    CalibLogModel.addWeights(builder, iWt)
    CalibLogModel.addPreds(builder, iPred)
    CalibLogModel.addBounds(builder, iBound)
    val iModel = CalibLogModel.endCalibLogModel(builder)
    //builder.finish(iModel)
    modelOffsets :+= iModel
  }
  val iModelVec = GroupModels.createModelsVector(builder, modelOffsets)
  GroupModels.startGroupModels(builder)
  GroupModels.addModels(builder, iModelVec)
  val iGoalModel = GroupModels.endGroupModels(builder)
  builder.finish(iGoalModel)
  builder.sizedByteArray()  // this is finally returned
}

T1 Permissioning Requirements for Endpoints

byoa-api endpoints T1 User Role (min requirement) T1 User Type (min requirement) Other
GET /campaign_settings/{campaign_id} Reporter Agency has access to campaign_id
PUT /campaign_settings/{campaign_id} Manager Agency has access to campaign_id
DELETE /campaign_settings/{campaign_id} Manager Agency has access to campaign_id
GET /campaign_settings/{campaign_id}/strategies/{strategy_id} Reporter Agency has access to campaign_id
PUT /campaign_settings/{campaign_id}/strategies/{strategy_id} Manager Agency has access to campaign_id
DELETE /campaign_settings/{campaign_id}/strategies/{strategy_id} Manager Agency has access to campaign_id
GET /data/{namespace}/models/{model_id} Reporter Agency has access to namespace (orgID)
PUT /data/{namespace}/models/{model_id} Manager Agency has access to namespace (orgID)
DELETE /data/{namespace}/models/{model_id} Manager Agency has access to namespace (orgID)

Terminology

Executor Id For Custom Brain, this value should always be set to “1”.
Namespace Namespace should always be set to the organization ID in which the BYOA campaign or strategy resides. To upload a new model to an orgnization’s namespace, the user needs to have Edit permission to at least one advertiser within that organization. See: T1 Requirements for Endpoints
Model Data For Custom Brain, Model Data is a generic JSON object that includes the actual coefficients and model parameters that T1 will use
Model Id For Custom Brain, users choose a Model ID when they are uploading Model Data that they will use to configuring BYOA Campaign Settings
Price Engine Internal MediaMath component that distributes Bid Requests to Executors based on Campaign Settings
MM Logbrain Logistic model format that MediaMath uses for in-house optimization, which can also be configured for Custom Brain

AB Test Setup

We will use campaign 12345, under 6789 namespace as example. Client has a model called “BYOA_MODEL” and wants to do 50% AB Split. Note Executor_ID would be 1.

  • compare with tree brain in campaign setting use namespace as “mm” and “mm_tree_brain” as modelID.
curl https://api.byoa.mediamath.com/campaign_settings/12345 -X PUT -d '{settings": [{"executor_id": 1, "namespace": "6789", "model_id": "BYOA_MODEL", "low": 0, "high": 49}, {"executor_id": 1, "namespace": "mm", "model_id": "mm_tree_brain", "low": 50,"high": 99}]}'
  • compare with MediaMath logistic brain in campaign setting use namespace as “mm” and “campaign_{CampaignID}” as modelID. Note if logistic brain is not available for given campaign, it will default back to tree brain.
curl https://api.byoa.mediamath.com/campaign_settings/12345 -X PUT -d '{settings": [{"executor_id": 1, "namespace": "6789", "model_id": "BYOA_MODEL", "low": 0, "high": 49}, {"executor_id": 1, "namespace": "mm", "model_id": "campaign_12345", "low": 50,"high": 99}]}'