The MediaMath Log Level Data Service provides access to raw, event-level data generated through the MediaMath platform. Leveraging cloud resources like AWS, clients can build value-added analytics services on top of their data with a minimum of developer time, money, or hassle.

• Architecture
• Accessing Log Level Data Using AWS
  • How Data is Stored in S3
  • Data Security and Authorization
  • Obtaining Temporary Credentials to Access Log Level Data
• Working with the Log Level Data Service
  • File Formats and Schemas
  • Data Update Cycle - GMT (London) - EST (New York) - JST (Tokyo)
  • Using assets from the common/ folder
  • Update logs
  • Special Considerations for Loading Attributed Events
  • Schema changes
  • Best Practices for Loading Data
• Data Schemas - Impressions - Events - Attributed Events - Viewability Events
• Appendix - Day/Hour/Week Part Logic
• Appendix - Batch ID Logic

Data Platform Architecture

As campaigns and strategies created within MediaMath Platform bid and win, logs are written recording the dimensions and metrics of each individual impression. In addition, MediaMath pixel servers log every MathTag event that is generated on an advertiser's site (such as a page load, interaction or purchase) and app. These log files are collected into a central processing system where attribution is run.

At the end of this process, raw log files are made available for access using the Logs Service, which collects the different log types and stores each organization's data into a separate buckets using Amazon S3.

The following data sets are made available as part of Logs Service:

• Impressions -- Impressions served by the MediaMath Platform
• Events -- Event pixel fires from advertiser sites and apps
• Attributed Events -- Clicks, video and conversions based on campaign merit events
• Viewability Events -- Viewability event fires from viewability partners

Each event type is stored in a corresponding S3 bucket, partitioned by client account (organization ID), and secured using the AWS Identity and Access Management framework.

For Attributed Events, there is one additional partition, which is event type (event_type), and possible values for this partition are "click", "conversion", and "video". This added partition groups the attributed events into click events, campaign merit conversion events, and video events.

Accessing Log Level Data Using AWS

How Data is Stored in S3

Data sets in the Log Level Data Service are stored in S3 in the following locations:

In addition, each bucket has a public common directory which includes various utility data sets. See Using assets from the common folder for more details.

Data Security and Authorization

Data security is enforced using AWS Identity and Access Management. To access the Log Level Data Service you will need your own, valid AWS account and an IAM user with the sts:AssumeRole privilege.

To grant access to the Log Level Data Service, MediaMath takes your MediaMath account (e.g. organization) ID and your AWS account ID and creates a new role within the MediaMath AWS account. This role has a unique identifier called a Role ARN. This role grants access to your data, and can only be assumed from IAM users within your AWS account.

If you're working with LiveRamp, MediaMath will authorize LiveRamp's user (User ARN: arn:aws:iam::609251445204:user/svc-liveramp-ingestion-connectors) to access your data.

Below is a diagram showing how MediaMath leverages IAM to ensure the highest level of security for your data.

Obtaining Temporary Credentials to Access Log Level Data

In order to access your data from the MediaMath AWS account, first make sure that you have received a role ARN and an External ID from MediaMath. The role ARN should take the form of arn:aws:iam::794878508631:role/<ROLE_NAME> and the External ID should take the form of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX.

Next, attach a group policy to the IAM user who is trying to assume the role. The policy should use the role ARN provided by MediaMath and take the following form:

  {
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::794878508631:role/<ROLE_NAME>"
    }
  }

Then, you need to call the sts:AssumeRole action from within your own AWS account, using both the role ARN and External ID provided by MediaMath. If the call succeeds (you are allowed to assume roles, and have been authorized to assume the role MediaMath created) you will receive a set of temporary credenials to access your data.

The below Python script can be used as a starting point.

import os
import boto3 # 1.7.4
import sys
AWS_ACCESS_KEY_ID     = '' # Access key ID
AWS_SECRET_ACCESS_KEY = '' # Secret access key
ROLE_NAME             = '' # DPA-{USER}   ex: DPA-Mediamath
EXTERNAL_ID           = '' # MediaMath provided
ORGANIZATION_ID       = '' # MediaMath organization id
S3_BUCKET             = 'mm-prod-platform-impressions'
# ------------------------------------------------ DO NOT ALTER BELOW ------------------------------------------------ #
ROLE_SESSION_NAME     = 'data-platform'
BASE_ROLE_ARN         = 'arn:aws:iam::794878508631:role/'  
ROLE_ARN              = BASE_ROLE_ARN + ROLE_NAME
DURATION_SECONDS      = 3600
client = boto3.client(
    'sts',
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY
)
role = client.assume_role(
    RoleArn=ROLE_ARN,
    RoleSessionName=ROLE_SESSION_NAME,
    DurationSeconds=DURATION_SECONDS,
    ExternalId=EXTERNAL_ID
)
session = boto3.session.Session(
		aws_access_key_id=role['Credentials']['AccessKeyId'],
		aws_secret_access_key=role['Credentials']['SecretAccessKey'],
		aws_session_token=role['Credentials']['SessionToken']
)
S3 = session.resource('s3')
my_bucket = S3.Bucket(S3_BUCKET)
for object_summary in my_bucket.objects.filter(Prefix='data/organization_id=' + ORGANIZATION_ID + '/'):
    print (object_summary.key)

NOTE - If you have requested access to the Log Level Data Service partitioned at the MediaMath advertiser level, the below Python script can be used as a starting point.

WS_ACCESS_KEY_ID     = '' # Access key ID
AWS_SECRET_ACCESS_KEY = '' # Secret access key
ROLE_NAME             = '' # DPA-{USER}   ex: DPA-Mediamath
EXTERNAL_ID           = '' # MediaMath provided
ORGANIZATION_ID       = '' # MediaMath organization id
AGENCY_ID             = '' # MediaMath agency id
ADVERTISER_ID         = '' # MediaMath advertiser id
S3_BUCKET             = 'mm-prod-platform-impressions-v1-1' #'mm-prod-platform-attributed-events-v1-1' #'mm-prod-platform-events-v1-1'

# ------------------------------------------------ DO NOT ALTER BELOW ------------------------------------------------ #

ROLE_SESSION_NAME     = 'data-platform'
BASE_ROLE_ARN         = 'arn:aws:iam::794878508631:role/'  
ROLE_ARN              = BASE_ROLE_ARN + ROLE_NAME
DURATION_SECONDS      = 3600

client = boto3.client(
    'sts',
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY
)

role = client.assume_role(
    RoleArn=ROLE_ARN,
    RoleSessionName=ROLE_SESSION_NAME,
    DurationSeconds=DURATION_SECONDS,
    ExternalId=EXTERNAL_ID
)

session = boto3.session.Session(
		aws_access_key_id=role['Credentials']['AccessKeyId'],
		aws_secret_access_key=role['Credentials']['SecretAccessKey'],
		aws_session_token=role['Credentials']['SessionToken']
)

S3 = session.resource('s3')
my_bucket = S3.Bucket(S3_BUCKET)
for object_summary in my_bucket.objects.filter(Prefix='data/organization_id=' + ORGANIZATION_ID + '/agency_id=' AGENCY_ID + '/'):
    print (object_summary.key)

Temporary security credentials last a maximum of 1 hour. See the best practices section for information about loading many days worth of data in bulk.

When connecting to a Log Level Data Service bucket using AWS CLI, you will need to configure the client to assume the correct role. To do this, we will need to set up two things:

1. A valid Access Key ID and Secret Access Key from the source account in .aws/credentials
2. The role ARN provided by MediaMath and source profile (from step 1) in .aws/config

For instance:

.aws/credentials:

[log_level_source]
aws_access_key_id = myaccesskey
aws_secret_access_key = mysecretkey

.aws/config:

[profile log_level]
role_arn = arn:aws:iam::794878508631:role/DPA-MyCompany
source_profile = log_level_source

With this set up, you can use the CLI with the --profile flag, like such:

$ aws --profile log_level s3 ls s3://mm-prod-platform-impressions/common/

The AWS CLI will take care of assuming the role and caching the temporary credentials for you. When the token provided is expired, the CLI will also re-issue the AssumeRole command and store the new temporary credentials. You can actually see the result of the AssumeRole command if you look in .aws/cli/cache—there should be a file named, in this example, log_level--arn_aws_iam__794878508631_role-DPA-MyCompany.json.

Working with the Log Level Data Service

The following section details information needed to process the raw data available via the Log Level Data Service.

File Formats and Schemas

Log level data sets are stored as tab-separated files. TSV is an easy to parse format, ready for loading into any downstream system such as an RDBMS or Hadoop.

Any string fields containing tab characters are cleansed so tabs are replaced as . to avoid breaking downstream parsers.

Unless noted otherwise, all columns should be considered nullable. A null is written in the TSV files as \N.

See Data Schemas for a detailed description of the schemas, including data types.

Data Update Cycle

Data is updated 3x per day. However, due to ever changing data sets the completion times of the refresh is not held to a strict SLA. The times below are estimates - please allow for delays as data growth can produce unexpected delays.

GMT (London)

EST (New York)

JST (Tokyo)

Using assets from the common/ folder

The common/ directory of each S3 bucket has several utility data sets including

• Create table statements for different RDBMs
• Header row files which list the column titles
• Refresh logs that can be used to tell when Data Platform data sets have been updated

Update logs

Every time a batch of logs are completely uploaded to S3, an empty file is uploaded to the /common/completion-status/ directory of the bucket (this is true for impression logs, event logs, and attributed event logs). The first time a batch of data is processed and uploaded to S3 a 0-byte file of the name

• mm_impressions_batch_[BATCH_ID].done
• mm_events_batch_[BATCH_ID].done
• mm_attributed_events_batch_[BATCH_ID].done

is loaded to /common/completion-status/. The batch ID is an 8-digit code of the type YYYYMMDDNN where YYYYMMDD represents a date and NN is either 00, 01, or 02 (representing the first, second and third batches of data processed every day).

For example, on 7/1, we would normally expect the directory s3://mm-prod-platform-impressions/common/completion-status/ to contain the following three files:

• mm_impressions_batch_2018070100.done - loaded upon completion of the first batch on 7/1
• mm_impressions_batch_2018070101.done - loaded upon completion of the second batch on 7/1
• mm_impressions_batch_2018070102.done - loaded upon completion of the third batch on 7/1

It is important not to download or begin processing any log data before the appearance of this '.done' file: a batch usually will upload to S3 several files, even for a particular organization and impression date. Thus even if some files from batch 2018070102 have appeared in /data/, we can not be certain that more files will not appear presently until we find the 0-byte mm_impressions_batch_2018070102.done file in /common/completion-status/.

Rarely, but on occasion, a particular batch of data may get processed later than expected. For example, suppose an unexepected service interruption prevented batch 2018070102 of impression log data from being processed on 7/1 (batch 2 is usually available in S3 no later than 20:30 GMT), and instead this batch of data was not processed until 7/3. In this case, upon completion of the upload of data from batch 2018070102 the file mm_impressions_batch_2018070102.done would get uploaded into

/common/completion-status/

Rewrite window

From time to time there may be changes to the data after its being written into s3 (for example later arrival data, margin changes, latent conversions etc.). For those cases we have automated a rewrite of the data as follows:

• Impressions - every day the past 5 days are being rewritten (into the original partition date along with the current date included in the batch name)
• Events - every day the past 5 days are being rewritten (into the original partition date along with the current date included in the batch name)
• Attributed events - every day the past 30 days are being rewritten (into the original partition date along with the current date included in the batch name)
• Viewability - There is no lookback days window defined. Rewrites will be done on the original partition date along with the current date included in the batch name

Please implement a reload logic on your side to assure completness of data.

Special backfills

On the rare occasion of a major incident where we are unable to backfill the data during the standard rewrite window, we will write a new batch (and a .done file) outside of the standard 00, 01, 02 batch names and will write it into the partition of the date it was written in s3.

Special Considerations for Loading Attributed Events

Clients have the ability to change the merit pixel, post-click (pc) window, and post-view (pv) window at any time throughout the duration of a campaign. If changed, the changes to the window will only be reflected moving forward. MediaMath systems and services will not recalculate the past data with the new window.

Attributed events data is not lzo-compressed.

Schema changes

From time to time we may add columns in a non-breaking change to the end of our schema. We may send out a courtesy notification when this occurs to let users know the description of the new field and what it can be used for; however, the processing of these files from your side must be set up to handle these additional fields.

Please note: an up-to-date schema can always be found on each bucket under the "common/create-table/" prefix.

Best Practices for Loading Data

• Load into a staging area
• Load N days at a time to account for late arriving data
• Split the load into phases to so your credentials don't expire
• Reload conversion data for past 30 days and for impressions for the past 5 days
• Be defensive with invalid lines

Data Schemas

All columns should be considered nullable unless otherwise noted.

Please note: an up-to-date schema can always be found on each bucket under the "common/create-table/" prefix.

Impressions

Notes on 1P ID columns

This is information that you as a client need for the values you will see in these columns.

Merkury ID - Merkle provides clients with user data in their M1 platform. Merkle can pick up client logs from a client's S3 bucket to ingest data with Merkury IDs. Please loop your client team in with your Merkury client and provide written permission for Merkle reps to pick up this data. Merkle will then be permissioned to your data in our S3 buckets.

LiveRamp RampID - Currently, the RampID (formerly IDL) provided in logs is in the MediaMath format (LiveRamp encrypts RampIDs per partner, so MediaMath's RampIDs are different from a client's RampIDs). In order to receive impression log data, along with their own RampID, clients can work with LiveRamp to share logs in their format/compression of preference. If needed, MediaMath can assist the clients in setting up a new feed to have logs shared with LiveRamp according to their desired specs, note that an SOW with an additional charge will apply.

Notes on total_spend_cpm field

As stated, total_spend_cpm includes the campaign's margin cpm. MediaMath allows for use of the margin management tool that applies changes to margin going back five days in arrears. When this happens, the log level impression data does not update to reflect this change. Therefore when comparing total spend to the Performance report in MediaMath you may see a discrepancy. However, when comparing Media Cost or Impression Count you should find little to no discrepancy as these metrics are not affected by retroactive changes in margin.

Events

Attributed Events

Viewability Events

Appendix - Day/Hour/Week Part Logic

NOTE: The below values should really only be used for Bring Your Own Algorithm integrations.

Day/hour/week part columns are generated from GMT timestamps using the below formulas, all based on week_hour_part (with values of 0 for Sunday 12:00 – 12:15am, through 671 for Saturday 11:45 – 11:59pm).

// 24 hours * 4 15-minute parts in an hour
day_of_week = (int) week_hour_part / 96;

// 7 days * 4 15-minute parts in an hour
day_hour = (int) week_hour_part / 28;

// 7 days * 4 parts * 6 hours in a day part
day_part = (int) week_hour_part / 168;

// 5 days * 4 parts * 24 hours
week_part = (week_hour_part < 480) ? 0 : 1;

// Check if it's after Friday then divide by 7 days
// times 4 hours.  Add 24 hours if over weekend
week_part_hour = (week_hour_part < 480) ? (int) week_hour_part / 28 : (int) (week_hour_part / 28) + 24;

// Check if it's after Friday then mod by 4
week_part_hour_part = (week_hour_part < 480) ? week_hour_part % 4 : (week_hour_part % 4) + 4;

// Straight mod by 4
hour_part = week_hour_part % 4;

// Divide by 4 parts then mod by 24 hours
week_hour = (int) (week_hour_part / 4) % 24;

Appendix - Batch ID Logic

Data platform data files for impressions and events are of the format:

/data/organization_id=[ORGANIZATION-ID]/[LOG-TYPE]_date=[YYYY-MM-DD]/mm_[LOG-TYPE]s_[ORGANIZATION-ID]_[YYYYMMDD]_[BATCH-ID]_[FILE-NUM].txt.lzo

• LOG-TYPE: "impression" for impression log files, "event" for event log files
• ORGANIZATION-ID: the six-digit TerminalOne organization ID
• YYYY-MM-DD/YYYYMMDD: the GMT date of the data contained in the directory/file
• BATCH-ID: the eight-digit identifier of the batch that generated the file, of the format YYYYMMDDNN where YYYYMMDD is the processing date and NN is either 00, 01 or 02, corresponding to the first, second, and third batches of processed date each day. Note that the date in the batch ID is not necessarily the date of the data in the files; for example, batch 2014070100 would be expected to contain a great deal of data both from 2014-06-31 and 2014-07-01, and would write impressions files both as:
  • impression_date=2014-06-31/mm_impressions_123456_20140631_2014070100_0000.txt.lzo
  • impression_date=2014-07-01/mm_impressions_123456_20140701_2014070100_0000.txt.lzo.
• FILE-NUM: a particular batch may write many files for a single date. If the first batch on 2014-07-01 wrote, for organization 123456, two files of 2014-06-31 data and five files of 2014-07-01 data, the files would appear as:
  • impression_date=2014-06-31/mm_impressions_123456_20140631_2014070100_0000.txt.lzo
  • impression_date=2014-06-31/mm_impressions_123456_20140631_2014070100_0001.txt.lzo
  • impression_date=2014-07-01/mm_impressions_123456_20140701_2014070100_0000.txt.lzo
  • impression_date=2014-07-01/mm_impressions_123456_20140701_2014070100_0001.txt.lzo
  • impression_date=2014-07-01/mm_impressions_123456_20140701_2014070100_0002.txt.lzo
  • impression_date=2014-07-01/mm_impressions_123456_20140701_2014070100_0003.txt.lzo
  • impression_date=2014-07-01/mm_impressions_123456_20140701_2014070100_0004.txt.lzo

The file numbers are sequential but have no meaning otherwise; i.e., they do not represent a distribution of records by a particular campaign/creative/pixel, and are not sorted by time.