Experiments

Background

Customers can target different line items to sets of non-overlapping users, allowing comparison of results to test performance. Also known as “A/B Testing” or “Randomized Control Studies”, the Experiments feature creates isolated subsets of users, and ensures these users are only exposed to one Line Item setup.

Use Cases

  1. Compare Bidding Strategies: "I want to test two or more different bidding methods to see which gives me the best results." Ideally this is done with identical line items for targeting but with audience splits. 
  2. Compare Multiple Creatives: "I want to test two or more sets of creatives / landing pages to see which generates more conversions." This is done by creating one line item per set of creatives and assigning a different test group to each.
  3. Persistent Control Group: "I want to set aside a control group of creatives (PSAs) to measure incremental lift." This is done by using the X Percent Hold-out Test Plans. Assign the PSA line item to the Control Group (small portion of users), and all other line items should be assigned to the Test Group (large portion of users).

How to use the Experiments feature:

Experiments can be enabled at the Campaign level, with Test Plans being assigned at Campaign-level and Test Groups at the Line Item level:

1. A new toggle on Campaign: Campaigns have a new “Experiment” toggle

    1. When the toggle is set to “Yes”, a new “Experiment” tab will appear on the object that will allow a Test Plan to be associated to the campaign. When the toggle is set to “No”, the experiment tab will not appear. 
    2. The toggle is set to “No” by default.
    3. If the campaign is toggled to “Yes” but no Test Plan is selected in the Experiment tab before closing the window, the toggle will reset to “No”. Thus, it is recommended to set up the Experiment after both the Campaign and Line Items have been created.

2. Test Plans and Test Groups: Test Plans and Test Groups are pre-configured.

    1. There are 14 Test Plans available. Each test plan has a different split of users arranged into Test Groups, allowing users to pick the plan that fits their use case the best.
    2. Test Plans are associated at the campaign level and Test Groups are associated at the line item level.
    3. You will only be able to assign test groups to line items whose campaigns have a test plan assigned.


Test Plan Test Groups User Index Range*
Five Percent Hold-out Plan (ID: 1) Control Group (5%) (ID: 1)
Test Group (95%) (ID: 2)
1-50, 51-1000
Ten Percent Hold-out Plan (ID: 2) Control Group (10%) (ID: 3)
Test Group (90%) (ID: 4)
1-100, 101-1000
Twenty Percent Hold-out Plan (ID: 3) Control Group (20%) (ID: 5)
Test Group (80%) (ID: 6)
1-200, 201-1000
Two-Way Split Plan (ID: 4) Group A (50%) (ID: 7)
Group B (50%) (ID: 8)
1-500, 501-1000
Three-Way Split Plan (ID: 5) Group A (33.3%) (ID: 9)
Group B (33.3%) (ID: 10)
Group C (33.4%) (ID: 11)
1-333, 334-666, 667-1000
Ten-Way Split Plan (ID: 6) Group A (10%) (ID: 12)
Group B (10%) (ID: 13)
Group C (10%) (ID: 14)
Group D (10%) (ID: 15)
Group E (10%) (ID: 16)
Group F (10%) (ID: 17)
Group G (10%) (ID: 18)
Group H (10%) (ID: 19)
Group I (10%) (ID: 20)
Group J (10%) (ID: 21)
1-100, 101-200, ..., 801-900, 901-1000
Three-Way Split with Fifty Percent
Hold-Out Plan (ID: 7)
Control Group (50%) (ID: 22)
Group A (25%) (ID: 23)
Group B (25%) (ID: 24)
1-500, 501-750, 751-1000
Twenty Five Percent Hold-out Plan (ID: 8) Control Group (25%) (ID: 25)
Test Group (75%) (ID: 26)
1-250, 251-1000
One Percent Hold-Out Plan (ID: 9) Control Group (1%) (ID: 27)
Test Group (99%) (ID: 28)
1-10, 11-1000
Three-Way Split with Ten Percent
Hold-Out Plan (ID: 10)
Control Group (10%) (ID: 29)
Group A (45%) (ID: 30)
Group B (45%) (ID: 31)
1-100, 101- 550, 551-1000
Three-Way Split with Eighty Percent
Hold-Out Plan (ID: 11)
Control Group (80%) (ID: 32)
Group A (10%) (ID: 33)
Group B (10%) (ID: 34)
1-800, 801-900, 901-1000
Three-Way Split with Ninety Percent
Hold-Out Plan (ID: 12)
Control Group (90%) (ID: 35)
Group A (5%) (ID: 36)
Group B (5%) (ID: 37)
1-900, 901-950, 951-1000
Four-Way Split Plan (ID: 13) Group A (25%) (ID: 38)
Group B (25%) (ID: 39)
Group C (25%) (ID: 40)
Group D (25%) (ID: 41)
1-250, 251-500, 501-750, 751-1000
Four-Way Split with Ten Percent
Hold-Out Plan (ID: 14)
Control Group (10%) (ID:42)
Group A (30%) (ID: 43)
Group B (30%) (ID: 44)
Group C (30%) (ID: 45)
1-100, 101-400, 401-700, 701-1000

*User Index Range:

Each Test Group is assigned a User Index Range, which indicates the size of a Test Group and represents how the overall Experiment audience has been split. The index value of a user displays which Test Group (or index range) they fall into, and therefore which messaging/strategies they have been exposed to. The User Index value is also made available in the win-logs and allows further analysis of test results.

3. Test Plan and Test Group Association:

    1. From the Campaign object, users can associate Test Plans to a Campaign by selecting the Experiments Tab and selecting a Test Plan from the drop down.
    2. Once a Test Plan is selected, the user can select a Test Group for each of the line items within the campaign.
      1. All line items within the Campaign must have a test group assigned in. If not, they will not be able to save the associations.
      2. Multiple line items can be assigned to the same test group. 
      3. Not all test groups need to be selected. 
      4. Once a Test Plan has been selected, all new line items created will show an Experiments section in their Overview section.

Reporting

When using this feature, our reporting products will provide insights through the following:

Logs: 

You can find the full list of log headers here, however fields relevant to the Experiments feature include:

  • test_group_id
  • test_plan_id

Report Builder and Query Tool: 

Conversion Report and Performance Report include the following fields:

  • Test Group ID
  • Test Group Name
  • Test Plan ID
  • Test Plan Name

API

You can read more on our API support for Experiments feature here, and the new campaign_experiment_assignment service here.

FAQs

Q: How did you come up with the uniformly sized user groups?
A: Based on our research, these are the most prevalent use cases for running experiments. 

Q: Why can't I apply tests to the creatives instead of Line Items? It is a lot of extra work to create multiple Line Items.
A: When we were evaluating the different options, one important criterion is which implementation provides you with the most flexibility and allows the most use cases. Applying test groups on the line item level is the far better option in that aspect.

Q: What happens if the user doesn't have a cookie?
A: This user will not see anything within this campaign. The line items within the campaign all have a test group assigned, which requires the user to fall into a test group. If we encounter an unknown user, we simply won’t bid on the request. 

Q: What happens if the user deletes their cookie and it gets reset?
A: The user may be assigned to a different user group. 

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.

Still need help? Contact Us Contact Us