Experiments

    Background

    Customers can target different line items to sets of non-overlapping users, allowing comparison of results to test performance. Also known as “A/B Testing” or “Randomized Control Studies”, the Experiments feature creates isolated subsets of users, and ensures these users are only exposed to one Line Item setup.

    Use Cases

    • Compare Bidding Strategies: You want to test two or more different bidding methods to see which gives the best result. You can do this with identical line items for targeting but with audience splits. 
    • Compare Multiple Creatives: You want to test two or more sets of creatives or landing pages to see which generates more conversions. You can do this by creating one line item per set of creatives and assigning a different test group to each.
    • Persistent Control Group: You want to set aside a control group of creatives (PSAs) to measure incremental lift. You can do this by using the X Percent Hold-out Test Plans. Assign the PSA line item to the Control Group (small portion of users), and all other line items should be assigned to the Test Group (large portion of users).

    How to use the Experiments feature:

    Experiments can be enabled at the Campaign level, with Test Plans being assigned at Campaign-level and Test Groups at the Line Item level:

    1. A new toggle on Campaign: Campaigns have the Experiment toggle.

      • When the toggle is set to Yes, a new Experiment tab appears on the object that allows a Test Plan to be associated with the campaign. 
      • When the toggle is set to No, the experiment tab does not appear. The toggle is set to No by default.
      • If the toggle is set to Yes but no Test Plan is selected in the Experiment tab before closing the window, the toggle gets reset to No. It is recommended to set up the Experiment after you create both the Campaign and Line Item.

           2 . Test Plans and Test Groups : Test Plans and Test Groups are pre-configured.
        • There are 14 Test Plans available. Each test plan has a different split of users arranged into Test Groups, allowing users to pick the plan that fits their use case the best.
        • Test Plans are associated at the campaign level and Test Groups are associated at the line item level.
        • You can only assign test groups to line items whose campaigns have an assigned test plan.

      Test Plan Test Groups User Index Range*
      Five Percent Hold-out Plan (ID: 1) Control Group (5%) (ID: 1)
      Test Group (95%) (ID: 2)
      1-50, 51-1000
      Ten Percent Hold-out Plan (ID: 2) Control Group (10%) (ID: 3)
      Test Group (90%) (ID: 4)
      1-100, 101-1000
      Twenty Percent Hold-out Plan (ID: 3) Control Group (20%) (ID: 5)
      Test Group (80%) (ID: 6)
      1-200, 201-1000
      Two-Way Split Plan (ID: 4) Group A (50%) (ID: 7)
      Group B (50%) (ID: 8)
      1-500, 501-1000
      Three-Way Split Plan (ID: 5) Group A (33.3%) (ID: 9)
      Group B (33.3%) (ID: 10)
      Group C (33.4%) (ID: 11)
      1-333, 334-666, 667-1000
      Ten-Way Split Plan (ID: 6) Group A (10%) (ID: 12)
      Group B (10%) (ID: 13)
      Group C (10%) (ID: 14)
      Group D (10%) (ID: 15)
      Group E (10%) (ID: 16)
      Group F (10%) (ID: 17)
      Group G (10%) (ID: 18)
      Group H (10%) (ID: 19)
      Group I (10%) (ID: 20)
      Group J (10%) (ID: 21)
      1-100, 101-200, ..., 801-900, 901-1000
      Three-Way Split with Fifty Percent
      Hold-Out Plan (ID: 7)
      Control Group (50%) (ID: 22)
      Group A (25%) (ID: 23)
      Group B (25%) (ID: 24)
      1-500, 501-750, 751-1000
      Twenty Five Percent Hold-out Plan (ID: 8) Control Group (25%) (ID: 25)
      Test Group (75%) (ID: 26)
      1-250, 251-1000
      One Percent Hold-Out Plan (ID: 9) Control Group (1%) (ID: 27)
      Test Group (99%) (ID: 28)
      1-10, 11-1000
      Three-Way Split with Ten Percent
      Hold-Out Plan (ID: 10)
      Control Group (10%) (ID: 29)
      Group A (45%) (ID: 30)
      Group B (45%) (ID: 31)
      1-100, 101- 550, 551-1000
      Three-Way Split with Eighty Percent
      Hold-Out Plan (ID: 11)
      Control Group (80%) (ID: 32)
      Group A (10%) (ID: 33)
      Group B (10%) (ID: 34)
      1-800, 801-900, 901-1000
      Three-Way Split with Ninety Percent
      Hold-Out Plan (ID: 12)
      Control Group (90%) (ID: 35)
      Group A (5%) (ID: 36)
      Group B (5%) (ID: 37)
      1-900, 901-950, 951-1000
      Four-Way Split Plan (ID: 13) Group A (25%) (ID: 38)
      Group B (25%) (ID: 39)
      Group C (25%) (ID: 40)
      Group D (25%) (ID: 41)
      1-250, 251-500, 501-750, 751-1000
      Four-Way Split with Ten Percent
      Hold-Out Plan (ID: 14)
      Control Group (10%) (ID:42)
      Group A (30%) (ID: 43)
      Group B (30%) (ID: 44)
      Group C (30%) (ID: 45)
      1-100, 101-400, 401-700, 701-1000

      *User Index Range: Each Test Group is assigned a User Index Range, which indicates the size of a Test Group and represents how the overall Experiment audience has been split. The index value of a user displays which Test Group (or index range) they fall into, and therefore which messaging/strategies they have been exposed to. The User Index value is also made available in the win-logs and allows further analysis of test results.

      3. Test Plan and Test Group Association:

        1. From the Campaign object, you can associate Test Plans to a Campaign by selecting the Experiments Tab and selecting a Test Plan from the dropdown.
        2. Once you select a Test Plan, select a Test Group for each of the line items within the campaign.
          • All line items within the Campaign must have an assigned test group. If not, then you cannot save the associations.
          • Multiple line items can be assigned to the same test group. 
          • You do not need to select all test groups.
        3. Once a Test Plan has been selected, all newly created line items show an Experiments section in their Overview section

    4. IPs and LiveRamp IDL: Users can run experiments on all devices using IPs and LiveRamp IDL, especially for CTV/OTT devices. When running experiments using LiveRamp IDL, users can split test across all devices (for a person or entire household). 

    In the Campaign > Experiment section, select an ID Type from the dropdown to run your experiment. 

    Experiment ID Type Options

    Experiment ID Type
    ID for assigning users to experiment groups
    Standard (Default)
    • Bito for web 
    • MAID for app 
    • IP for CTV devices 
    IP
    • IP 
    LiveRamp Person
    • LiveRamp Person ID 
    LiveRamp Household
    • LiveRamp Household ID

    Note: All campaigns currently running experiments have ID Type set to Standard.

      Reporting

      When using this feature, our reporting products will provide insights through the following:

      Logs: 

      You can find the full list of log headers here.

      Fields relevant to Experiments feature includes:

      Note: Experiment ID type gets added to the Conversion and Performance reports in Report builder.

      Report Builder and Query Tool: 

      Conversion Report and Performance Report include the following fields:

      • Test Group ID
      • Test Group Name
      • Test Plan ID
      • Test Plan Name

      API

      You can read more on our API support for Experiments feature here, and the new campaign_experiment_assignment service here.

      FAQs

      Q: How did you come up with the uniformly sized user groups?
      A: Based on our research, these are the most prevalent use cases for running experiments. 

      Q: Why cannot I apply tests to the creatives instead of Line Items? It is a lot of extra work to create multiple Line Items.
      A: When we were evaluating the different options, one important criterion is which implementation provides you with the most flexibility and allows the most use cases. Applying test groups on the line item level is the best option so far.

      Q: What happens if the user does not have a cookie?
      A: That user will not see anything within this campaign. The line items within the campaign have assigned test group, which makes the user part of a test group. If we encounter an unknown user, there will be no bidding on the request. 

      Q: What happens if the user deletes the cookies and it gets reset?
      A: The user may get assigned to a different user group. 

      Q: Is IP and LiveRamp IDL available globally?
      A: IP is available globally, however, LiveRamp IDL for Experiments is only available to US users only.

      Q: What should we use for Experiments using IP: IPv6 or IPv4? What happens if IP gets obfuscated?
      A: IPv6 takes precedence when both types of IPs are present. 
      If bid request has obfuscated IPs, then the bid request is not eligible for line items using IP as an experiment type.

      Q: What happens if a bid request does not contain the ID required for the experiment?
      A: If an ID is not available on the bid request, the bid request is not eligible for campaign and line items running experiments.

      Q: Can Experiments using LiveRamp IDL fallback like frequency cap?
      A: No, experiment groups do not fallback. 

      Q: Are Tapad IDs available for Experiments?
      A: Not available currently. 

    Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.