Google Play Experiments Tool: The Definitive Guide

Buse Kanal

22 June 2022

There are different ways for you to optimize your Google Play Store app page and one of them is through the help of the experiments tool. If you want to learn more about the Google Play experiments tool, you need to continue reading this post.

What are Google Play Store Listing Experiments?

Also known as GEx, the Google Play Store listing experiments allow app developers and marketers to test various elements or creatives in their app page. By testing them, the experiments tool helps you identify the best configuration of your app store page.

Why are Google Play Experiments Important?

Google play experiments are important for several reasons. These may include the following:

Helps you know the best app icons, short and long descriptions, and screenshots for your Google app store page.
Better understand how users’ actions and behaviors relate to your app page.
Gather data based on locations, which is helpful if you’re targeting specific countries.
Make data-driven decisions in choosing which creatives should be used.
Saves you time, energy, and money.
Increase conversion rate.
Improve app visibility and user retention.
Drive more revenue.

How do Google Play Experiments fit into ASO?

App Search Optimization (ASO) is primarily focused on improving the visibility of your app in the app search within app stores such as the Google Play Store and Apple App Store.

While the focus of experiments is to identify the best version of your app page, it also plays a vital role in ASO.

For example, you can better improve your chance of getting noticed in the app store when you have the right elements working together on your app page. It is through experiments that you identify the best configuration.

Once you have already created the best app page after doing a series of experiments, doing ASO is easier. You won’t struggle so much in converting people into users.

ASO and Google Play Experiments work hand in hand. ASO is there to help increase your app’s visibility and experiments are there to convince people to use your app. In a way, it is ASO that leads people to your app page and it is experiments that keep people on your app page and eventually convince them to install your app.

It’s worth noting as well that ASO is a broad term that encompasses all activities that you perform to optimize your app’s performance. So, in a way, ASO activities also include the experiments tool for Google Play.

How does the Google Play Experiment work?

Google Play experiments allow you to A/B test certain elements of your app page. You can create different versions of your creatives and expose them to your chosen audience.

At the end of the experiment, you will then quickly identify which of your app page element versions give you the best results.

There are two primary types of experiments on Google Play; the localized and global experiments:

Table

Localized experiment

The localized experiment is done by grouping your audience by their language. This allows you to specifically target a certain group with your experiment.

You can run up to five localized experiments simultaneously. Choose localized if you have a significant number of users in a specific language.

Please note that localized experiments allow you to test with text and graphics.

Here’s how to create a localized experiment at a glance:

Google

Default graphics (global experiment)

The global experiment allows you to run your test and involve all users around the world, wherever your app is available for download.

You can only run one global experiment at a time.

With global experiments, you may only be able to test graphics. Moreover, your default graphics experiment involves all users if you have set your app with only one language.

Here’s an overview of the steps you should take when creating a default graphics experiment:

Google

What can you test with Google Play Experiments?

Google play experiments can test the following:

Screenshots
Icon
Description
Short description
Long description
App preview video

Regrettably, at the time of this writing, you can’t use Google Play Experiments to test your app title.

How to Perform Google Play Experiments

Now that you already have an overview of Google Play Experiments, it’s time for you to dig deeper into this topic.

The best way for you to better understand GEx is to know the steps on how to do it.

So, let’s get going!

Step 1: Set goals

The main goal of Google Play Experiments is to know which of your google app listing version would be the best one to show your users.

Of course, that’s too broad. You need to be specific.

First, you want to identify which one you want to experiment with. Are you planning to test alternative versions of your app icon, description, promo video, or screenshot?

You need to also set the baseline and targets. For example, know the current conversion rate of your app before running the experiment. From there, you can get a general idea of whether the changes you will make will lead to higher conversion rates.

Step 2: Prepare alternative versions of your app listing

Before you run your experiment, you should already have the alternative creative graphics or text that you want to test. Whether it is a different app icon or app preview video, you must have them available.

You can create your app creatives or hire someone to prepare different versions.

Step 3: Create the experiment

All experiments are done in the Google Play Developer Console, under the Store presence section. Once you’re there, select Store listing experiments.

From there, select New Experiment. You can then choose between Global or Localized.

To finally create the experiment, give it a name. Be sure to make the name as descriptive as possible so you can easily identify it.

In 2022, Google made a few upgrades and changes to the Google Play Experiment. It seems that Google doesn’t want to be outplayed by Apple as they also released their own version of experiments almost at the same time.

One of the biggest changes Google made was the “Experiment goals” section. Here’s what it looks like:

From here, you can specify the following:

Target metric
Variants
Experiment audience
Minimum detectable effect
Confidence level

In the Experiment Goals section, you can specify to Google your criteria for a successful test.

For instance, the target metric lets you decide whether to choose Retained first-time installers or First-time installers.

As shown on the screenshot above, you choose the first option if you want to measure how many users installed your app and then kept it for at least a day. The second option lets you include the number of users who installed your app and whether they have kept using it or not.

Another option that you have is to choose your Target Metric. It lets you decide how many variants to use. Remember, the more variants you use, the longer your test will run. It is advisable you only choose one variant.

For the Experiment audience, experts advise you choose 50%. It’s the maximum percentage of your audience who will get exposed to your test. Obviously, we want as many people as possible so that you get a clearer result.

For the Minimum detectable effect, you would need to specify the lowest change in the conversion rate that you want to detect.

Finally, the Confidence level, the higher it is, the more you’ll have confidence in your result.

After setting all the Experiment goals, you are then led to the Completion conditions. It is here where you set the conditions that will decide when your test will be complete.

According to Google, the recommended duration of running the experiment should be at least 7 days. This should allow you to capture user behavior both on weekends and weekdays.

Step 4: Evaluate the result

Once you’re done with setting up or running an experiment, you can go to the Store listing experiments page. On that page, you’ll see details about your current and completed experiments:

Google

You can end your experiment at any time, but as much as possible, be sure that your test period is complete and that you have enough sample size.

Here’s a quick way to access your test results:

Play Console > select an app > Store presence > Store listing experiments > select the experiment you want to check > Metric > results.

Your results would more or less look like this:

(source: TimeTune)

You can hover over the “?” to learn more about what Audience, Current installs, Scaled Installs, and Performance mean on the graph.

Google will let you know whether the alternative variant is better than the current version or not. If you see under the Performance that the graph is fully green, then it shows your test variant is the winner.

However, if it shows a red portion like this:

The result isn’t fully positive and it is better not to apply the change.

There are times that even after the set time frame of your experiment, Google will tell you that they still need more data. You will see this message:

message

If this happens, you may consider the experiment as not final yet. You can continue with the experiment or end it and provide new details.

If you wish to apply your test version, you can simply click on Apply winner. Review the changes and go live. On the other hand, you can click on Keep to reject changes and complete the experiment.

You can also click on the Stop experiment if there’s no clear winner in your test.

Step 5: Plan ahead

Now that your experiment is over and you got your result, there’s more to do than just accept or reject your test version.

For example, if you want to counter-check your results, you can run your test once again. However, this time, you would switch the variables. This means your test version would now be the current version while the current version is the test version. If you get the same result, then this will give you more confidence in your decision.

If ever you decide to apply changes, you need to track the conversion rate. Check whether the conversion rate does increase like what the A/B test predicts. This will confirm whether the test result was decisive or not.

You can also run other app A/B tests. You can create as many test versions as you like and finally get the right configuration.

Testing is not a one-time event. You don’t simply do it in just a few days, weeks, or months.

You can test as many times as you like throughout your app’s lifespan to constantly make necessary adjustments that will lead to more installs and conversions.

Quick tips when doing Google Play Experiments

Now that you got an overview of Google Play experiments, there are a few more things you might want to remember.

Here are some of them:

Limit your tests to just two versions. The more versions you test, the more complicated the test results become. Not only that but your result would yield lower statistical confidence, which might lead to false positives or false negatives.
When testing app preview videos, you have to upload them on YouTube. While the most sensible thing to do is put it in public, it may affect your A/B tests. Thus, you may want to set your app preview as an unlisted video. This way, you can prevent people outside of the app store from interfering with your A/B tests, making your results more accurate.
Aside from A/B testing results, you can also use YouTube Analytics. It should give you more details and data about your app preview video performance.
When running your experiment, it is to just live it alone until you have enough data. It’s quite tempting to check results every once in a while. However, this might cause you to prematurely stop the test. Remember, data are not final yet until your A/B test runs its course completely.
If you’re doing a localized test, as much as possible, choose only one language. Using too many languages can lead to inaccurate results.

Improve your app’s success

There’s no doubt: Google Play Experiments are super helpful in improving your app store listing. It drives more traffic, increases installs, and even improves conversion.

However, as impressive as GEx is, it comes with some limitations.

For example, experiments can only measure activities within Google Play. It’s not effective in taking into account external factors that could greatly affect the result of your test.

If you have conducted marketing campaigns outside of Google Play Store, experiments won’t be able to tell you whether your external marketing effort affected your A/B testing or not and if yes, by how much.

That’s why you can’t fully rely on experiments. If you wish to get more accurate, reliable, and comprehensive data, you need the help of ShyftUp.

ShyftUp offers its best user acquisition tools to help you get as many users as possible. They have the right knowledge, proven experience, and cutting-edge technologies to increase installs and support conversion.

Get in touch with them today and see how they can improve your app’s success!

Are Google Play Experiment results accurate?

Some app developers mentioned that GEx is not always 100% accurate. There are times that GEx predicted growth in conversion rate, but it never happened even after weeks of implementation. Thus, you need to weigh all possible factors to ensure you get accurate results.