Run A/B Tests with PostHog and Flask

Jose Salvatierra -

When your application starts growing, you start wondering... Should my checkout button say "Checkout" or "Buy now"? Which one will make my customers buy more? Also, should it be green or blue?

If you have these questions, then A/B tests are the answer!

An A/B test lets you compare two versions of your code side by side, and check which version of the code achieve more of your goal.

Your goal may be increased conversions (purchases), or it may be more clickthroughs, more returning visitors, or anything else!

In PostHog, A/B tests are implemented using feature flags. If you haven't read our article on using feature flags with Python and PostHog, I recommend reading it first!

As a quick recap, once you've created your feature flags (or A/B tests), you can retrieve them in your Flask app with this snippet:

@app.before_request
def before_request():
    if "user_id" not in session:
        if current_user.is_authenticated:
            session["user_id"] = current_user.id
        else:
            session["user_id"] = str(uuid.uuid4())

	if current_user.is_authenticated and current_user.id != session.get("user_id"):
        posthog.alias(session["user_id"], current_user.id)
        session["user_id"] = current_user.id

    try:
        session["feature_flags"] = posthog.get_all_flags(
            session["user_id"], only_evaluate_locally=True
        )
    except:  # noqa
        session["feature_flags"] = {}

How to create an A/B test with PostHog

Let's say that you want to check whether the text "Purchase", "Buy now" or "Subscribe" performs better in the main homepage button.

First, click on the A/B tests section of the navigation, and then click on "New experiment":

Then you'd create your A/B test using PostHog:

Here you'd also give each test variant a value. Leaving one as "control" is the norm, so you always know that that's the default. The other can get a name that's meaningful for what you are testing. In this case, "buy_now" and "subscribe" may be suitable.

In the A/B test configuration you can select conversions as the metric you want to see a change in.

If you haven't set up event tracking with PostHog, read our article on how to do that!

Finally, you can optionally specify the minimum acceptable conversion. This helps PostHog tell you for how long the experiment should run, and when it has achieved the goal.

It can happen that there is no statistically significant difference between your variants! PostHog will tell you that in this screen as the experiment progresses, and in that case it won't matter which variant you opt for.

How to implement the A/B test in code

Now that we've got the A/B test created, we can use the PostHog client in our code to implement the two paths. For changing the button text, we'd do it in the template:

<button class="px-4 py-2 rounded-md shadow bg-green-500">
    {%- if session["feature_flags"].get("purchase_button") == "control" -%}
        Purchase
    {%- else if session["feature_flags"].get("purchase_button") == "buy_now" -%}
        Buy now
    {%- else if session["feature_flags"].get("purchase_button") == "subscribe" -%}
        Subscribe
    {%- endif -%}
</button>

When the experiment ends—after you've selected the winning variant in PostHog—it's important to go back and delete these if statements to simplify the code. At that point you'd just keep the code from the winning variant.

Simplify accessing feature flags in Jinja with a filter

Although that works fine, if we have a lot of feature flags, we can shorten the code quite a bit by using a custom Jinja filter.

In your app definition file, you can add this:

@app.template_filter("is_active")
def flag_is_active(feature_flag: str, test_name: str = None) -> bool:
    flag_value = test_name or True
    return session["feature_flags"].get(feature_flag) == flag_value

This is a simple filter called is_active that will let us check wither:

  • A feature flag is active or not; or
  • An A/B test has a specific value.

To use it in your template, you'd do this:

<button class="px-4 py-2 rounded-md shadow bg-green-500">
    {%- if "purchase_button" | is_active("control") -%}
        Checkout
    {%- else if "purchase_button" | is_active("buy_now") -%}
        Buy now
    {%- else if "purchase_button" | is_active("subscribe") -%}
        Subscribe
    {%- endif -%}
</button>

That's for an A/B test. If you wanted to use it for a feature flag without a payload, you could just do:

{% if "new_feature" | is_active %}

Conclusion

That's everything for now! I hope you've learned something new about running A/B tests with PostHog and how to use them.

If you want to learn more about web development with Flask, check out our Web Developer Bootcamp. You can enrol for a one-time fee, or subscribe to gain access to all our courses!