Whenever we roll out an improvement on our platform at HackerEarth, we love to conduct A/B tests on the enhancement to understand which iteration helps our users more in using the platform in a better way. As the available third-party libraries did not quite meet our needs, we wrote our own A/B testing framework in Django. In this post, we will share a few insights into how we accomplished this.
The basics
A lot of products, especially on the web, use a method called A/B testing or split testing to quantify how well a new page or layout performs compared to the old one. The crux of the method is to show layout "A" to a certain set or bucket of users and layout "B" to another set of users. The next step is to track user actions leading to certain milestones, which would provide critical data about the "effectiveness" of both the pages or layouts.
Before we began writing code for the framework, we made a list of all the things we wanted the framework to do:
- Route users to multiple views (with different templates)
- Route users to a single view (with different templates)
- Make the views/templates stick for users
- A/B test visitors who do not have an account on HackerEarth (anonymous users)
- Sticky views/templates for anonymous users as well
- Support for A/A/B or A/B/C/D…./n/ testing
- Analytics
A/B for Views

A/B for Templates

Getting the logic right
To begin with, we had to categorize our users into buckets. So all our users were assigned a bucket number ranging from 1 to 120. This numbering is not strict and the range can be arbitrary or per your needs. Next, we defined two constants—the first one specifies which view a user is routed to, and the second one specifies the fallback or primary view.
AB_TEST = {
tuple(xrange(1, 61)): 'example_app.views.view_a',
tuple(xrange(61, 121)): 'example_app.views.view_b',
}
AB_TEST_PRIMARY = 'example_app.views.view_a'
Next, we wrote two decorators which we could wrap around views—one for handling views and the other for handling templates. In the first scenario, the decorator would take a dictionary of views, a primary view, and a Boolean value which specifies if anonymous users should be A/B tested as well.
Decorator Code
"""
Decorator to A/B test different views.
Args:
primary_view: Fallback view.
anon_sticky: Determines whether A/B testing should be performed on anonymous users.
view_dict: A dictionary of views (as string) with buckets as keys.
"""
def ab_views(primary_view=None, anon_sticky=False, view_dict={}):
def decorator(f):
@wraps(f)
def _ab_views(request, *args, **kwargs):
view = None
try:
if user_is_logged_in():
view = _get_view(request, f, view_dict, primary_view)
else:
redis = initialize_redis_obj()
view = _get_view_anonymous(request, redis, f, view_dict, primary_view, anon_sticky)
except:
view = primary_view
view = str_to_func(view)
return view(request, *args, **kwargs)
def _get_view(request, f, view_dict, primary_view):
bucket = get_user_bucket(request)
view = get_view_for_bucket(bucket)
return view
def _get_view_anonymous(request, redis, f, view_dict, primary_view, anon_sticky):
view = None
if anon_sticky:
cookie = get_cookie_from_request(request)
if cookie:
view = get_value_from_redis(cookie)
else:
view = random.choice(view_dict.values())
set_cookie_value_in_redis(cookie)
else:
view = primary_view
return view
return _ab_views
return decorator
Helper Function
def str_to_func(func_string):
func = None
func_string_splitted = func_string.split('.')
module_name = '.'.join(func_string_splitted[:-1])
function_name = func_string_splitted[-1]
module = import_module(module_name)
if module and function_name:
func = getattr(module, function_name)
return func
Putting things together
Let’s assume that we have already written the A and B views. Let’s call them view_a
and view_b
. To get the entire thing working, we will write a new view called view_ab
and wrap it with the decorator.
@ab_views(
primary_view=AB_TEST_PRIMARY,
anon_sticky=True,
view_dict=AB_TEST,
)
def view_ab(request):
ctx = {}
return ctx
For the sake of convenience, we require this new view to return a dictionary.
Finally, we need to integrate analytics to collect data. We used Mixpanel at the JavaScript end to track user behavior. You can use any analytics or event tracking tool for this purpose.
This is just one of the ways you can do A/B testing using Django. You can always take this basic framework and improve it or add new features.
P.S. If you want to experiment with an A/A/B or A/B/C testing, all you need to do is change the AB_TEST
constant.
Feel free to comment or ping us at support@hackerearth.com if you have any suggestions!
This post was originally written for the HackerEarth Engineering blog by Arindam Mani Das.