Split testing is a core lean startup practice we use regularly at BackType. Whenever we release a new feature we usually split-test it first to measure its effect. This post walks through how we setup experiments, measurement and reporting using PHP, Javascript and Mixpanel:

Experiments

We configure experiments with a number of parameters:

class Experiment
{
    function __construct($feature, $label, $id, $rates, $active=true)
    {
        $this->feature = $feature;
        $this->label = $label;
        $this->id = $id;
        $this->rates = $rates;
        $this->active = $active;
    }
}

Our experiments are identified by a label and id; if they're active, they apply to a feature at a split rate. We use this simple Experiment object for a one-line split test:

$EXPERIMENTS[] = new Experiment('index', 'welcome_message', 'm-s1', true, array('control' => 80, 'test1' => 20));

We organize multiple experiments for execution with the following class:

class Experiments
{
    function __construct()
    {
        // save experiments
        $this->list = array();
        // generate a seed for the current user
        $this->seed = hexdec(substr(session_id(), -2)) % 100;
    }

    function add($experiment)
    {
        $this->list[$experiment->feature.':'.$experiment->label] = $experiment;
    }

    function do($key)
    {
        $experiment = $this->list[$key];
        $ret = 'control';
        if ($experiment->active == true)
        {
            // select a mode to run the experiment in
            $r = 0;
            $cutoff = ($this->seed + abs(crc32($experiment->id))) % 100;
            foreach ($experiment->rates as $var => $rate)
            {
                $r += $rate;
                if ($cutoff <= $r)
                {
                    $ret = $var;
                    break;
                }
            }
        }
        return $ret;
    }
}

Once experiments have been configured and added, they can be used in your template layer:

if ($GLOBALS['config']['experiments']->do_experiment('index:welcome_message') == 'test1')
{
    // display test1
}
else
{
    // display control
}

At BackType, we wanted to be able to evaluate our experiments in real-time, with visual reporting that would help us measure the macro effects of any tested hypothesis. That's where Mixpanel comes in handy.

Measurement

We perform all of our measurement via Javascript (instead, you may want to do it server-side). Follow the Mixpanel Integration Guide to get started, but in Javascript it's pretty simple:

try {
    mpmetrics = new MixpanelLib('YOUR KEY');
} catch(err) {
    null_fn = function () {};
    mpmetrics = { track: null_fn, track_funnel: null_fn, register: null_fn, register_once: null_fn };
}

Javascript (which we use with the help of Prototype) gives us lots of flexbility. For example, rather than proxying all outbound links from our site we can measure SERP conversions (whether people click on search results) with something like this:

mpmetrics.track_funnel('Search Conversion', 1, 'Impression', measure);
$$('a.mixpanel').each(function(o) {
    o.observe('click', function() {
        mpmetrics.track_funnel('Search Conversion', 2, 'Goal', measure);
    });
});

Mixpanel has code examples for performing measurement in Javascript, PHP, Python and Ruby.

Reporting

After sending data to Mixpanel, they organize nice little reports that can help validate hypotheses. For example, this is how funnel results are displayed:

Conclusion

I highly recommend split-testing heavily, but measuring selectively. What you measure depends on your application. In any case, I recommend evaluating your split-tests against a few key metrics — no more than five or so. I'll leave you with three guidelines to get started with split-testing from Eric Ries:

1. Start simple
2. Make a firm prediction
3. Don't give up