My colleague Timur recently spoke at Google Play about how we can easily make the wrong decisions by looking at top line metrics in test results, it’s a cool topic so I thought I’d post about it too. I’ll share a couple of basic examples from recent tests we’ve run that involved a bit of deeper thinking.

“There is nothing more deceptive than an obvious fact.” 
― Arthur Conan Doyle, The Boscombe Valley Mystery

Test: Sending a heart in chat

On Badoo, we tested adding a button into the messenger component that allows users to send a heart with one tap. The heart is synonymous with Badoo and is one of our most sent emojis, so we thought it’d be a nice touch.

At first glance the results looked great,

  • 20% of users sent at least 1 heart
  • Messages sent were way up

But when we dug deeper we realised that when someone was sent a heart, women were 35% less likely to respond to it than a normal message and for men it was 6% worse. We also realised that they tended to kill a conversation (a flow of messages between users)

The most important metric in this case was not the number of messages sent, but the quality of the conversations – so we rolled back the change

Test: Encouraging better chat openers

On Bumble, we know when someone sends a great, personalised opening message – they’re more likely to get a reply and end up having a meaningful connection. My colleagues John and Ben ran a project to encourage users to send better openers by prompting them when they wrote a generic phrase like ‘Hey’ ‘Hi’ ‘Hey there’ etc.

On initial glance, we had a drop in women initiating chats. This is a pretty terrifying statistic for a dating app where only women chat initiate chats!

But when we pushed past this it wasn’t the whole story. We also had

  • More women initiating at least one chat
  • A better reply rate to those chats that were sent
  • An increase in initiated chats that led to a meaningful conversation.

I could give many more examples, and I’m sure that we’ve made the wrong decision for some tests in the past by looking at top level metrics and getting nervous.

The way to beat this is building a strong hypothesis and making sure you know deeply the behaviour you’re trying to change – which may go beyond measurement in your normal KPI’s