Chapter 22

Measuring and monitoring

Numbers are powerful. They help us to figure out what needs to change, where to spend money and how to plan for the future. Dozens of web apps can measure thousands of different data points about your customers and product, but this often comes with unfortunate consequences:

When you reach 10,000 customers you can collect and analyse every conceivable dataset to eke out minor improvements. Until then, concentrate on key actionable data. In this chapter I’ll highlight ten core metrics that you can use to make important decisions about your app. I’m also going to assume that you use Google Analytics, the free analytics software that has almost everything you need.

Metrics

A sales funnel is a useful way to separate and measure how customers discover, buy and use your app.

img-22_1

A five-stage sales funnel

The typical funnel starts when you capture the attention of part of your target market, usually through marketing and advertising. Anyone who arrives at your website can be considered a prospect, a potential customer. Some prospects may leave, but others will give a signal that they are interested in the app (signing up to the newsletter, clicking through multiple pages, trying the demo) and become qualified or engaged prospects.

If your app offers a trial or freemium option, some engaged prospects will sign up to become users. Those who eventually pay for the app become customers, and with any luck some will develop into advocates who share and promote your app, bringing more prospects into the top of the funnel.

You ideally want the funnel to be wide all the way down, by converting the maximum number of people at each stage. Let’s take a look at the metrics we can calculate at each stage to identify obstructions in the funnel and take corrective action.

Stage 1: Prospects

At this stage of the funnel you need to collect data about where your prospects are coming from so that you can better target your marketing.

Metric 1: Acquisition channels

Data collected: channel; number of prospects; cost per channel; cost per prospect.


For example:

Date Channel Number of prospects Total costs Cost per prospect
May 2011 Google search 459 $0 $0
Google AdWords 160 $49.31 $0.31
Twitter 77 $0 $0
Email Campaign 301 $19.00 $0.07
June 2011 etc…

Example acquisition channel data

With this data you can gauge the effectiveness of your marketing efforts, though not their value, until we know which are more likely to convert to paying customers. The data should inform where to invest in marketing, whether previous changes had a negative or positive effect, and, therefore, whether to change tack or not.

Stage 2: Engaged prospects

You need to decide what behaviour constitutes an engaged prospect for your app. For a market that has a long sales process, like enterprise software, an engaged prospect might be defined as one who views three or more informational pages, or has a session lasting over five minutes. A low-cost consumer application might define an engaged prospect as one who clicks the demonstration link or subscribes to the blog RSS. Whatever the engagement goal, you need to record how many prospects reach it.

Metric 2: Engagement goals

Data collected: channel; number of engaged prospects; percentage of total prospects.


For example:

Date Channel Number of engaged prospects Percentage of total prospects
May 2011 Google search 184 40%
Google AdWords 110 69%
Twitter 9 12%
Email Campaign 102 34%
TOTAL 405 41%
June 2011 etc…

Example engagement goal data

This data isn’t only used to better assess the value of marketing channels, but can highlight opportunities for increased engagement. From the data above, we could reliably conclude that the target keywords of the AdWords campaign are effective at capturing engaged prospects, and so incorporate them into the Google search channel through SEO updates. We might also notice that AdWords prospects are directed to a specific landing page rather than the generic home page, and so we can start to use this same tactic for the more poorly performing email campaign.

img-22_2

Setting up a goal to measure engaged prospects in Google Analytics

Stage 3: Users

Visitors who sign up to the app, though not yet as paying customers, represent an opportunity to analyse the efficiency of the registration process and app use.

Metric 3: Usage

Data collected: number and position of clicks using a click heat map.

A number of free and commercial web apps will measure and display the positions of clicks from your visitors. Search for click heat maps to find a tool that suits your budget1.

A click heat map of your app interface will literally highlight the most-used features of the app and, by omission, reveal features that are not frequently used. If there are valuable features that first-time users aren’t noticing, adjust your introductory help text and app interface accordingly to convince more users to convert to paying customers.

Click maps also expose false affordances in your app interface, sign-up form and marketing website. These are elements of the interface that appear to be clickable (or the user expects them to be clickable) but aren’t, and consequently cause frustration and dissatisfaction. Remove, re-style or create links for all false affordances that the heat maps reveal.

The event tracking2 feature of Google Analytics provides a quantitative alternative or addition to the qualitative heat map reports. Embed the event tracking JavaScript calls into your interface code to record clicks on specific parts of the interface: download links, menu items, action buttons, and so on. This is particularly useful for highly dynamic interfaces with pop-ups, pull-downs and other interface elements that can’t accurately be tracked on a flat heat map.

If your sign-up form is more complex than a few form fields you should also integrate event tracking code that logs changes or focus to each field3. The resultant data will reveal form abandonment rates: how far users reach in the form registration process before they give up. This data is difficult to record through click heat maps alone, as they do not take account of keyboard tabbing between fields.

Metric 4: Sign-up form abandonment

Data collected: form field name; number of times the field has focus.


For example, if an event is logged when a field has focus:

Event (field) name Event quantity
Full name 105
Choose username 104
Choose password 87
Agree to terms 83

Example form abandonment data

This data shows that most users arrive at the username field but a significant number don’t move on to the password field. It implies that the client-side username validation code needs further attention: perhaps it is too restrictive on username format or doesn’t suggest enough useful alternative usernames. Whatever the reason, we have pinpointed a specific problem that can be addressed to increase the conversion rate.

Stage 4: Customers

Paying customers are the most important section of the funnel to measure. They represent the segment of the market that you most want to attract (and that you appeal to most) and their longer-term loyalty measures the ongoing success of your product/market fit.

Metric 5: Customer conversions

Data collected: channel; number of prospects; number of customers; conversion rate; cost per acquisition.

The first few metrics assessed the ability of different marketing channels to attract prospects. Now we can look at the actual monetary value of each channel by examining which ones create the most customers for the lowest spend on marketing.

Date Channel Number of prospects Number of customers Conversion rate Cost per acquisition
May 2011 Google search 459 18 4% $0
Google AdWords 160 14 9% $3.52
Twitter 77 1 1% $0
Email Campaign 301 9 4% $2.11
TOTAL 997 42 4% $1.63
June 2011 etc…

Example customer conversion data

This data shows that the AdWords campaign converts the highest proportion of prospects to customers, but the email campaign acquires customers at a lower cost per customer. A small marketing budget would be better spent, therefore, on an expansion of the email campaign rather than AdWords, assuming that a follow-up campaign could target a larger segment of the market reached by the original campaign.

Metric 6: Retention rate and monthly cohort

The retention rate measures how many customers you keep from one period to the next. Web apps usually bill monthly, so this makes a good period for measurement. Retention rates are usually expressed as percentages or decimal fractions of 1: if you had 100 customers on 1 May but 25 of those had cancelled by 1 June, your retention rate would be 75% or 0.75. The rate R, as a fraction, can be calculated using the equation* where Cm is the number of accounts cancelled during a particular period (month) and Nm is the total number of accounts at the start of the period4.

img-22_3

Unless your app is designed specifically for use in short bursts, aim for a bare minimum retention rate of 70%. Any less and you should temporarily shift your focus away from marketing and towards increasing retention, to prevent the bleeding of customers from your sales funnel. The easiest way to find out why customers aren’t sticking around is to ask them, perhaps through a field asking their reason for leaving on the cancellation form or customer survey.

As well as calculating your retention rate each month, you can also track the long-term retention rate for the group of users who join each month. For example, 62% of May’s new customers are retained in June, 53% of them remain in July, only 45% in August, and so on. Tracking these monthly cohorts of customers allows us to examine the longer-term effects of our marketing.

The above graph shows example retention rates for four monthly cohorts: customers who joined in May, June, July and August. Note how we have less data for the most recent cohorts, as we are tracking the number of months since they joined.

img-22_4

Tracking the retention rate for four monthly cohorts

In this example, it was decided at the start of July that a new weekly marketing email would be sent to customers. The retention rate for July’s customers (green) shows an improvement over June’s customers (red) in the first month, but we can also see that the aggressive email campaign appears to cause a steeper drop-off in retention in month two. At the end of August, the emails were slowed to every two weeks and altered to include helpful tips and an easier opt-out. This seems to improve the retention during the second month for the August cohort.

Metric 7: Return on investment

Data calculated: average selling price; lifetime value; return on investment.


It’s time to put a price on people’s heads.


If your app is available in a range of prices, you may find that customers from different marketing channels tend to choose different price points. For example, your AdWords campaign might be targeted at higher-spending enterprise customers and your email campaign at mid-price self-employed consultants.

The average selling price for each channel is simply the total value of all accounts created via that channel, divided by the number of accounts. If AdWords has referred 23 customers at $4.99 and 7 customers at $9.99, the average selling price (ASP) is

img-22_5

We can use the ASP in conjunction with the retention rate (R) to calculate the total amount of revenue generated by an average customer over the lifetime of their subscription. This lifetime value (LTV) can be approximated using the equation

img-22_6

If you wanted to be more accurate, you could calculate and use a retention rate per channel rather than an average across all channels.

The return on investment (ROI) is simply the lifetime value of a customer minus the cost of acquiring the customer (cost per acquisition, calculated earlier), and represents an approximate profit per customer, not taking into account any development, hosting or other non-marketing costs.

Channel Average selling price Lifetime value Cost per acquisition Return on investment
Google search $5.54 $22.16 $0 $22.16
Google AdWords $6.16 $24.64 $3.52 $21.12
Twitter $5.30 $21.20 $0 $21.20
Email Campaign $6.08 $24.44 $2.11 $22.33

Example return on investment data

A large ROI will highlight the cost-effective marketing channels, but it will also show you where you can spend more aggressively to gain market share. Although high profits are attractive, it often makes sense for new start-ups to keep profits relatively low and re-invest as much possible in early customer acquisition.

Stage 5: Advocates

Word of mouth is the best form of marketing for a small web app. It costs next to nothing and refers visitors who are more likely to convert.

Metric 8: Viral coefficient

The viral coefficient (V) measures the virality of your app: for every visitor or customer, how many friends do they bring with them? It can be expressed with the equation:

img-22_7

Where N% is the percentage of users who send an invitation, I is the average number of invitations sent, and A is the percentage of users who accept the invitation. Percentages should be expressed as fractions (75% = 0.75) and invitations can also represent other sharing methods, like posting a link to your app on Facebook or Twitter.

For example, if 20% of your customers send an invitation to an average of 7 friends, and 60% of those accept the invitation, your viral coefficient is:

img-22_8

The higher the number, the more viral your app. As the number essentially represents the number of new customers that each customer brings with them, a viral coefficient greater than one represents viral, exponential-like growth. If each new customer brings in slightly more than one new additional customer, you have a self-sustaining viral growth mechanism.

This is important to measure because viral sharing is not something that happens only by chance, but is something that can be influenced, optimised and improved. In the most basic sense, this can be the inclusion of a Tweet this or Facebook Like button, but real virality must be built into the app features. Consider how a customer’s use of your app could improve because they share it with their friends, and how you can expedite the sharing process, through contact importers, invitation URLs, and so on.

Metric 9: Satisfaction

The traditional metric for customer loyalty is the Net Promoter Score5, which asks customers to rate from 0 to 10 how likely it is that they would recommend a company to a friend or colleague.

Web marketing expert Sean Ellis6 devised a small twist on this question: ‘How would you feel if you could no longer use [app name]?’ with three possible answers: very disappointed; somewhat disappointed; not disappointed.

The interesting thing about this metric is that on the surface it looks like a question about loyalty or satisfaction, but what you are really testing is the product/market fit of your app: how well are you meeting the important needs of your customers?

Ellis suggests7 that you can use this measurement to calculate when to switch focus from product development (striving to reach product/market fit) to marketing and scaling up the business. Using data from nearly 100 start-ups, he proposed that at least 40% of your customers should answer ‘very disappointed’ to signify that you’ve reached product/market fit.

The survey.io8 web plug-in is an easy tool with which to survey your customers using this question.

Segments

This last metric is a bit of a cheat because it actually encompasses multiple metrics.

Metric 10: Other segments

We’ve previously discussed a number of measurements that segment data by marketing channel. Metrics 1, 2, 5 and 7 measured the quantity of visitors, how many engaged, how many converted, and their monetary value, each segmented by the original referring channel (AdWords, Twitter, email, and so on).

As your app grows, it can be useful to re-segment visitor and customer behaviour by other demographic or geographic factors, data permitting. For example, a geographic segmentation may reveal that customers from particular countries convert more easily or spend more money. Armed with this information, you can take action with newly targeted adverts and additional interface translations.

If you decide to explore other segmentations, be careful not to confuse correlation with causation. It’s easy to assume that the US is your most attractive market because 80% of your current customers are American, but be aware of inherent biases in your app interface and marketing text. It your app is hosted (and performs faster) in the US, displays prices in US dollars only and suggests through ‘ZIP code’ and other language choices that it’s a US-specific service, you will inadvertently bias the composition of your customer data.

Monitoring

Poor performance and availability have a direct effect on revenue. Although you don’t necessarily need to analyse these figures on a daily basis, it makes good sense to arrange a monitoring solution to notify you of significant changes to uptime or response speed.

*They will also wake you up at 3am with a terrifying downtime SMS alert

Pingdom9, Chartbeat10 and other monitoring apps are simple to configure and give you some basic peace of mind*. Most also offer a free version that is good enough to get you started.

As your app grows and your infrastructure expands to multiple servers and interdependent services, you can move to a comprehensive monitoring dashboard like Nagios11 or Hyperic12, both of which offer free open source versions of their software.

For additional confidence in your app availability, use Nagios or Hyperic in conjunction with a Selenium plug-in. This enables you to re-use your Selenium browser workflow tests for active monitoring. Rather than just test the uptime of your home page, which may be static and not representative of the availability of specific functionality, your monitoring tool can use the Selenium tests to repeatedly run through the important customer workflows on your live system (sign up, login) and alert you if they fail.

Summary

You should be measuring:

Web App Success book coverAlso available to buy in a beautiful limited edition paperback and eBook.

This work is licensed under a Creative Commons Attribution 4.0 International License.