facebookRetention-1

The problem of computing n day retention

Of all the components of the user lifecycle, retention is probably the most important metric for freemium apps and Free to Play games. Retention is fact a direct factor in the computation of the lifetime customer value (LTV) of the users acquired. For instance, early retention metrics such as Day 3 retention can serve as proxies to assess traffic quality for ROI-positive user acquisition campaigns. Also, retention is also a strong indication of the general quality of the app and its user experience.

However retention has been for app publishers a problem. Everyone talks about it, but there seems to be no clear consensus on a common definition of what retention really means nor how it is actually calculated...

The truth is that there are several ways to compute this metric, all of which lead to sensibly different results. In turn, people often compare apples with oranges so doens't have any sense...

We won't focus here to find out which definition is the right one. We'll rather look for establishing a list of the main retention computation methods, and take a look at what they are good for. There won't be a method better that the other, they are just suited for different analytical goals.

General definition

Generally speaking, all of the retention metrics can be calculated the following way:

\mathrm{nDayRetention}=\frac{Number\ of \ user\ retained\ on\ day\ n}{Number\ of\ users\ that\ installed\ the\ app\ on\ day\ 0}

The problem is that the numerator of the previous equation can have multiple definitions. We'll look across them to understand the underlying problem for each of them. We'll compare for each of the following methods the example of computing 30-day retention.

 

Main types of user retention

Let's see different approaches and examples to understand each one of the retention types. We'll use these two icons to: Para ello utilizaremos dos iconos importantes:

  • tickShow the minimum days that the user needs to open the app in order to be considered a retained user.
  • tickExample of a day that the user is not considered retained despite he opens the app.

1. Full n-day retention

  • Computation: by calculating which proportion of users come back every single day to the app until day n.
  • Why is it used: this definition is really restrictive and not really used for companies, but it gives the best idea of the level of engagement in the app.
  • Data Storage: in order to calculate it, the company must store, for every single user exactly which days the user logs in. If a user has all the login flags for all the days between 0 and n,  then we consider that the user has been retained for n days.
  • Day 30 example: only users who come back every single day from day 1 to day 30 are considered retained for this model.
Full retention

Full retention

2. Classical n-day retention

  • Computation: by calculating which proportion of users come back to the app on day n.
  • Why is it used: this definition is the easiest to compute analytically and also the most used for companies. It gives a general idea of the level of engagement in the app.
  • Data Storage: in order to calculate it, the company must store, for every single user exactly which days the user logs in. If a user has the login flag exactly for day n, then we consider that the user has been retained for days.
  • Day 30 example: only users who come back exactly on day 30 are considered retained for this model. It doesn't matter if they haven't come back on the 29 previous days or how ofter they come back.
Classical Retention

Classical Retention

3. Not yet churned n-day retention

  • Computation: by calculating which proportion of users come back to the app on day or after that day.
  • Why is it used: this definition is one of the most common in business intelligence apps like Flurry, and is an approach focused on measure the churn rate of the app instead of the retention (i.e. users that have totally abandoned the app). In fact, this approach of n-day retention is just 1- n-day churn rate.
  • Data Storage: in order to calculate it, the company only needs to store for every user the first and last login date, so this kind of n-day retention is given by computing how many users have a difference between the last login and registration date greater or equal than n. 
  • Day 30 example: only users who come back exactly on day 30 or after this day (for instance, on day 60) are considered retained for this model. It doesnt' matter the behaviour of the user on the first n days, just from this day onwards.
Not yet churned retention

Not yet churned retention

4. Return n-day retention

  • Computation: it's the proportion of users that come back to the app at least once within the first n days since registration moment.
  • Why is it used: This definition is really used on gambling games, where the drop off on the first session tends to be really high. It gives an idea of how many people didn't drop out after the first open. They usually call this people retainers.
  • Data Storage: in order to calculate it, the company must store, for every single user exactly which days the user logs in. If there is any login within the first n days, then the user is considered retained for this model.  
  • Day 30 example: only users who come at least once before day 30 are considered retained for this model. It doesnt' matter how many times the user comes back on the first 30 days nor what happens from the day 30 onwards.
Return retention

Return retention

5. Interval n-day retention

  • Computation: it's a specific and restrictive case of return retention. In this case, we define an interval between typical retention marks such as day 1, 3, 7, 30, 60, etc... Day m is then defined as the lowest bound of the interval and n as the highest bound. The metric will measure the proportion of users that come back to the app at least once between days m and n.
  • Why is it used: This method is useful to understand user behaviour and to find out usage patterns of the app.
  • Data Storage: in order to calculate it, the company must store, for every single user exactly which days the user logs in. If there is any login between days and n, then the user is considered retained for this model.  
  • Day 30 example: taking for instance n = 30 and m = 7, users coming back on day 13 or 22 are considered retained for this model. However, users coming back on day 5 but not after day 7 are not considered retained.
Interval Retention

Interval Retention

To sum up

Efficiency (on both computation and data storage sides) vs Meaningful (restrictive and relevance). It all comes to a point where each Data Scientist must say which one he prefers. This is my point of view, but yours can be different. Tell us your thoughts!

Restrictive Relevance Data Storage Computation
Full 5 3 Huge Hard
Classical 4 4 Huge Very Easy
Not churned 3 5 Low Easy
Return 2 1 Huge Normal
Interval 1 2 Huge Normal

Limitations

When retention is required to be computed in real time, the most efective one is the not churned approach, that doesn't requires to look after data on each day but only of days that have passed from registration to last login. Of course, this approach loses lot of data between those dates. For instance, if the user has not loged on the first n days, and suddenly the marketing team launches a huge reengagement campaign on day n. This will for sure raise the Not churned n-day retantion but it won't be really realistic...

There is also another underlying problem: while the approaches that need to store data from any single session of the user won't change the n-day retention as time goes by, the not churned one will suffer changes day by day. For instance, imagine that we have 15% of 30-day retention, with the aim of raise this value we do a remarketing campaign (like email, push notification, ads,...) then:

  • Full retention won't be almost changed whenever we fire the reengagement campaign.
  • Classical retention will be changed if the campaign is set on day or a few days before.
  • Not churned retention will be changed whenever the campaign is set.
  • Return retention will be changed if the campaign is set on the first n days since registration.
  • Interval retention will be changed if the campaign is set between days m and n since registration.

So, it seems that this is a quite difficult to compute kpi... Not really stardardized and with different results depending on, not only how it is computed but also how  is it affected my other marketing and reengagement actions.

Always keep in mind that retention metrics are highly dependent on the type and category of the app/game.

So, next time you hear someone talking about retention, don’t forget to ask them: Yes, but which one?

Leave a Reply

Your email address will not be published. Required fields are marked *