GDPR crashed my userbase
Retention is one of the KPI most valued in the F2P Mobile Industry. It becomes a priority to focus first on the on boarding and D1 retention when analyzing and improving a game. This was the case of our client where we were running a routine on the tutorial steps and D1 retention features. So the first release was doing fine, retention moved up a few points on D1 and D2, but fixes where needed so we made a Second Release.
After doing a 100% roll out on the Second Release, we suddenly saw a big impact on retention. It raised 15 points in one week. We used Firebase from Google to measure retention on this project, so we were confident there was no mistake on the engine metrics.
The client had total control over the production team and asked the developers to fix a bug with the GDPR. The fix was to deactivate Firebase at the startup and only activated it after the Privacy Policies were accepted.
The fix changed the amount of players measured on the on boarding process, but kept the amount of players on D1 as nothing really changed on organic acquisition. The first_open event was triggering less times for the same amount of new users. So comparing retention before the roll out with the retention after roll out showed up a fake increment. You can see in this example, how tricky is this situation.
first_open: from 10,000 to 6,360 users (-36% increment)
D1 users: stayed at 3,500 users
D1 retention: from 35% to 55% (+36% increment)
(this numbers works as an example, the real information is protected by an NDA Agreement)
The D1 amount of users is the same, but the first_open event is trigger less times. So the increment in retention is due to a change on how analytics are used, and not due to a better onboarding implementation.
To validate this hypothesis we needed a second point of reference to compare the delta increments. We used Google Play Console metrics. We calculated the daily delta on each month for: installs on Google Play Console, first_open event and D1 retention on Firebase. Then we compared all three deltas to see the differences.
To calculate deltas we just set the first day of the month as the control sample and calculate the offset. You can see next all three deltas together in the same chart from May to August. Check symmetry on the curves.
BLUE: Installs in Google Play Console
RED: first_open in Firebase
YELLOW: D1 retention in Firebase
We can validate the consistency of the deltas from May to July, with a small difference during a User Acquisition Campaign. “The Error Found” red area in August shows up right before the Second Release including the GDPR fix. This erratic shape only appears when the user base changes in the middle of the month and for only one metric (first_open hits).
We can confirm the issue by calculating the deltas starting on the New Release date (August 11th). The result is that all three variables: Google Console installs, first_open and Retention D1 on firebase; aligns again.
We could not used the first_open event on Firebase to calculate improvements anymore. The event was removed from all the funnels and all the possible retention improvements could not be registered or identified as part of our work. We will never know if part of that increment was due to our new implementations. All further analysis comparing periods (before and after the GDPR fix) will have to ignore retention.
We learnt from this experience that all the fixes and features that enters the release pipeline must be shared with the Product Team, even if it means to just change a boolean in a line. The production process and team members must be aware of this kind of issues, and be prepared to validate the development commits and document a proper changelog.
We also learnt that even if a feature was confirmed on an A/B Test, to validate the real impact in the game, it must be shipped alone. Do not mix features on a same build, as you may mix the impact on the KPI and end with irrelevant information. As happens with the sound waves, two features can cancel each other.
As a final lesson, we now always monitor the consistency of unique users on daily retention apart from the percentage of the current Analytics Engine to detect sudden changes on the user base and avoid the corruption of the analysis.