The Value of Data Consistency in Fund Raising

[Editorial notes : Corporate governance is becoming an important aspect of running a business. Lately, technology sector (including startups) have been on the spot for negligence. This guest piece by Mayank Khanduja of SAIF partners throws some clear perspective from an investor’s point of view.]

Coporate Governance
Coporate Governance

As an early stage investor, I talk to at least 100 teams a month and almost all of these conversations involve going through data on traction. But given the sheer volume, my ingoing hypothesis is that the information shared is both accurate and represents a fair picture of the business.

In some cases though, while validating this data through due-diligence, one may find a jack-in-the-box spring a surprise that tells a different story altogether. Over time, I have built the below checklist of things that either give me comfort or raise questions on the quality of the given information.

1.    Use of standard analytics tools

For most consumer facing web or mobile products, there exist standard analytics and tracking tools that have gained global acceptance, e.g., Google Analytics, MixPanel, Google PlayStore Developer console etc. Sometimes custom dashboards may be required since such standardised solutions may not be able to capture all the metrics relevant for a business.

But my red flags are raised when I see a company use a custom dashboard, even when not necessary for basics like visits, downloads, installs, active users, click-through’s etc. I have come across examples where the difference in in-app CTRs has been to the tune of 2X between custom dashboard and Google Analytics. A deeper dive into the code of one such case revealed that the click event was being counted twice in the custom dashboard!!!

2.    Use of standard definitions

“Monthly active user” for an app is typically defined as the user who has used the app at least once in that month. Let me throw a couple of cases that forced me to challenge this definition of active user

Consider the below data for an app

  • Installs at the beginning of the month – 20,000
  • Monthly active users – 7000, i.e., 35%; not too bad one would think!
  • Of these, 5000 are new users acquired in the month. All being counted as active users only because they opened the app as soon as they downloaded it and may not have used it since!
  • Thus, only 2000 of the installed base came back to the app, which reflects a completely different picture of the business!!!

Another case involved users being counted as active even when not using the app! This happened because the app tracked the user location, and a user was “Active” just because the app interacted with the server sending across location information!!! A better metric to track here could be the number of home screen opens.

3. Coverage of all the important metrics for the business

When I come across dashboards that do not track a very important metric for the business, it gets me thinking – either the founder is not completely aware of what all is important to measure or there is some bad news lurking in those missing numbers. The first one could simply be a case of over-sight. But in some instances, the untracked metric showed the business in very poor light, which leads me to question the motivation.

Two important elements, which are more subjective in nature, are also a part of this process

4.    Consistency among team members on the causes for errors in data

As naive as it may sound, the easiest way to test if the data inconsistency is due to over-sight or intentional, is to ask the same set of questions separately to the different team members. Surprisingly, it is often tough for every team member to have the same cover up story when asked independently!!!

5.    Accepting errors upfront instead of trying to cover them

I have come across teams that seem to have an explanation, whether logical or otherwise for every inconsistency in data. I am then hard pressed to think if I would be able to trust this team on the monthly MIS they share with me post investment. It is an important question considering I am signing up for a relationship with the team for the next 8-10 years!!!

No one expects founders to be “Super-Humans”. In most cases, these errors/inconsistencies crop up inadvertently – smart and honest teams accept this early enough and take remedial action. Once the data is scrubbed and I get comfort on the qualitative elements as well, I revisit the opportunity to see if it is still exciting. And quite often it remains so!!!

[Reproduced from Mayank’s blog]