3 Product Metric Pitfalls You Have to Keep away from

In my e book, The Insights Pushed Product Supervisor, I cowl why it’s necessary to trace much less to create extra focus and spend extra time on extracting true insights out of your information.

The following step is to make it possible for on the nitty-gritty stage, you might be monitoring what I name “good high quality metrics.” This put up—an excerpt from chapter 7 of my e book—will give attention to the best way to enhance the general high quality of your metrics, the best way to make them extra actionable, and what pitfalls to keep away from.

Pitfall #1: Self-importance metrics

A number of years in the past I used to be engaged on a B2B SaaS product for managing workplace environments. We simply launched the product and began to run our first paid promoting campaigns, so I arrange a dashboard that tracked the overall variety of sign-ups over 30 days:

The numbers appeared to be going up, so we have been fairly glad in regards to the momentum.

The issue was that whereas this graph appeared good on shows, the laborious reality was that solely 4% of the brand new sign-ups become precise conversions and income, and because of this, we didn’t meet our month-to-month recurring income targets.

It’s a cumulative chart, so the worst case can be that the graph would simply plateau if we don’t purchase any new customers, however the quantity can by no means lower. It’s a traditional instance of an arrogance metric:

• Taking a look at this graph made us really feel good.
• This metric was particularly helpful in stakeholder shows.
• It didn’t give us any perception into whether or not we have been truly doing properly, or not.
• As a result of it didn’t give us any perception, it didn’t drive us to take motion to enhance the product or options in any means.
• And regardless of taking a look at this metric each day, it took us two months (by the point all of the 30-day trials had completed and churned) to seek out out that there was a problem.

Whereas one a part of the issue was our lack of expertise on the best way to measure higher metrics at the moment, the issue in organizations typically lies quite a bit deeper: most groups or stakeholders are merely not prepared to listen to the reality from their metrics, so we search for the numbers that make us look good.

In my interview with Crystal Widjaja, CPO at Kumu and author for Reforge, she summarized fantastically the best way to view information as a strategy to capitalize on failures and drive enhancements as an alternative:

“When individuals make errors (failed experiment, failed deployment, and many others), you’ve already paid that value. We should always consider information as a strategy to capitalize on errors and study from them. Slightly than ‘paying the schooling of the failure’ and firing the person, use the info insights to inform us WHY it was a failure, study from it, and leverage it for the subsequent iteration to be 10x higher than the primary.”

– Crystal Widjaja

So as to get extra insights out of your information you actually need to cease monitoring self-importance metrics, and as an alternative use information to uncover the reality and drive precise enhancements. If you happen to look carefully, it’s fascinating how typically groups are exhibiting very selective metrics to appease sure stakeholders or make the numbers sound higher than they really are. Be careful for different traditional examples of self-importance metrics resembling:

• Variety of web page views or guests
• Variety of followers/likes
• Time spent on web site (session size)
• Variety of downloads

Metrics like web page views and session size are nonetheless closely utilized in web site analytics, the place the main target is to measure site visitors, consciousness and preliminary engagement. They offer you some perception on what we name the highest of the funnel—the preliminary acquisition of shoppers—however not whether or not prospects are literally activating and interesting with the product, which could have a way more significant correlation along with your wider product and enterprise targets.

The right way to do it higher: to actually perceive if a metric is nice or unhealthy, we have to put numbers into context. On the very minimal, you need to attempt to evaluate a quantity over completely different time intervals, resembling evaluating your sign-up numbers this month versus the earlier month.

One other efficient strategy to make your metrics extra helpful is to make use of ratios as an alternative of whole numbers. Ratios are inherently comparative. For instance, accountants don’t simply take a look at whole income, however usually evaluate the prices of manufacturing a product with the gross sales they produced from it. This manner accountants can observe their revenue margin (a fantastic instance of a helpful ratio) over time to evaluate whether or not the enterprise is wholesome.

Examples of higher, extra comparable metrics:

• % of sign-ups per acquisition channel
• % of sign-ups who accomplished the complete sign-up course of
• % of sign-ups who carried out a key activation metric
• % of customers utilizing the product after 4 weeks

Pitfall #2: Solely monitoring lagging metrics

A giant drawback was the period of time it took us to seek out out whether or not we have been hitting our conversion targets (or not). The product had a 30-day free trial and our purpose was to transform them to paying prospects after the tip of the trial, so whereas the primary month appeared good by way of sign-ups, we might in the end solely know by the tip of the second month what number of of these sign-ups transformed to paying prospects.

This can be a traditional instance of a lagging metric. Lagging metrics report retrospectively on previous outcomes. For instance, your income numbers for the 12 months are lagging metrics like most of your different operational metrics. You solely know whether or not you probably did properly upon getting the outcomes.

The true worth in monitoring person conduct by means of your product analytics is you could begin to search for earlier indicators than having to attend in your closing income numbers. In case your main metrics don’t carry out properly, you could have the prospect to course-correct earlier than it’s too late. For this reason I designed the Holistic Metrics One Pager in chapter 5 of my e book to incorporate each buyer conduct and operational metrics, so groups can observe a wholesome mixture of main and lagging metrics to get the complete image.

Some of the highly effective main metrics is the activation metric. A superb activation metric represents the proportion of shoppers who take a key motion of organising or beginning to use the product. Many firms have found out that if customers do a sure motion inside their product throughout onboarding, they have an inclination to appreciate the true worth of the product which results in increased engagement additional down the road. Some name this activation step reaching the “aha second” of their product.

Listed below are some easy examples of main activation metrics:

• Social community product: a traditional instance was Fb’s first activation metric of including a minimal of seven buddies in 10 days.
• Dashboard aggregation product: the worth proposition is to bundle a number of instruments into one view, so that you would possibly discover that customers who add a minimal of two or three instruments throughout onboarding understand the complete potential of the product.
• Utility product: your worth proposition could be to simplify or digitize a process resembling monitoring gross sales conversations in a CRM, so you can observe the variety of customers who full their first buyer entry as shortly as attainable as an activation metric.
• Consideration product: in case your product is centered round leisure and content material you would possibly observe customers who consumed a specific amount of content material within the first week of signing up .

Lagging metrics usually are not inherently unhealthy, by the best way. Actually, they’re a crucial a part of reporting, particularly for measuring enterprise metrics resembling your monetary outcomes. Their benefit is that they characterize the ultimate outcomes, the actual details.

Main metrics however typically embody some quantity of assumptions like the idea {that a} excessive quantity of chilly calls each day will increase the variety of paying customers additional down the road. As you get extra information it is best to take a look at whether or not these assumptions are literally true, however even then there may be nonetheless some uncertainty on whether or not the activation metric actually prompted the rise in retention, or whether or not different elements contributed to it.

This implies main metrics won’t ever be as correct as lagging metrics, however they’re essential to getting true insights out of your metrics. They permit us to study from buyer conduct and establish early indicators which will change our product selections to optimize for higher enterprise outcomes additional down the road. Utilizing the Holistic Metrics One Pager template forces you into monitoring each main and lagging indicators, in addition to to consider how these affect one another.

Pitfall #3: Metrics nobody understands

After I interview product managers I typically hear that analytics information and information insights get hidden away in darkish mysterious corners of places of work, with occasion names that nobody however a few extremely specialised analysts perceive. Each month these specialists would meet with numerous product groups in an try and share and translate a few of their findings.

If we wish our product groups and stakeholders to create a shared understanding of our information and focus on enhancements to the product collaboratively, we have to actively work on democratizing our information, be sure our metrics are accessible to everybody and straightforward to grasp.

Intercom shared their learnings from doing an enormous occasions clean-up just a few years in the past. That they had round 350 occasions for his or her product that type of appeared like this:

Does this look acquainted to you?

Intercom shared that they failed a key precept of analytics: they made little or no sense to anybody however the analytics group. They redefined and rebuilt their complete naming construction of all their occasions to introduce higher readability as a key step to democratize their product analytics information.

It’s additionally necessary to make studies extra accessible for numerous stakeholders and groups within the group. Sadly, I typically see groups afraid of opening up their dashboards, as it will once more uncover the true engagement or acquisition numbers that will not look nice to stakeholders. To keep away from uncomfortable conversations or pesky questions, it’s typically simpler for groups to cover behind a veneer of complexity.

The right way to do it higher:

• Step 1: Work along with your engineering groups and analysts to simplify your product analytics occasion names: “Accomplished Onboarding” and “Added Dashboard Widget” are actions that everybody will perceive.
• Step 2: You probably have an analytics group, embody them higher in your product groups. The extra context the analysts have of what your product group is engaged on, which experiments you’re testing and which questions have to be answered, the higher they will help you to dig into the info to seek out probably the most related insights. It needs to be a collaboration reasonably than an outsourcing method.
• Step 3: Make your analytics dashboards and studies accessible to the broader group. Your dashboards ought to mirror your product’s key metrics (which you’ll outline utilizing the Holistic Metrics One Pager from the e book). That is crucial for scale (your group doesn’t need to get flooded with handbook reporting requests each day) in addition to to really construct a extra data-driven tradition inside the wider group.

“When groups are requested in regards to the state of the enterprise, they’ll both go look it up or make up hypothetical guesses. It’s crucial to make the previous the simplest, default means for management to answer these requests by constructing customized, easy-to-use drill-down dashboards for issues like cohorts, funnels, and person occasions.”

-Crystal Widjaja

Do not forget that the job we employed our information to do is to uncover the reality so we are able to take motion and enhance our product experiences. Making your metrics straightforward to grasp and extra accessible are key steps to incorporate information insights into the day-to-day resolution making in your group. A robust product group needs to be extra motivated than ever to resolve these issues as soon as they know the place the issues lie.

The right way to enhance your metrics utilizing the metrics guidelines

I created a easy guidelines that summarizes key traits of fine high quality, actionable metrics that can enable you get extra insights out of your information. Use this guidelines to evaluate and enhance all of your current metrics:

It takes actual follow to really get your key metrics proper, and you’ll discover the satan typically lies within the particulars. It’s completely regular, and in reality inspired to continuously revisit the metrics you could have chosen, and to refine them a number of occasions to make them extra helpful.

Be careful for the pitfalls of sharing self-importance metrics, focusing an excessive amount of on lagging indicators the place you haven’t any time to course-correct, and be sure you simplify and democratize your metrics to really stage up the info maturity in your group.