How Are You Being Evaluated
This product metrics interview question tests whether your process of defining metrics is sound and structured. Do you tie metrics to goals? Are the metrics you specified actionable or lead to key insights? Do you prioritize which metrics to implement?
The interviewer is evaluating you on the following:
- Did you begin by stating what the product goal is before talking about metrics? Talking about metrics without knowing the product goal is a red flag.
- Is your answer structured or do you ramble?
- Do you provide precise metric definitions or do you tend to give a general description such as, “I would find out if people used the feature frequently.”
We suggest structuring your answer in the following way:
- Ask clarifying questions to confirm your assumptions (e.g., What does the product do? Who is the product for?).
- State or verify with the interviewer what the goal of the feature is (e.g., to increase conversion, revenue, profits, sales, engagement, retention, or other?).
- Apply the product’s user experience flow to drive your analysis of what to measure. For example, think of what to measure when the user opens the app or opens a feature and interacts with it.
- At each stage, identify behaviours that can be quantified with a metric(s).
- Provide criteria to prioritize the metrics.
- Summarize your analysis.
INTERVIEWEE: Okay, first I want to confirm that my understanding of Snap Map is correct. Snap Map is a location-based feature that is part of Snapchat, is it not?
INTERVIEWEE: And it enables users to share their location and Stories associated with those places with friends.
INTERVIEWEE: I think it works in the following way: a user opens the Snap Map feature by pinching on the Snapchat camera screen. A scrollable map appears displaying Bitmojis of friends around the world. The user can tap a friend’s Bitmoji to send a direct message or view any Stories the friend may have shared. Alternatively, the user can create a Story too.
INTERVIEWER: Yes, that is correct. And, in addition to friends’ stories, the map can also display stories featured by Snap. For example, news or events happening real time, like BART being on strike in San Francisco or a summer music festival nearby.
INTERVIEWEE: I see. So overall, it looks like this feature is a way to visualize Stories in a more lively way.
INTERVIEWEE: In light of the increased competition from Instagram and Messenger Story features, this seems to be a step in the right direction. My guess is that the goal of the Snap Map feature is to increase user engagement?
INTERVIEWEE: Got it. So what I would like to do next is to talk about the user experience flow. The flow will help me identify which user behaviors are indicative of user engagement, and think of metrics to quantify these behaviors. The metrics will help monitor different forms of user engagement and evaluate if Snap Map is successful in increasing user engagement. Do you agree with this approach?
INTERVIEWER: Sounds reasonable, go ahead.
INTERVIEWEE: The user experience flow for Snap Map follows this process:
(The interviewee draws the following diagram.)
I will go through each stage of the flow and talk about behaviors that arise. I will score each behavior on whether it represents a strong indicator of user engagement from 1 to 5. The score will help determine which metric to prioritize for implementation.
Leaving Snap Map
So let’s start with the user opening Snapchat and then pinching to open the Snap Map feature. One thing that can happen is that the user leaves without trying the feature. Not trying the feature could mean that the user did not understand how to use the feature, or they were curious and opened it but were not interested in doing anything more. Regular Snapchat users are savvy users, so if they opened Snap Map then immediately closed it is most likely because they were not interested.
If the user immediately closes the map, one reason could be that they didn’t see any Bitmojis on the Snap Map. To test if this is accurate, I would measure the following:
- The fraction of times a user opened and left Snap Map per day and the average of this metric over all users. A decrease in this metric after launch would be a sign of higher engagement.
- The average number of stories a user sees when they open the Snap Map feature. If the feature is successful in engaging users, this number should increase after launch and not be too small.
Knowing whether the user is not interacting with the feature is very important, so I would give these metrics a priority score of 5.
Tapping on Bitmoji
If the user doesn’t leave Snap Map after opening it, one thing they may do next is tap on a Bitmoji. I would measure this behavior by measuring the number of times a user taps a Bitmoji over the number of Snap Map sessions, and average them. An increasing average would indicate higher engagement. At this point, the user may or may not view a Story if there is one associated with the Bitmoji. Tapping on a Bitmoji is a behavior that shows a light level of engagement. We can call it the ‘curious’ level, and I would give it a score of 2.
Tapping on Bitmoji + view Story
Now say a user is more curious about a friend’s activity and taps on a Bitmoji. At that point the user can decide to view the friend’s Story or leave. If the user views a Story, that shows strong engagement, and it would be important to track that. So I would measure that behavior by calculating the fraction of the number of Stories viewed per person and also calculate an average over all users. I call this behavior ‘curious +’ and would prioritize these metrics with a score of 3, higher score than the previous metrics.
The user can prevent their location from being visible to anyone else on the map by turning Ghost Mode on. And if a user has Ghost Mode on all the time, that would be a strong indication that they probably don’t want to be tracked and don’t care about using Snap Map. I would measure this behavior by calculating the fraction of users that have Ghost Mode on per day, per week, and per month. If the Snap Map feature is successful in engaging users, this fraction should go down after the launch of Snap Map. This fraction metric is informative but is not a very strong indicator of user engagement. Because even if the user has Ghost Mode off, which means their location is being tracked, this does not mean that they are actively sharing Stories. They may have just forgotten to turn Ghost Mode on. So I would score this metric with low priority 1.
The user can add a Story to Snap Map and share it with friends, which is an indication of a very pro-active user engagement behavior. I would measure this behavior by tracking the number of Stories created on Snap Map per user per day, and per month. If this metric increases after the launch of the feature, it would indicate that user engagement is increasing.
Users can also create Stories in the traditional way, outside Snap Map. And if the Snap Map feature is indeed successful then the sum of Stories created daily via Snap Map and outside Snap Map should be higher than before the launch of the feature.
I would give these metrics the highest priority score of 5 because they measure behavior that indicates strong user engagement.
Monthly Active Users
Market analysts like to use Monthly Active User (MAU) as a way to evaluate and compare competitive products, so I would define and measure this metric too. I would define MAU as the number of different users that did any of the following in a given month: tapped a Bitmoji, viewed a Story, or added a Story to Snap Map. Since this is a standard metric referenced by analysts, I would give this metric a score of 5.
To summarize, the goal of the Snap Map feature is to increase user engagement. I proposed several metrics to track and monitor the achievement of the goal after launch. By analyzing the user experience flow, I identified user behaviors that point to user engagement. I also defined metrics to quantify those behaviors and prioritized these metrics based on their strength as indicators of user engagement. Based on my analysis, I recommend prioritizing those metrics related to these behaviors: leaving Snap Map without interactions, tapping on a Bitmoji + viewing a Story, and creating a Story. And as a metric to share with analysts, I would also keep track of MAU.