How Are You Being Evaluated
This product metrics interview question tests whether you understand what the purpose of using metrics is and how to define them.
The interviewer is evaluating you on the following:
- Did you begin by stating what the goals of the feature are before jumping into defining metrics that evaluate whether the feature is meeting those goals?
- Is your answer structured or do you tend to talk about random points without a coherent thread?
- Are you precise in the metrics definitions or do you tend to give a general description such as, “I would find out if people used the feature frequently.”
Answer Structure
- Explain what the goal(s) of the feature is(are).
- Explain what the feature does.
- If the feature uses several subcomponents, which metrics would you define in order to measure the effectiveness of the component in achieving the goal(s)?
- Summarize your analysis and key insights.
Answer Example
INTERVIEWEE: I would like to start by talking about the goals of Facebook Stories, how this feature works and which metrics to use to measure if it is meeting those goals.
INTERVIEWER: Sounds good.
INTERVIEWEE: I think the overall goal of the Stories feature is to increase engagement and retention of Facebook users. Similar to Snapchat Stories, the new Facebook feature allows users to share a disappearing story. The hope is that people will find this new format of sharing more fun and engaging, to the extent that people create and share more user-generated content and open the app more frequently. An added bonus for Facebook Stories would be to attract Snapchat’s segment of 30-year-olds and younger and the 40+ segment that Snapchat does not focus on.
Before talking about metrics, I want to make sure I understand how Facebook Stories works. My understanding is that Facebook Stories only works on mobile devices and can be shared in three ways: publicly as a touch-enabled circle above the Facebook News Feed; as a Direct Message to a friend; and as a regular post to the user’s feed. And, a story disappears after 24 hours when shared publicly or when sent directly to a friend.
INTERVIEWER: That is correct.
INTERVIEWEE: In addition, Stories from friends that the user interacts with more frequently are prioritized and displayed before stories from other friends.
INTERVIEWER: That is right.
INTERVIEWEE: Also, the Facebook Camera is an integral part of the Stories feature. The Facebook Camera has features like filters, stickers, captions, geofilters, color brushes and more that people can use to enhance their photos or videos.
INTERVIEWER: Yes.
INTERVIEWEE: Okay, thanks. Now, I will talk about metrics to evaluate whether these features, Facebook Camera and Facebook Stories, are being effective in increase engagement and retention.
INTERVIEWER: Okay.
INTERVIEWEE: Let’s start with the Camera.
- The filters, stickers, captions and all other widgets in the Camera enable users to make their videos or photos more fun and to induce them to share with friends. So, the question is whether these widgets are indeed doing that? So, I would measure the fraction of the Stories that a user embellishes with the widgets on average.
- Knowing which filters are most popular can help guide future filter designs, so I would measure the top 10 most frequently used filters per user. I would also use the same metric for the other widgets.
- Geofilters are very popular in Snapchat, so it would be good to know if geofilters for Facebook Stories are equally popular.
- Are people finding filters they want to use? If not, is that demotivating them from creating and sharing a story? To evaluate this situation, I would measure the average number of times people browse filters but do not send a video/photo after browsing.
- Is the Camera easy to use? One reason Snapchat doesn’t have more users ages 40 and older is because the Snapchat UI is not intuitive to use, with its hidden gestures. The Facebook Stories feature should be easier to use than Snapchat if Facebook wants to win the 40+ segment, therefore I would check if users are having difficulty using the Stories feature. I would measure on average how many steps a user takes to create and share their story. And, how many steps they took before desisting from sharing.
- Is a max length of 40 seconds used frequently for video? The Facebook Camera allows a longer recording time than Snapchat because the hypothesis is that people want to have longer recordings. So, I would measure the average length of videos shared per user and check if the majority of users tend to use 20+ seconds.
Let’s move on to metrics for the Stories feature.
- To see if the Stories feature is popular, we’ll need to measure how frequently people are using the feature. Useful metrics are: the average number of times a user shares a Story per day, per week, and per month.
- Is the Stories feature causing people to post more? My understanding is that there has been a progressive decline in the number of personal posts on the News Feed throughout the years. So to see if the Facebook Stories feature is being successful in reversing this, I would measure how frequently users are posting a Facebook Story to their news feeds. I would segment users into users that never post, users that post seldom and those that post frequently. If those that never post to their news feed or post seldom, are posting Facebook Stories to their news feeds, that would be a successful indicator that they are more engaged in posting.
- If users are using all three channels: Story, Direct Message and Post for sharing the same stories, that would be an even bigger win for engagement, since a story would be shared multiple times. So, I would measure on average of how many channels a user shares a Story with. And, is this number increasing or decreasing?
- To test whether users want to first view Stories from close friends (friends that they interact with more frequently) or whether they want to see stories from all their friends, I would measure how many times a user clicks on Stories from friends they don’t typically engage with before they click on stories from their close friends. If people click on Stories randomly without prioritizing stories from close friends then a more appropriate ordering of Stories should be thought out.
- If the Stories feature is successful in increasing engagement, the expectation is that people that click on a Story, may respond with their own Stories. So, I would measure what fraction of the the people that click on a friend’s Story, publish their own Story shortly after.
To summarize, I analyzed the different ways in which the components of the Facebook Camera and Stories features work. I also defined metrics to monitor, measure and evaluate whether those components are achieving the overall goals of increasing engagement and retention. Other insights from these metrics would be whether Facebook Stories is encouraging users that don’t post regularly to start posting more and whether the younger and older users are equally fond of creating Facebook Stories. This would be a win over Snapchat, which mostly attracts the younger segment.