The Question
What is a Google product you love? Imagine you are responsible for one of its features. How would you monitor its use, and how would you measure its success or failure?
Answer Structure
This is a question that combines product design and metrics analysis and like in many product questions the key is to identify the goal. You are being asked how would you measure the success or failure of a product feature, but before you can answer you need to know what the goal of the product feature is. Once you know what the goal is, you can start thinking of metrics that measure whether the goal is met or not.
Ask clarifying questions if you need to narrow the scope of the question, and then take a minute to structure your answer on paper. Next, tell your interviewer how you will tackle the question (based on your structure), analyze each point, and wrap-up by summarizing your main conclusions.
Here is one possible structure:
- Select a product and describe what it does.
- Pick a feature/features then explain the goal and what the feature does?
- For each feature, describe how it is used by a user.
- For each feature, what metrics are needed to measure the feature’s effectiveness relative to goal.
- For each feature, how would you implement the measurement?
- Wrap up
Answer Example
Select a product and describe what it does
INTERVIEWEE: A Google product that I use every day and like very much is Feedly. Feedly is a news aggregator that enables the user to save articles and share them across a multitude of channels including social networks, social media management platforms, email and customized channels like your own web site. Feedly has a freemium model offering a free subscription, a PRO subscription for about $6 a month, and a Team subscription. I will focus on the single user offerings, since that is what I use.
Describe the features you will analyze
The free version offers a maximum of 100 feeds, the option of saving articles locally and sharing articles on three social networks: Facebook, Twitter and Pinterest. The PRO version offers search, unlimited number of feeds, saving articles on multiple third party tools like OneNote, Instacart and Pocket, and sharing articles on the most popular social networks and via email. For this question, I will pretend I am responsible for three features:
- the articles feed,
- saving, and
- sharing features using third party tools integrations.
Explain the goal or goals of the features
The overarching goal of these three features is retention and conversion of free users to paid subscribers. The saving and sharing features of the PRO version, in particular, aim at conversion.
The Feed feature
Let’s look at the feed feature first. The feed displays articles by the minute from sources selected by the user or suggested sources. There are various actions the user can take. He can choose to display the articles in four different modes: Title Only, Magazine, Card and Full Article. If the user hovers the mouse on a Title Only line, different saving and sharing options are displayed. The user can also mark articles as read to filter them out or sort articles by age. If a user decides to read a full article, he clicks on the article and the article window expands to show a visit website link so the user can go to the original article.
The use of the layouts, filtering, and sorting features indicate a level of engagement with the application. More compact layout options for example make reading easier when there are a large number of articles, while less compact layouts are best for short feeds. To get a sense whether this feature is useful to the user I would monitor how often users change layouts and find if there is a correlation between a layout mode and size of feed. I would expect that more compact modes will be correlated with larger size feeds.
To gauge the usefulness of the filtering features, I would monitor the number of times a user clicks on the mark read button per session and track the average per user.
And, to gauge the usefulness of the sorting feature, I would monitor the number of times the sorting by age feature was turned on per user, and track the average per user.
The higher these averages are the more useful these features are to the users, and therefore the likelier it is that a user will continue using the application.
Let’s talk about the feed feature. When viewing the feed, the user only sees excerpts of the articles. Usually, if the title is interesting to the user, he would click on the title to see the first paragraph, and if the article seems really interesting, he would click deeper by clicking on the visit website link to read the full article at the source. We can say that the first click indicates curiosity while the deeper click indicates real interest. Both activities are worth measuring because they give an indication of how useful the content is to the user. Therefore, I would measure the number of partial articles a user clicks on per session, and the number of full articles that a user reads per session. The higher these numbers are, the more value the users extract from the application and the more likely it is that they return to use the application.
The Saving and Sharing features
Now, let’s look at the saving and sharing features. These features have two goals: to retain users and to convert users from free to paid subscribers. When people are doing research on the web, they usually do two things: they save content and share content. They save content on tools like Evernote, OneNote or Pocket, and they share content via social networks or email, or use social management tools like Buffer and Hootsuite to schedule the publication of articles across multiple social networks. To support these use cases, Feedly’s saving and sharing features offer integrations with the most popular third party tools. And the hypothesis is that saving and sharing of articles is so ubiquitous that free users would want to convert to paid subscribers in order to use them.
To gauge how critical these features are, I would measure the frequency and the breadth of use of these features with the following metrics:
- average number of articles saved per user per month broken down by third party tool
- average number of articles shared per user per month broken down by third party social platform and email
- month-to-month growth rates of the average number of saved and shared articles
- For each third party saving and sharing tool, how many times the user uses each per month
The average numbers of saved and shared articles and month-to-month growth rates measure usage per month and whether usage is increasing or not. And, the frequency of use of third party tools measures breadth of use, which indicates if users find most of the third party integrations useful or not. In addition, to evaluate whether these third party tools features are critical for driving conversion to paid subscriptions, I would measure the correlation between frequency of use of these features and the number of paid subscribers and compare with the correlation of another feature and paid subscriptions. Search, for example, is another feature of paid subscriptions. If Search were the main driver of conversion instead of the save and share features, we would expect the save and share feature correlations to be significantly lower.
In summary, as a PM of the the feed, saving and sharing features of Feedly, I would evaluate the success of these features based on whether they drive retention and conversion to paid subscription. I proposed several metrics to evaluate whether these features drive retention and conversion. For the feed feature, I proposed metrics that measure usage of the feed’s layout, filtering and sorting features and level of interest in reading the articles. The higher the value of these metrics, the higher the likelihood that users would re-use the application. For the saving and sharing features, I proposed metrics to measure frequency and breadth of use of third party tools integrations. The higher these numbers the higher the likelihood that users find these features useful. And finally, I proposed correlating the frequency of use of these features to the number of paid subscribers, to evaluate whether they are indeed strong drivers of conversions to paid subscriptions.