StellarPeers is a community platform that helps professionals prepare for interviews. We think the best way to prepare, is to work through questions and practice mock interviews as much as possible. We meet weekly to discuss product management interview questions on product design, product launch, strategy, marketing, pricing, and others. Last week, we worked on a product metrics question.
The key to a product metric question is to start by defining what success means by identifying what the business goals are for the product. Once you have defined the objectives, think about what outcomes should happen in each stage of the customer journey for these objectives to be achieved. The journey towards becoming a monetizable customer usually involves these stages: awareness, acquisition, conversion, engagement, retention, and monetization. Think of measurable outcomes for each stage that would give an indication of the success or failure of the business goals.
Here is a suggested structure would be:
- Define the business goals.
- For each phase of the customer journey such as awareness, acquisition, conversion, engagement, retention and monetization, think of ways to measure customer actions that would indicate success or failure of the business goals. List each action as a metric.
- Wrap up. Summarize what the business goals and the most relevant metrics are.
INTERVIEWEE: I would like to start by defining what success is, which depends on what the business goals are. And then, follow with metrics to measure the achievement of those goals.
The business goals should include the target buyers, which are consumer households, but also the developers of Alexa Skills, which are essential to providing a variety of service options to consumers.
My belief is that Amazon’s primary goal is to grow revenue of consumer purchases from Amazon.com. While a secondary goal is to grow revenue for Amazon Web Services (AWS) by growing the number of developers building Alexa Skills.
INTERVIEWER: Yes, generally I would agree with that.
INTERVIEWEE: Ideally, the revenue from purchases through Echo should be additional revenue to that generated from selling products through the Amazon website. The way this additional revenue is likely to be generated is through:
- Frequent purchases that are additional to purchases through the web site. Since the Amazon Echo is on 24×7, it is easy to see how people may find it more convenient, faster and fun to order products by voice rather than logging into the Amazon website.
- Increasing the number and usage of Alexa Skills, with a view to monetize Skills’ transactions via a revenue-sharing model between Amazon and developers.
- Increasing the number of subscriptions to Amazon Music Unlimited. Just last year, Amazon released its Music Unlimited subscription service to compete in the music streaming market. Amazon is hoping that Echo and its family of speaker devices, along with low subscription prices, will incentivize consumers to sign up for its music service.
- And, increasing in the number of Amazon Prime subscribers. Furthermore, since Amazon Prime subscribers get a slightly better price than Non-Prime subscribers, there is also an incentive for Non-Prime users to convert, which for Amazon would be a fantastic outcome.
In order to achieve these goals, consumers have to be aware of the product, buy it, use it, and continue to use it. In other words, they go through the stages of awareness, conversion/purchase, engagement and retention ending in a final monetization stage.
Let’s walk through these customer journey stages to understand what kinds of interactions are involved and how to measure the success of those interactions.
At this stage of the customer journey, the customer is learning about Amazon Echo through marketing campaigns. A good campaign should make customers curious about the product. We should see an increase across the following activities: research about the product, word-of-mouth activity, visits to the Echo website, click-through rates, blogs views, etc.
To measure the success of awareness campaigns, I would use metrics such as:
- The number of impressions per campaign across different ad types like display ads, search ads, and print ads.
- The reach of content marketing campaigns.
- An increase in the number of keyword searches after and during a campaign.
- The number of new visitors to the Echo website during and after a campaign.
- The number of repeated visitors to the Echo website during and after a campaign.
- The number of referred visitors due to word of mouth or social virality during and after a campaign.
Conversion (Purchase of Echo)
At this stage, customers are buying the Amazon Echo. Some of them will become Prime subscribers and some will not. To measure the success in the conversion of Non-Prime to Prime subscribers, it will be important at this stage to keep track of who was originally not a Prime subscriber. These are the metrics I think will be relevant at this stage:
- The number of Echo and Dot products sold per month. Is it increasing?
- The revenue from sales of Echo and Dot products per month. Is it increasing?
- How many owners are Prime and Non-Prime subscribers?
At this stage, the customer is using Echo and we should be measuring usage activity related to our four goals: 1) whether purchases through Echo are additional to the site; 2) whether the usage of Skills is increasing; 3) whether the number of subscriptions to Amazon Music Unlimited is increasing; 4) and whether Non-Prime customers are converting to Prime.
To facilitate measuring changes in usage, purchases, and prime conversions as time goes by, we will need to segment customers into cohorts. By putting customers into cohorts based on on when they bought an Echo, we will be able to monitor changes in behavior as customers tenure increases.
Here are metrics I think we should use to measure success at this stage:
On conversion to prime subscriptions
Percentage of Non-Prime customers in a cohort that converted to Prime subscribers per month. This will indicate whether Echo has been successful in increasing Prime memberships.
The fraction of customers in a cohort that make at least one purchase per month; this metric will indicate how frequently customers continue to use Echo to purchase products after they bought Echo.
The fraction of the same customers in a cohort that made at least one purchase every month since they bought Echo. This will indicate which fraction of the cohorts are hard-core Echo users, the more the better. Another metric I would be interested in is what their average monthly dollar purchase is? This will indicate how habitual Echo is becoming as a channel for purchases, and if people are spending more per purchase on this new channel.
On additional purchases through Echo
What is the number of monthly purchases per household through Echo compared to monthly purchases per household through the Amazon website? Are the number of purchases through the website decreasing but increasing using Echo? This will tell us if Echo is being used as a channel for additional purchases or if Echo is replacing the Website as a channel.
Are people buying new products using Echo or buying similar products that are purchased through the website? This will tell us if Echo is incentivizing people to buy things they would otherwise not have bought through the website. Does Echo enable more impulsive buys?
- How many Skills per month are Echo customers using?
- What are the top 10 skills per month by frequency of use?
- What are the top 10 skills per month by dollar amount?
On Amazon Music Unlimited
How many Echo buyers that were not existing music subscribers, subscribed to the Amazon Music Unlimited service after they bought an Echo? This will confirm whether Echo has been a trigger to get consumers to sign up for the new music service.
How many hours per day are Echo users listening to music? Is the average increasing or decreasing? This will indicate whether people are valuing the service.
At this stage, we are interested in knowing whether customers are likely to continue using Echo or churn. It is also important to know the interval of time between consecutive usage of Echo. The shorter the time the better for retention.
Metrics to measure retention:
- From day to day, what fraction of the same users in a cohort use Echo?
- From month to month, what fraction of the same users in a cohort use Echo at least once?
- From month to month, what fraction of the same users in a cohort use Echo to make at least one purchase?
- From month to month, what fraction of the same users in a cohort use at least one skill?
- Per month, how many days does it take on average for the a user in a cohort to reuse Echo a second time?
- Per month, how many days does it take on average for a user in a cohort to make an Amazon purchase using Echo since the beginning of the month?
- Per month, how many days does it take on average for a user in a cohort to reuse Echo to listen to music?
- How many Echoes are returned per month? Is it increasing or decreasing?
Finally, we can quantify success in achieving the four goals: additional purchases, usage of Skills, subscriptions to Amazon Music Unlimited and Amazon Prime conversions with several metrics.
Metrics to measure monetization:
- Is the revenue from Amazon purchases through Echo decreasing or increasing?
- How much of the revenue from Amazon purchases through Echo is additional revenue or replacement revenue for purchases previously made through the Amazon website?
- If there was already a shared-revenue model in place, how much revenue was generated from Skills’ transactions?
- How much revenue from Amazon Music Unlimited subscriptions is from Echo owners?
- How much revenue is from new Amazon Prime subscriptions?
And, let’s not forget developers. One important objective should be to attract more developers in order to increase the number and variety of Skills offered. To measure success in this objective:
- What is the number of developers and monthly growth?
- What is the average number of Skills per developer? The more Skills developed by a single developer the more committed they are to the Echo platform.
- How much revenue was generated from hosting services and providing computing resources to Skills developers through Amazon Web Services (AWS)? An increase in revenue per developer indicates higher developer commitment.
In summary, I defined business goals with respect to selling Echo to consumers and developers. With respect to consumers, I talked about four business goals: 1) increasing additional purchases from Amazon using Echo; 2) increasing the number and usage of Skills; 3) increasing the number of subscriptions to Amazon Music Unlimited; and 4) increasing the number of Amazon Prime subscribers.
With respect to developers, I think the goals should be to increase the number and variety of Skills offered. By looking at each stage of the customer journey: awareness, conversion, engagement, retention and monetization, I defined different metrics that would help measure success towards these business goals. If I were to choose and prioritize the most important metrics to support, it would be the engagement, retention and monetization metrics, because they are direct indicators of revenue creation, which is our bottom line. In particular, I would prioritize these metrics related to increasing purchases and increasing subscriptions to Amazon Music Unlimited and Amazon Prime subscriptions, which are key revenue generating activities.
And as a final thought, as part of measuring the success of Amazon Echo, I would include device sales from competitive products like Google Home. Although device sales does not reveal insights into customer behavior, it provides a high level measure of competitiveness.