How Are You Being Evaluated
This product metrics interview question tests whether you understand the process of defining metrics.
The interviewer is evaluating you on the following:
- Do you talk about feature goals before jumping into defining metrics?
- Is your answer structured or do you tend to talk about random points?
- Do you define metrics precisely or give a general description such as, “I would find out if people used the feature frequently.”
We suggest structuring your answer in the following way:
- Ask clarifying questions to confirm your assumptions (e.g., What does the product do?, Who is the product for?, etc.).
- State or confirm with the interviewer what the goal of the product is (e.g., increasing conversion, revenue, profits, sales, engagement, retention, etc.).
- Use the product’s customer lifecycle to drive your analysis of what to measure. For example, think of what to measure at the awareness, engagement, retention and monetization stages.
- At each stage, identify behaviours that can be defined and quantified with a metric(s). Be detailed and ready to answer how you would gather the data to implement the metric(s).
- Summarize your analysis.
INTERVIEWEE: Before we begin, I would like to confirm how this feature works. My understanding is that the Lyft Shuttle feature targets commuters and operates during commuting hours, from 6:30 am to 10 am, and 4 pm to 8 pm.
INTERVIEWER: That is correct.
INTERVIEWEE: Also, the Shuttle service is not operated by Lyft but by independent drivers who use their own vehicles. Is this right?
INTERVIEWER: Yes. Moreover, the user must activate the feature in the mobile app, to display the closest pickup locations.
INTERVIEWEE: Are the times for pick up fixed or is it an ad hoc system like the current Lyft Line option?
INTERVIEWER: The pickup times are an extension of the current Lyft Line, so the Shuttle feature works similarly. The difference is that Lyft Shuttle runs on fixed routes and has particular pickup and drop-off locations. When a user activates the Lyft Shuttle feature, the routes closest to the user will be displayed. The user can select any of these routes and request a Shuttle. But, instead of a door-to-door service, the user must go to a fixed station along the route to be picked up.
INTERVIEWEE: So, this feature is really targeting commuters that live within a city and need to commute short distances within the city.
INTERVIEWER: Yes, commuters are the target users.
INTERVIEWEE: How about cannibalization of the door-to-door Lyft service, is this an issue?
INTERVIEWER: We don’t think so. This Shuttle feature is an extension to Lyft Line, so most likely the user would be someone that wants to save money and doesn’t mind sharing a ride. If anything, some cannibalization may come from Lyft Liners that are willing to walk and wait at a pickup station. But, we are hoping to gain users that currently take public transportation.
INTERVIEWEE: Got it. So, is the goal of the Lyft Shuttle feature to increase revenue by expanding its user base to commuters?
INTERVIEWEE: I understand. Now, to increase revenue through this new feature, several things need to happen. First, users need to be aware this feature exists; second, users need to have a pleasant experience while using the service; third, users need to continue using the service; and lastly users need to recommend Lyft Shuttle to their friends. To put a structure to my analysis, I will split it into four phases: awareness, usage, retention and recommendation.
INTERVIEWER: Sounds good.
INTERVIEWEE: Since one objective is to expand the user base with commuters who have never used Lyft, I would track the number and percentage of Lyft Shuttle users that are new registered Lyft users. If this percentage becomes large in time, that is a good sign.
Okay, so let’s start with the awareness stage.
If users are not aware of the new Shuttle feature, they will not activate it. So, I would measure activations since launch and its growth rate, and compare these numbers with target benchmarks.
We should also check that the feature is working correctly. For example, are the routes and Shuttle locations appearing on the app after the feature is activated? To check for this, I would instrument the app such that it sends ping signals to the backend to confirm that the Shuttle routes are displayed when they should be. I would monitor that the pings are received at the appropriate intervals after users, in proximity to the routes, open the app.
Now let’s move to the usage stage.
A good way to measure the frequency of use is with daily active users (DAU). The higher, the better.
For this feature to work, there need to be drivers available to meet the demand of commuters for the service. I would measure the average number of available drivers per request per route. And, I would be looking for a good balance such that the number of drivers and user requests should not exceed the other by too much.
Long waiting times are bad for user experience. So, I would measure the waiting time for the user from the moment the user arrives at the pickup station.
Cancellations are not good for drivers, so I would measure the number of cancellations per route per day. And, monitor changes throughout the commuting times.
If the user cancelled a request, I would be interested in knowing the reason. Was it because the driver was late? Or something else? So, for every cancellation that was sent by a user after he arrived at a station, I would measure how much longer the user would have had to wait for the driver. If the driver would have arrived much later than expected, this could be the reason a user cancelled. Then I would correlate cancellations to delays and see if the correlation is high. Naturally, the best way to find out the reason for a cancellation is to ask the user directly. But generally, most users don’t bother to provide feedback.
Keeping drivers happy is hugely important, and I would imagine that for them a ride at full capacity is better than a ride with just one passenger. So, I would also measure how full the rides are by measuring the average number of riders divided by the car’s capacity per route per day, and monitor changes. If rides are not full for some routes, then it would be better to drop them.
Moving on to the retention phase.
One important indicator that users are being retained is whether they use the service repeatedly. To monitor this, I would measure the number of days until the user repeats a request for a shuttle for the different routes. The shorter the number of days, the better.
It would also be interesting to monitor changes in behaviours of existing users of the regular Lyft line. For example, if occasional users of Lyft Line suddenly start using the Lyft Shuttle service, this is an indication that the feature is converting occasional users into frequent users. To monitor this, I would measure daily active use per user and have the system detect if there is a noticeable change in choice of service from regular Lyft to Lyft Shuttle. And count those users that shifted as conversions.
In terms of cannibalization, I would monitor if users of the regular Lyft service are choosing Lyft Shuttle instead. Again, I would measure daily use per regular Lyft user and have our analytical system detect this change in preference.
And finally, recommendations or referrals are hugely important for growing the funnel. To promote and measure virality of referrals, I would create promo codes that users can send to friends to try Lyft Shuttle. With the promo codes in place, we could track how many friends received promo codes for the Lyft Shuttle feature, and what fraction of those friends try the Shuttle feature. The higher these numbers, the better.
To summarize, I identified different user behaviours at each stage of the customer lifecycle that could be indicators of the success or failure of the Lyft Shuttle feature. I proposed metrics that quantified these behaviours in addition to metrics for measuring virality of referrals which is key to growing the funnel. If I were to prioritize which metrics to implement first, I would start with the ones that measure awareness and daily usage. Since the feature needs to be activated by the user, it is crucial that users be aware of it. And since the target users are commuters that most likely take the same route every day, monitoring daily use is necessary to confirm whether indeed they are using the Lyft Shuttle feature to commute. The other metrics provide additional insights that can help improve the feature and help with product roadmap decision making.