How to Ship a Product and Focus on the Right Priorities

Over the years, working with talented engineering managers and product managers has helped me distill down a few rules of thumb about how to ship builds and find 10x improvements. Whether it’s building an internal tool or consumer facing feature, here are a few tips I learned the hard way after many trials and errors.

Start with a strong vision or missional statement

A vision for where your team could ultimately produce, however far distant that vision might be, is an very important exercise that should not be skipped. Making something great takes a whole team of talented people singularly focused on the task for a long time. So at a high level, a great inspiring vision can greatly help inspire the team to do their best work and get buy-ins from other teams.

In practice, coming up with the mission statement pushes you to ask many necessary questions.

1. Stakeholder needs: To the stakeholders, what is the job to be done for them. What is the ideal scenario?
2. Believability: How much of the ideal is pure fantasy vs inspiration that one can reasonably believe in?
3. Scope: How broad is the scope of this vision? Too broad of a vision could potentially derail your focus and produce something that does a lot of things, but none great.
4. Ecosystem benefits: Last but not least, how much of that ideal aligns with the ecosystem that the company is trying to deliver?

Answering these questions is harder than it looks but the end result is that your team will have a measuring stick to know if you are spending time on the right passion.

Understanding the job-to-be-done

There is a saying that people don’t want to buy a quarter inch drill, they want a hole in the wall. In everything I built, whether it be an internal tool or a consumer facing funnel, I found that understanding the actual problem to be solved can mean the difference between spending a quarter on an overkill feature vs. a simple 5-story-point feature that effectively solves the same problem. Given a team of talented EMs and SWEs, you can solve almost anything but trouble comes if you are spent weeks solving the wrong solution. You’ll stretch the team thin trying to have the cake and eat it too, without shipping a product that is best in the market. I always tell myself to checkoff this list to make sure I am thinking corectly about the problem space:

1. Keep asking whys: Ask users of the feature the 5 whys — i.e. start with why they want a feature and then ask a series of whys to their answers to gather a complete picture of the underlying motivation.
2. A Measure for PMF: Ask the question how sad would the user be if you took away this feature. Studies have shown that if 40% of users you ask answer with “very disappointed,” there is significant product market fit and likelihood of viral adoption. This will help you understand what is a nice to have vs what will solve a pain point.
3. Don’t jump to solutioning: Ask your engineers and designers what is the best. Empower them to do what they are hired to do. They will surprise you.

Be metrics driven and create an operating model

There are three different types of metrics to keep track of, four if you are in ML. I divide them into two buckets

Online metrics: These are the metrics that are directly tied to the business outcomes.

• Leading metrics and North star — They are the funnel metrics that lead to a north star metric. They are often the KPIs in your OKRs.
• Counter metrics — these are metrics you have to monitor to make sure that by optimizing the North Star metric, you don’t make other parts of the ecosystem worse. They often also capture scenarios where things can be gamed. For example, if your north star metric is amount of screen time per day, a counter metric might be # sessions that had an active engagement within 30 mins in case people just left the site open. Or it could be number of core pages/profiles visited per day, just in case people are spending time on parts of the app that the company want to deprioritize.
• Guardrail metrics — these are metrics help you sanity check against the lead metrics in an A/B test. For example, if a test should

Offline metrics: This is very important for ML products. These are metrics that measure the goodness of ML models at their predictions. Precision, recall, F1, AUC, ROC and a confusion matrix are often monitored.

Forecast your online metrics

Not everyone likes excel spreadsheets but I find that having an operating model (i.e. a means of forecasting metrics) is essential to knowing the right priorities.

Without objective metric forecasting, prioritization exercises tend to be based on gut-feelings and a merry-go-round of my-word-against-your-word deadlocks.

Structure the model such that the columns are months in a year and the rows are all of the metrics that are important to the project. These metrics may be online metrics (e.g. metrics belonging to an UX funnel) or offline metrics (e.g. number of labelers, number of unlabeled samples and model F1 scores). The idea is that you want to put down variables that are intertwined such that tweaking one metric impacts another. They should all add up or multiple together to produce the north star metric at the bottom — the last row is the OKR result you are trying to hit in six months (e.g. revenue, CAC, AUC, perfect labels delivered etc). The metrics above should multiple together to impact this bottomline metric.

Once you set up this operating model,

1. 10x vs 1.2x: When you have this operating model, you can play around with the numbers and see which levers move the needle the most and de-risk everything else. This is true especially if you were to put down an almost impossible upper bound on a metric that you thought was important and notice that it only 1.2x the bottomline down the road. Oppositely, the operating model will show you where the 10x opportunities are and guide you on hard KPIs for next quarters’ OKRs.
2. Robust reporting structure: An operating model requires many metrics updated in real time. So it forces the team leader to be data centric as well as provides visibility across the organization.
3. Objective metrics: In a meeting where buy-ins are needed and different teammates have different arguments about priority, it is a very potent argument to be able to show that maxing out a metric will not move the needle much. It is very objective and hard to ignore.

Be technical

I did production level code for years and so I may be biased here but being technical has helped me a number of times to ship faster and better. I wasn’t the smartest engineer compared to our ML engineers but being able to understand the technical difficulties of a feature/pipeline/model and how to estimate its impact vs effort tradeoff will help your team avoid derailment to the wrong focus or help your team jump on the right priorities. As a team leader, your role is ultimately not to come up with the solution. In fact, empower the technologists to do what they do best but being technical will help you guide the discussion and avoid spreading your team thin chasing after difficult solutions that don’t move the needle much. If you enjoy being on the technical side, I recommend:

1. Weekly publication reviews: Set up a weekly publication review or latest tech review with the engineers where they can discuss one new research publication, release, or API that may be relevant to your project.
2. Help your ML scientists prioritize: As the team leader, you know the rationale behind certain OKRs that are motivated by things outside of technical feasibility (e.g. business case, client needs, revenue potential, etc). These inputs are often outside the scope of an engineer’s responsibility but very helpful input while deciding which solution to pursue. In fact, challenge your EMs on the paths they decided to go on. Particularly focus on alternative paths that may be blindspots. Sometimes, not knowing the conventional wisdom on a problem may be a good thing.

Get buy-in and resolve conflicts for your team

Everyone has a different objective and different priorities. Getting alignment and resolving conflicts are often the bottlenecks in shipping.

It doesn’t matter what the title is. Leaders, the higher the the title, the more they realize that persuading different teams toward a vision is better than strong arming because, over time, top-down leadership styles tend to introduce blindspots and less motivated teams.

What I found the most effective has been to tailor the message to the audience I am trying to convince while staying on message. The way to communicate to engineers, who may care about technical feasibility, is very different from the way to communicate to the operational team who care about manual workload. They may not realize that they are tradeoffs that could be made that will satisfy both camps. So centering the conversation around gains to their objective and lessening to their concerns will make a difference.

Conclusion

Shipping a product on time and consistently yielding better results are an art than a science. Here I highlight a few of my must haves — strong vision, knowing job-to-be-done, being metrics driven, being technical, and getting buy-ins. If you like what you read, let me know in the comments what has made a difference for you and your team!