The "Joel Test" for Product Managers

The product management profession has been around for a long time, but it’s still amazing to me how different the day-to-day job actually is from company to company. It wasn’t long ago that software engineering was on the same boat, so Joel Spolsky, of Microsoft/Fog Creek/StackOverflow fame, wrote the “Joel Test” outlining the best practices at the time. This idea gave engineers a common benchmark beyond their company walls.

What would a common benchmark for Product Managers look like today? It may be something like this:

1) Do you have an evidence-based backlog of problems worth solving?

2) Do you spend at least 25% of your time "outside of the building" talking to customers/end users?

3) Do you quickly validate new product concepts with customers early and often?

4) Are you the "mini-CEO" of your product?

5) Do people trust your decisions?

6) Can every single engineer/designer/marketer/etc in your team explain why she’s doing this work and why it matters?

7) Does your product team show incremental work to stakeholders every few weeks?

8) Do you understand how your product initiative and others fit an overall company strategy?

9) Do you have a clear product vision articulating what problems will be solved when you deliver it?

10) Do you have a roadmap explaining the milestones on the way to that vision?

1) Do you have an evidence-based backlog of problems worth solving?

If we reduce the process of innovation down to 3 questions, it goes something like this:

1) What problems should we solve?

2) How do we solve them?

3) Did our solution solve the problem?

If you agree with this sequence, then step 1 is the most important to get right. There’s pretty much nothing worse in the innovation world than to solve a problem not worth solving. Paradoxically, it is a well known phenomenon that most projects end up building something nobody wants. This sad, but frequent, result is squarely the Product Manager’s (PM) fault.

Have you ever been in a meeting where, after some polite deliberation, the highest paid person says what we should do and everyone immediately agrees? Well, that’s one way PMs stop listening to customers. Add a fair bit of ego and office politics into the mix and voila! You now have a faith-based backlog of problems you (and everyone else in the meeting) think are worth solving.

Great PMs will only answer Yes on this question if the majority of the items in their product backlog can be traced to a pattern observed in customer conversations and/or in quantitative research. If your boss/work buddy/squeaky wheel colleague told you, it doesn’t count as evidence! This leads us to the next point...

2) Do you spend at least 25% of your time "outside of the building" talking to customers/end users?

Great PMs balance their time between customers, product team and internal stakeholders intelligently. If there is a time crunch, customer time usually takes the hit. Why? Because it’s much easier and convenient to talk to people who are right there in the office and actually want to talk to you. Great PMs don’t fall prey to this trap and make customer development the core value add they bring to the table.

3) Do you quickly validate new product concepts with customers early and often?

“Early and often” varies greatly by industry, but you should know what is considered fast and frequent in your neck of the woods. Great PMs know that this is how you insure against building something nobody wants. "Fail often, fail fast" is better than fail once and go bankrupt because there’s no time/money left to try again.

The virtues of Agile development are well known at this point, just f&*@!ing do it! I mean REALLY do it: show your customers incremental work, get feedback, iterate. You will be better for it. A demo to your colleagues, boss or the senior VP of Whatever doesn’t count. If they’re not buying or directly using the product, they’re not a customer.

4) Are you the "mini-CEO" of your product?

Here’s a daily task list of two hypothetical PMs:

Bob, the “career” product manager:

- Document requirements for feature ABC
- Discuss design of feature 123 with UX team
- Roadmap update to exec team
- Review prototype of feature XYZ with engineering team
- Evaluate A/B test results from last week

Carlie, the mini-CEO:

- Prep marketing on upcoming release
- Discuss design of feature 123 with UX team
- Shadow sales calls to hear customer reactions on new feature X
- Respond to customers on the support tool regarding bug 4560
- Ask for budget to buy product team noise-cancelling headphones
- Roadmap update to exec team

What’s the difference? Bob is a “standard” PM doing what is expected of him. Carlie is a “mini-CEO” doing whatever it takes to win. Bob may not think it’s ok for him to step up and fill in the gaps for the team, or he’s tried many times but the company culture shot his initiatives down. Carlie helps first and asks questions later because she knows that, ultimately, product success is the only real metric by which product managers ought to be measured.

5) DO PEOPLE trust your decisions?

Big idea: the authority of a product manager is earned, not granted. I can’t think of any other role where the immediate team a manager is asked to lead doesn’t actually report to her! Well, this is the case of the PM role. When a PM does not inspire trust, every decision is met with resistance; things take more time because people are second-guessing themselves. When it’s working, there’s less talking and more doing across the board.

6) Can every single engineer/designer/marketer/etc in your team explain why she’s doing this work and why it matters?

Another quirkiness of the PM role is that, despite the fact that product success is the only metric that matters on this job, we don’t actually “make” anything! Therefore, great PMs make sure the “makers” on the team know the customer, the problems she faces and why it really matters to solve them well. In fact, great PMs go as far as removing themselves as the proxy between customers and makers so they connect directly.

“But won’t this distract the engineers? They should be coding!” Again, building something people want is the name of the game. What’s a better way to achieve that goal than to empower your team with the skills to empathize with customers? Great PMs don’t feel the need to be the “voice of the customer”. They coach teams to listen and know when to insert themselves in order to keep delivering value.

7) Does your product team show incremental work to stakeholders every few weeks? 

This issue becomes more important as company size increases, but the vast majority of PMs have at least some internal stakeholders to keep updated as part of their jobs. Aside from the obvious benefit of letting people know what the team is up to, you should honestly examine your motivations around WHY you're updating stakeholders:

Bad motivations: to validate solutions internally because it's easier than experimenting with customers or to go fishing for product direction because you don't actually know what customers want. This is bad for 2 reasons: 1) this is actually your job, not your stakeholders' and 2) stakeholders are not customers, therefore they will tell you what they THINK customers want. This game of telephone is easily how most teams end up building something nobody wants.

Good motivations: to educate the company on what problems customers have, to get permission to experiment with potential solutions, to get other departments' support in executing those experiments and to report back on past experiments.

8) Do you understand how your product initiative and others fit an overall company strategy? 

In almost every all-hands-type meeting where strategy is discussed, there comes a point where the presenter will say something to the tune of “…and YOU are the people who are going to make this happen!”. Well, in the case of Product, that message hits pretty close to home. Product, after all, is the “something” a business makes for customers to buy.

The reason it is critical for PMs to know how to tie their particular product initiatives to the company strategy is because that's the only common language people across various departments have to communicate with each other. You need to get Marketing, Sales, Ops, etc to immediately understand why they should help you and how your collaboration will make everyone win. The bigger the company, the more important this skill becomes.

9) Do you have a clear product vision articulating what problems will be solved when you deliver it? 

A product team without a vision is like a ship without a destination. To paraphrase Alice in Wonderland, "If you don't know where you're going, any road'll take you there”. What is a product vision anyway? In a nutshell, it’s what success looks like when it’s all said and done. Whose life will be made better? What problems will cease to exist? You don’t need to know HOW you’re going to solve these problems, but you do know you WILL solve them.

Aside from the obvious benefit of knowing where you’re headed, a product vision has a few additional benefits:

- It aligns stakeholders as to what we actually are talking about. Never underestimate the ability of people to assume completely different things based on the same exact information.

- It helps when the vision also outlines what we are NOT talking about, i.e. these are the problems we don’t think we will solve.

- For what it’s worth, a clear vision is one of the most effective ways for PMs to earn their stripes inside an organization. That’s not to say you need a complicated vision; in fact, the more succinct and believable your vision for the product is, the better.

10) Do you have a roadmap explaining the milestones on the way to that vision? 

This is basically your stepping stones towards that grand vision. A logically sequenced roadmap makes things believable to the organization, improves team morale as you check things off the list and ensures that incremental value is being delivered to customers along the way. 

The product vision and roadmap are meant to be living documents. As you learn about the world, your opinions will change. These key pieces of collateral will help you communicate insights using a simple before/after presentation of what was important before vs what we think is important going forward.  

There you have it, folks, the “Joel Test” for product managers. Regardless of how organizations are setup, PMs do and should have a common reference of what is expected of them. As Joel himself put it, "a score of [10] is perfect, [9] is tolerable, but [8] or lower and you've got serious problems. The truth is that most software organizations are running with a score of 2 or 3, and they need serious help".

It's well understood nowadays that an underperforming product management practice creates just as much problems as an engineering one. As a PM, you can drive change in your company by simply pointing out how well you’re scoring on this test. Knowing there is a problem is the first step towards fixing it!