If a great product is the result of combining a real customer need with a solution that’s just now possible, then it’s easy to see why the relationship between the product manager and the engineering team is so critical.

The product manager is responsible for defining the solution, but the engineering team knows best what’s possible, and of course they must deliver that solution.

Product managers quickly learn that if you have a good relationship with engineering, then this can be a great job. If you don’t, let’s just say that you’re in for some very long days.

One key to this relationship is to each be very clear that you are peers. Neither should view themselves as subordinate to the other. You are responsible for defining the right product, and your engineering counterpart is responsible for build the product right. You need both or you have nothing. You need to let your engineering counterpart do what he believes necessary to build a quality product, and he needs to give you the room to come up with a useful and usable product.

That said, you can actually both be a huge help to each other.

Your engineers can be a big help to you as you work to define a winning product. Remember that they know what’s possible better than anyone. Here are three ways you can use your engineers to come up with a better product:

1. Get your engineers in front of users and customers. They will learn a great deal from seeing users struggle first hand. Not only will they get a better appreciation for the issues and their severity, but more importantly this is often the inspiration for much better solutions. You can do this easily by inviting an engineer along to your prototype testing. See Prototype Testing.

2. Enlist the help of your engineers in exploring what’s just now possible. Brainstorm the different technologies that are available or coming available, and how they might help solve the problems at hand. See The Smartest Guy in the Room.

3. Involve your engineers (or at least a lead engineer) from the very beginning of the product discovery process to get very early assessments of relative costs of the different ideas, and to help identify better solutions. One of the most common mistakes that product managers make is to come up with some great product definition, and then throw it over the wall to engineering. That just postpones the critical negotiation process until there’s not enough time to be able to make good and informed decisions. See Product Discovery.

Similarly, you can actually be a big help to your engineers. Here are three ways you can help them do their job:

1. Keep the focus on minimal product. Remember your job is not to define the ultimate product, it’s to define the smallest possible product that will meet your goals. This point alone will fundamentally improve the dynamic between product management and engineering. See Great Products By Design.

2. Do everything you can once engineering begins to minimize churn. Churn is changing requirements and product definition. Some churn will be unavoidable, and engineers understand that some things are beyond your control, but remember that this is not the time for trying out your newest ideas. See Strategy vs. Execution.

3. There will inevitably be questions that arise during implementation. Cases that were missed or weren’t completely thought through. This is normal, even in the best of product teams. However, your mission in life during the implementation phase is to jump all over their questions and get them answers as fast as humanly possible, always trying to keep to minimal product and minimizing churn.

One related note. I always try to encourage the best engineers to come try product management. I argue to them that it doesn’t matter how great the engineering is if the team is not given something worthwhile to build. And I point out the many great products and companies that have been created by an engineer that knew what was possible tackling these bigger questions of what to build. It’s great for their career development, and can be great for the product (and hence the customers and the company).

Email to a friend

Sign up for the free newsletter here.

Product management is all about choices. Making decisions about what opportunities are worth chasing, which problems are worth solving, what features will provide the most value, what the best time-to-market trade-offs are, and which customers are most important. While you’ll never make all the right choices, you have to make most of them right for your product to succeed.

One of my favorite tools for helping to make the hard decisions is a persona (aka user profile). For those that don’t know what a persona is, they are a technique for capturing the important learnings from interviewing users and customers, and identifying and understanding the different types of people that will be using your product. The persona is an archetype description of an imaginary but very plausible user that personifies these traits – especially their behaviors, attitudes, and goals.

The tool was first described in 1998 in one of my all-time favorite books, “The Inmates are Running the Asylum,” by Alan Cooper. If you haven’t read this book you should. It’s a classic for product managers, designers and engineers.

There’s a good chance that your designers already use personas in the design process. The design community seems to have adopted this technique as most of the design teams I meet use this tool. Each has their own spin on what makes a good persona, and some are much more formal about them than others, but in my view it's all good.

And there’s even a chance your marketing people use personas as they create their messaging and advertising programs. This use of personas is similar, and they’re both useful, but they’re not quite the same thing as they are used for somewhat different purposes. The marketing folks are trying to determine the best ways to reach the target customers and appeal to the underlying emotions. The designers care most about the user’s goals and online behaviors.

As product manager, this is all extremely useful to you.

Unfortunately, while this is a truly powerful tool, it is often not employed until later in the product definition/discovery process than it should be. Often it is the designers that drive this, and they are all too often brought in later in the process than they should be.

To get the true potential of personas, the product manager needs to be deeply involved in the creation and prioritization of the personas, and especially the user interviews and research that goes into identifying them. The creation of the personas should be a collaboration between the product manager and interaction designer, and if you are lucky enough to have them, your user research team. But whatever you do, don’t delegate this task. For the same reason that the product manager needs to be at every usability test of his product, he needs to be at every user interview. The product manager needs that deep understanding of the target user that comes from talking with as many users and customers as possible.

So I try to encourage product managers to actively participate in the creation of these personas, and make sure they are done as early in the process as possible.

The design community has written pretty extensively on personas so I won’t try to duplicate that here, other than to point out some issues as they relate to product managers.

There are quite a few benefits of using personas as a tool for product management:

- Personas help you prioritize what’s important. If you have decided to make “Mary” the target for this release, then if this feature is critical for “Mary” then put it in, if it’s for “Sam” then it’s out. As you can see, just as important as deciding who a release is for, is deciding who it is not for. It is an extremely common mistake for a product to try to please everyone and end up pleasing no one. This process can help prevent that.

- One of the most common mistakes product teams make is confusing themselves with their customers. I’ve written elsewhere about this problem, but one thing I really like about personas is that they help shine a light on this all too prevalent problem.

- Many products have many types of users – different types of end-users, managers, administrators, etc, and it’s easy to think that you should just put some features in for each of these people, and again, end up with a muddled mess. This is partly a design problem, but personas often help you prioritize the importance of these different users, and also realize where you need a separate user experience.

- Personas are a very useful tool for describing to your entire product team who the product is for, how they will use it, and why they will care.

- More generally, just like the manifesto/product principles, personas have the benefit of rallying the team around a common vision. There are literally thousands of details that will have to be addressed in the course of a product release. The Product Manager (or designer) can’t possibly make every one. If every manager, designer, writer, developer, and tester has taken the product principles and personas to heart, then when faced with an open question, they are more likely to make the right call.

These are some pretty great benefits. But there are also some pitfalls to watch out for:

- Some teams create personas but they don’t take the next step which is to make the hard choices about which persona should be the priority. It’s not ok to say your product is for everyone. You’re deluding yourself. While this is extremely difficult for most product managers, I try hard to get the product manager to focus each release on a single primary persona. It’s not that the release won’t be useful and usable by others, but your focus on each release should be to do a great job for this one type of target user.

- Sometimes teams create personas based on their assumptions and stereotypes of their users, and they don’t actually take the time to talk to real users and verify if these theoretical people really exist. I have been surprised many times. So many times in fact that I have learned just to consider my initial impressions as just a theory, and hold off on the real judgments until after talking with real users.

- One question that often comes up is that as you recruit users for your prototype testing, do you only test with users from your primary persona? Certainly you need to make sure your product is great for the people that it is intended for, however, you’ll want to test with some people from outside this persona as well, as you won’t have the luxury of having only primary personas using your product. So for prototype testing, you’ll want to recruit a range of reasonably-possible users.

If you haven’t done so already, consider creating a persona to describe your primary target user for your next release, and see if you can’t use this tool to help you make the hard choices.

Email to a friend

Sign up for the free newsletter here.

Earlier I’ve written about how important it is to clearly distinguish the roles of product management and product marketing (see Product Management vs. Product Marketing). But many companies suffer from a related problem, which is when the roles of product management and project management are combined.

I’ve also written earlier about how the nature of product management and project management are different and typically attract different types of people to each (see Strategy vs. Execution).

But I haven’t written yet about why this situation exists, and I think this sheds light on several related problems.

The reason so many Internet companies still define product management as including project management is because many of our practices came from the shipped software world. In the shipped software world, like what Microsoft became famous for, it is common to have product managers cover the project management role (in fact, Microsoft still does this under the job title “Program Manager”). But this is one of those areas that doesn’t migrate well to web services.

To explain, first a little bit of Internet history. When Internet services came about, around 1996 or so, at first we struggled with whether to continue to call ourselves “Product Managers” because things like a Travel site are more of a service than a traditional product, but we quickly got over that (although sometimes we called the role “producers” to emphasize the media and content nature of the services, but that wave too passed fairly quickly).

However we initially tried to continue having the product manager cover the project manager role. Early internet companies like Netscape and Yahoo did this. But we all ran into a problem with this. In the shipped software world, the product was generally shipped as a self-contained unit. So the “product” generally was in the same granularity and frequency as the “project” so it’s not so hard for the product manager to double as the project manager. But in the web services world, with many releases underway in parallel, this model breaks down.

Not to complicate matters too much, but most internet service companies found that they needed to make frequent smaller releases to a larger common code base. And since the duration of a typical project is longer than the release interval (usually ranging from weekly to monthly), this quickly turns into parallel development and the software train model of releases. Most internet companies that are beyond the startup phase use this train model.

The train model is really a topic in itself, but suffice it to say here that a train requires active and strong project management, which is not tied to specific projects. A train typically contains features from many product managers. A train has significant release management, engineering, site operations, customer service and product management coordination requirements.

Some internet companies refer to the project manager of a release train as the train’s “conductor.”

If you use the train model, and you have project managers dedicated to the release trains, you generally don’t need product managers to cover project management too.

So as the release process at companies like Yahoo, Netscape/AOL, and others became more sophisticated, the project management responsibilities were untangled from the product management role, and all of these companies developed very strong dedicated project management competencies. Many newer Internet companies like eBay and Google could not release the quantity and quality of software they do without their very strong project management team spanning product management, engineering and site operations (see eBay's Secret Weapon )

For Internet services companies, it really is important that the roles be separate. You’ll thrash in release management if you don’t, and releases will consistently be delayed and take longer than they should.

If you are still doing shipped software, I still think it’s useful to separate the roles, but this is more due to the nature of discovery-oriented product management versus execution-oriented project management.

Email to a friend

Sign up for the free newsletter here.

Readers of these articles know that I view the high-fidelity prototype as the primary means of describing the product to be built. I have written elsewhere why a prototype is significantly more useful to the product team than the typical paper-based specification. However, that's really the secondary benefit. The primary reasons to create a high-fidelity prototype are to help you gain a much deeper understanding of your product, and ultimately so that you can actually test your ideas with real users before you have your engineering teams take months to go build something that you have no real evidence will serve its purpose.

In this article I'd like to talk about how to actually do this prototype testing. I'll warn you up front that this article is relatively long, but I will also say that testing your ideas with real users is probably the SINGLE MOST IMPORTANT ACTIVITY in your job as product manager.

If your company is large enough to have its own usability testing team, by all means secure as much of their time for your project as you absolutely can. Even if you can't get much of their time, these people are terrific resources and if you can make a friend in usability research or usability testing it'll be a huge help to you.

If your organization has funds earmarked for outside services, you may be able to use one of many excellent firms to conduct testing for you. But at $10,000 - $20,000 per round of testing (typically around 10 users) that most firms charge, chances are that you won't be able to afford as much of this testing as your product will need.

But if you're like most companies, you have few resources available, and even less money. But you can't let that stop you. It is absolutely essential that you test your ideas out with real users. It is arguably the single most important part of your job.

So I'll show you how to do this testing yourself. Don't get me wrong, you won't be as proficient as a trained usability engineer, and it'll take you a few sessions to get the hang of it, but in most cases you'll find that you can still identify the serious issues with your product, which is what's important.

One thing to note is that while most people think of "usability testing" (seeing if people can figure out how to actually use your product) I consider that just one type of testing. You also need to test the desirability or usefulness of your product (do people actually want to use it?), and we'll discuss both forms of testing here.

Finding Test Subjects

Before you do the prototype testing, you'll need to round up some test subjects. If you're using a lab they'll recruit and schedule the users for you, which is a big help, but if you're on your own, you've got several options:

- If you've established a "charter customer program" as I've described earlier, you should have quite a few readily available. If you haven't you should.

- If you're doing a product for business, then trade shows are a great source of target customers.

- It's increasingly common to advertise for test subjects on Craigslist. If you do this, try to keep your participant description a notch more general than specific, and then screen on the general when you call them to talk about this.

- For consumer products you can use your "friends and family" network - just try to avoid people too close to you, and those in the tech industry, unless that's specifically your target, and be sure to use subjects from outside this network too.

- If you have a list of e-mail addresses of your users you can do a selection from there - often your marketing team can help you narrow down the list.

- You can solicit volunteers on your web site - lots of major sites do this now - remember you'll still call and screen the people to make sure you don't get a bunch of early adopter types.

- One technique I especially like is to set up regular prototype test sessions, say every other Friday, where you arrange for 10 - 20 or so users to come into your offices for a couple hours each, and then your product managers sign up for the time slots, so a given user might test a couple prototypes each. I like this a lot because one person can do the logistics of invites and screening, and many PM's can count on a ready set of test users on a regular basis.

- You can always go to where your users congregate. If you're doing an e-commerce product, you may want to go to a mall. If you're doing a sports product, go to a sports bar. If your product is addressing a real need, you won't have trouble getting people to give you an hour. Just bring gifts, and try not to look like you're trying to convert their religion.

- If you're asking users to come to your location - especially for business use - you will likely need to compensate the people for their time. If you're doing a consumer service sometimes a big sincere thanks along with a hat with your company logo on it will suffice as people genuinely want to help in the creation of products, especially for companies they like. However, if you do compensate consider providing product like $50 of credit on your site.

- Realize that there's a very high no-show rate when you schedule people to come in. It's just a fact. Sometimes as much as 30%. But you can drop it to 5- 10% if you give people a personal phone call the day before -- even if you leave a voicemail. Note that email does not work equally well.

Preparing The Test

You'll need to define the usability tasks you'll want to test, and the interview questions concerning desirability:

- You will need to define in advance the set of tasks that you want to test. Usually these are fairly obvious. If you're building an e-mail client, your users will need to do things like compose a message, read new mail, and file away messages. There will also be more obscure tasks, but concentrate on the primary tasks, the ones that users will do most of the time. If you have time, you can get to less common tasks but it's essential the key tasks are tested well.

- There is a one-time only opportunity you have with each user you test. That is the opportunity to learn how they think about this problem today.

If you're testing a new online restaurant rating service, rather than start them out at your prototype's home page, maybe you want to just start them out with an empty browser and see what they do. What review sites do they use today? Do they use Google or Yahoo's search to find the specific restaurant, or do they go somewhere like OpenTable or Zagat? Do they search by neighborhood, by cuisine type, or price range?

This type of incredibly valuable information is missed if you jump right into your prototype, which will necessarily have quite a few assumptions built in. Once they play with your prototype they can tell you what they like better, but they won't be thinking about the problem anymore the way a first-time visitor would.

- Of course, you'll then want to get them to your prototype, but there's one more thing before you jump into your tasks. See if they can tell from the home page or landing page of your prototype what it is that you actually do, and especially what might be valuable or appealing to them. Again, once they jump into tasks they'll lose that first time visitor context, so don't waste the opportunity. You'll find that landing pages are incredibly important to bridging the gap between expectations and what the site actually does.

- After you've seen if your user can figure out how to do the tasks you're testing, it is now the right time to have a conversation with this user. Think of it as a one-person focus group. Does the user use a different product or site today with the same thing? Or is this something they do manually or off-line today? How much better is this than what they use today? And don't forget to ask my favorite question: How likely would you be to recommend this product to your friends? Now that the user has interacted with your prototype they understand the topic and you can have an extremely useful dialogue with them about this problem. Most importantly, you're trying to gauge how much this person values this product.

- One technique I like for gauging value is to ask how much the user would be willing to pay for it, even if you have no intention of actually charging for use this way. It's a way to assess value and especially to track how the average value goes up or down over time as you change the prototype. Also, it's useful if you structure your questions on a scale, like 1- 10. This is so that you can track the averages as they improve.

- Note that you don't have to wait until you have a complete prototype in order to begin testing. You can start with the main tasks, and it's ok if you have dead ends in the rest of the prototype. If the user wanders over to one of those dead ends, just ask "And what would you expect to happen if you did that?" This is a great question whether you have that path laid out or not. If you do not have it laid out, you can see if they match. And if you don't, you'll get important info about that direction anyway.

The Test Environment

Here's how to prepare your test environment:

- Formal testing labs will typically have set ups like two-way mirrors, or closed circuit video monitors, as well as cameras often capturing both the screen and the user from the front. Just know that while that's great if you have it, you do not need these things to have an extremely useful and valuable test. I can't count how many prototypes I've tested at a tiny table at Starbucks just big enough for the laptop, with three chairs around the table. In fact, in some ways this is preferable to the testing lab because the user feels a lot less like a lab rat.

- The other environment that works terrific is your customers office. It is time consuming to go there, but often even 30 minutes in their office will tell you a lot, and they are "master of their domain" and often very talkative. Also, all the cues are there to remind them of how they might use the product. You can also learn from seeing what their office looks like. How big is their monitor? How fast is their computer and network connectivity? How do they communicate with their colleagues on their work tasks?

- There are tools for doing this type of testing remotely, but while you can see where their mouse is, and what the user is clicking on, it's not the same as looking at the person's eyes and body language, so again, while the more testing the better, to me this is not a substitute for face-to-face testing.

- As product manager, you need to make sure you are at every single test. Do not delegate this. Real value comes from experiencing as many users as possible, first hand, interacting with and responding to your ideas. Even if you use an outside firm to arrange and administer the tests, you need to be there with them during the testing. No one knows your product as well as you do, and you will have insights from watching the slightest hesitation or confused look, or the nuance of a question that reveals that they don't understand the product model or particular feature. What gets summarized for you by a proctor will probably miss 4 or 5 key insights.

- Some people believe that the product manager (and the interaction designer) are too close to the product to do this type of testing objectively, and they may either get their feelings hurt, or only hear what they want to hear. My view is that the good product managers and interaction designers get past this very quickly. They know they will get the product wrong initially, and that almost nobody gets it right the first time, and they know that learning from these tests is the fastest path to a successful product.

- Ideally, you should have one person administer the tests and another person taking notes. It's very useful to have someone to debrief with afterwards to make sure you both saw the same things and came to the same conclusions. That said, if it's just you and your laptop, and you've got a ready and willing target user in front of you, do it. It is all good.

- If you as product manager have a user researcher or usability engineer along with you, let him or her administer the test and you take notes. Otherwise, you'll probably be the one that administers. It's great to invite others on the product team to be your note taker. Most often it will probably be the interaction designer, but the visual designer, developers, and especially mangers are all useful and they'll get a lot out of it.

Testing Your Prototype

Now that you've got your prototype ready, you've lined up your test subjects, and you've prepared the tasks and questions, here are a set of tips and techniques for administering the actual testing:

- When you first sit down with the test subject, make sure to tell him or her that this is just a prototype, it's a very early product idea, and it's not real, they won't be hurting your feelings, and you're testing the ideas in the prototype and most importantly you're not testing her. She can't pass or fail - only the prototype can pass or fail.

- Also, you should greet the person warmly and offer a coffee or something, but the sooner you get to the prototype the better. Tell them you'll chat with them about this after they test the prototype, but you want to get their untainted impressions first. Realize that the more you chat beforehand, the more clues you are giving away and the less of a true first impression your test subject can provide. If more than 5 minutes goes by without the user starting in on the prototype you are talking too much.

- When testing, you'll want to do everything you can to keep your users in "use mode" and out of "critique mode." What matters is whether users can easily do the tasks they need to do, and whether they value the product. It really doesn't matter if the user thinks something on the page is ugly or should be moved or changed. Sometimes misguided testers will ask users questions like "what three things on the page would you change?" To me, unless that user happens to be an interaction designer, I'm not really interested. If users knew what they really wanted, software would be a lot easier to create. So watch what they do more than what they say.

- During the testing, the main skill you have to learn is to keep quiet. Normally when we see someone struggle most of us have an urge to help the person out. You need to suppress that urge. You have to turn into a horrible conversationalist. Get comfortable with silence.

- There are three important cases you're looking for: the user got through the task with no problem at all and no help; the user struggled and moaned a bit but he eventually got through it; he got so frustrated he gave up. Sometimes people will give up pretty quick, so you may need to encourage them to keep trying a bit longer, but if he gets to the point that you believe he would truly leave the site and go to a competitor, then that's when you note he truly gave up.

- In general, you'll want to avoid giving any help or "leading the witness" in any way. If you see the user scrolling the page up and down and clearly looking for something, it is ok to ask the user what specifically he's looking for, as that's very valuable to you. Some people ask users to keep a running narration of what they're thinking, but I find this tends to put people in critique mode, as it's not a natural behavior.

- Act like a parrot. This helps in many ways. First, it helps avoid leading. If they're quiet and you really can't stand it because you're uncomfortable, tell them what they're doing: "I see that you're looking at the list on the right." This will prompt them to tell you what they're trying to do, looking for, whatever. If you ask a question, rather than giving a leading answer you can play back the question to them, "Will clicking on this make a new entry?" "You're wondering if clicking on this will make a new entry?" and they will take it from there because they'll want to answer your question: "Yeah, I think it will." Parroting also helps avoid leading value judgments. If you have the urge to say "Great!", instead you can say "you created a new entry." Finally, parroting key points also helps you note-taker because they have more time to write down important things.

- Fundamentally, you're trying to get an understanding of how your target users think about this problem, and to identify places in your prototype where the model the software presents is inconsistent or incompatible with how the user is thinking about the problem. That's what it means to be counter-intuitive. Fortunately, when you spot this it is not usually hard to fix, and can be a big win for your product.

- You will find that you can tell a great deal from body language and tone. It is painfully obvious when they don't like your ideas, and it is also clear when they genuinely do. They'll almost always ask for an email when the product is out if they like what they see, and if they really like it, they'll try to get it early from you. Recently I attended some prototype testing with one of my clients in Berlin, and even though I don't speak German, it was obvious what the issues were, and which ideas worked well and which ones didn't.

Updating the Prototype

The whole idea of this testing is to identify what you need to fix in the prototype to make it more useful and usable. So, as fast as possible, you'll want to work to correct the problems:

- Some people believe you have to freeze the prototype, the tasks and the questions for a complete round of testing (generally 6-8 users) before drawing any conclusions. I don't think that's true, and have found that you can significantly accelerate this process of getting to a good product faster by responding more quickly to the feedback.

- You don't have to be hit on the head by 8 users in a row to know you need to fix something big. So go ahead and fix it when you feel you've identified a problem, even if that's after only 2 or 3 users. The harder question is when you're done. Generally, if you can get through 6 consecutive users where they understand and appreciate the value, and then can get through the key tasks, you're in good shape and you've done your job.

- You might determine that you just aren't able to get people interested in this problem, or figure out a way to make this usable enough that your target users can realize this value. In this case, you may decide to stop there and put the idea on the shelf. Some product managers consider this a big failure. I view it as saving the company the wasted cost of building and shipping this product, plus the opportunity cost of what your engineering team could be building instead.

This whole process might sound complicated or difficult, but the remarkable thing is just how easy and effective it actually is. The best way to prove this to yourself is to take your laptop with your product or prototype on it to someone that hasn't seen it yet and just give it a try.

I want to give credit here to two important sources (and great resources for you).

The first is the book, "Don't Make Me Think" by Steve Krug. It's mostly a book on interaction design, but in the back third he makes a compelling case for this sort of informal testing, and he provides many useful tips. I have long recommended this book to both product managers and designers, and I hope you'll buy a copy and read it carefully.

Second, my favorite product testing firm is Creative Good (www.creativegood.com). These guys specialize in a form of testing they call "Listening Labs" which is a powerful form of undirected testing that can find huge issues with your product - functionality and design - that can result in pretty dramatic improvements to the business results. While most testing firms test the products on a task basis, these guys are good at looking at the big picture, which is where many of the big improvements are to be found. Several of the techniques above are adapted from their testing methodology.

Thanks also to a couple of world-class user research specialists, Kyrie Robinson and Audrey Crane, for their insights and contributions to this note.

Email to a friend

Sign up for the free newsletter here.