WebWord.com > Interviews > Spool Vision: An Eye on User Data (12-Apr-99) If you want to know when new interviews go online, subscribe to the WebWord.com Usability Newsletter! Spool Vision: An Eye on User Data An interview with Jared Spool, Founding Principal of User Interface Engineering User Data Is Gold What have been your most striking findings? What usability data have shocked you the most? Why? I think our most striking finding is how bad web sites are in general. We have yet to find a site where, if you choose questions at random based on information the developers have placed on the site, users can find the answers more than 50% of the time. (The best we've found is 42% of the time.) We think this means that, as designers, we still don't know what it takes to design a good web site. We now know that information design is an important part of web site design. How the information is distributed through out the site plays a huge role. For example, we've found that, in general, longer pages, packed with more information help users complete their mission better than shorter pages designed to prevent scrolling. You can see this if you compare the designs of the news sections of CNN and MSNBC. While MSNBC's menus are cool looking, it is much harder to get a picture of "what's really happening?" from the site than from CNN's simpler, but long page design. We also have found that graphic design is not as important as people think. The time it takes to generate nice graphics that download fast doesn't pay off in terms of the increased usability of the site. Some of the most popular sites are very text intensive. For example, Amazon could use the fonts for each book cover (like the fonts that drip blood often found on Stephen King book covers) -- to keep the "branding" of each book working for the users. Instead, they've chosen to use HTML text, because it appears instantly, whereas a downloaded graphic with the title would take a few seconds to appear. Users can read the description of the book while the image is still downloading. We have also found that users, when trying to accomplish their mission, tend to follow "a scent" and when they loose the scent, they fall into predictable behaviors, such as randomly clicking on links or hitting the back button. Sites where the target information has a strong scent are the ones that perform the best in our studies. For example, a recent site we tested allowed users to choose from a variety of cellular phone models. The users could click off the options (weight, size, features) that were most important to them. However, the results that were presented didn't give any further information. Instead it just displayed all the phones that had some or all of the desired features. To decide which of the 15 phones selected would be best, users had to click on each phone, memorize the specific features, hit the back button, and click on the next phone. There was no way for users to easily find out the thing that was most important to them: Which phone should they buy? Recently, we've been studying how marketing and design interact. On the web, you have a strange mix that never really happens in software or documentation design, the other two areas we specialize in. When people use software, it's usually already been purchased, whereas when people use a web site, it's often in a pre-sales situation. Designers now have to take marketing into account. For example, we've learned how to measure the effects of "branding" as people work through a site. We can tell if a brand has been strengthened or weakened after the user has worked with as site. From this, we found, for instance, that (for the users we tested) the Ford brand was strengthened after using the Edmund's site but weakened after using Ford's own site. We are beginning to understand what variables are important to ensure the organization's brands are strengthened. But, even armed with all this new knowledge about design, marketing, and the web, we don't know how to build a truly great site. Our quest is to find out how to build great sites. How have your clients reacted to the user data that you have given them? They love it. We can't get enough data to them fast enough. The more we share, they more they want to know. I give some details later in the interview about our popular usability course Web Sites that Work: Designing With Your Eyes Open. It has been selling out every time we offer it. Companies are calling us every day to bring it on site. We're hiring as quickly as possible to allow us to get the information out as we uncover it. What steps do you take when usability problems are apparent? Two words: Team awareness. Once the team is aware of the problems, they are usually in a much better position to determine the right course of action. They know what their resources are and how their technology works. We've yet to encounter a team that makes a "bad" decision, once given all the information needed. So, our job is to deliver information to the team. We do that through our research and through the techniques we've developed and refined that help teams collect their own info. In contrast to the previous question, how can good design be expanded and perpetuated? How does user data help? Design is gambling. When you take an existing design and make a change, there are only three possible outcomes: either you improve it, you make it worse, or you don't really have any affect. If you make changes at random, the odds of improving it are not very good (unless it was really bad to begin with). The name of the game is controlling the odds. Designers can control the odds when they make changes based on information about the users and how they will use the system. For example, what if you could predict the outcome of a given change? Let's say you were going to spend a week making changes, but before you started, I could tell you which parts of your new design were going be an improvement and which parts were going to be worse than what you had now. What would you do differently? For the parts that were going to be worse, what if I could tell you which of the three alternatives you came up with would work the best? What would you do with that information? As I said, the solutions are easy. It's knowing the problems that is difficult. That's where user testing comes in. That's the best tool we have for determining if our design is better or worse than before we made the changes. And with techniques such as paper mockups, we can determine the affects of our design in hours, instead of days. Some Background... What is User Interface Engineering? What is your role? User Interface Engineering is the largest usability firm in North America. I'm the founding principal. (Which means my office is "The Principal's Office," which is very comfortable for me -- I've spent a lot of time there as a child.) Tell us more about yourself. For example, what is your training? What is your experience with usability testing and user centered design? I started as a software developer, many years ago. I wrote one of the first voice mail systems (it wasn't my fault, really!), some of the first email clients, a couple of word processors, some spreadsheets and lots of system software. Initially, I wrote what I was told. It was all really complicated and krufty. I could tell that it would be hard to use and kept asking the designers and my managers as what we would do if people couldn't figure it out. Their response: "the users will fill out bug reports, like they always do." This didn't seem satisfactory. When I was at Digital Equipment Corporation, I connected up with the Human Factors group there. (The group contained such folks as John Whiteside, Dennis Wixon, Bill Zimmer, and Sandy Jones -- names which many old-timers will recognize.) They were focused on researching human factors issues, I was interested in building products that were usable. We taught each other a lot -- I taught them about getting products shipped, they taught me about user-centered design. Why is usability so important to you? In particular, why have you devoted your life to understanding and working with users? I guess I'm different from many of the key usability players in that I'm not a zealot trying to make the world more usable. Usability is an offshoot of what my interests are. Instead, my interest is in design. What makes a design good? Certainly, if it meets the user's needs, it's good. But how does a designer integrate the user's needs into the design process? If so many products that come out today are not well designed, what is it about our design processes that prevent good design? Our initial work in this area says that the #1 thing that prevents good design is that the designers are missing key information. They are missing information about who the users are, what they will do with the product, how the product fits into the user's lifestyle and goals, how users are different from each other and how they are similar. Most importantly, we don't have any mechanism that easily lets a developer tell if the wonderful design that they just imagined will actually make the product better or worse. We've all upgraded to new releases of products that were no better, just different. How do we prevent that from happening to our products? Gathering User Data Why should Web developers collect user data? Similarly, why do you think it is important to understand users? The goal isn't to collect user data. The goal is to build great web sites. Right now, the only tool we have for determining if a web site is usable is to watch users use it. If they can use it, great. If not, fix the problem -- try again. We've found that developers find it easy to generate multiple solutions to problems. However, knowing what problems need solving -- that's the hard part. The traditional development processes don't have any good way to identify the problems. So, developers end up solving things that don't need solving and leaving important problems unsolved. Web developers need user data to find out if the hard work they are about to invest will make it any easier for users to accomplish their mission. If not, why bother? What data gathering techniques and procedures do you typically use? For example, do you videotape users or record their activities? Yogi Berra once said "You can observe a lot just by watching." That's our motto. An experienced observer can capture everything they need on their notepad. In our course, Web Sites that Work: Designing with Your Eyes Open, we have several exercises where the class participants conduct simple usability tests with their classmates. The results are astounding -- within a few minutes, these designers have a list of changes that they are ready to make to the site, based on just those simple tests. Imagine what would happen if we had real users in front of our designs every day. For our clients, we show them how to do that. And we don't have any special techniques or procedures. Just sit quietly and watch. That said, we often videotape -- then we never watch the videotapes. We treat video like backup tapes -- 99% of the time you never look at them; 1% of the time you'll be very glad you made them. Sometimes they help when the development team can't physically make it to a test. There are few things more boring in the world than watching videotapes of usability tests -- unless it's your site that's being tested. One technique that we use a lot is paper mockups. This technique allows us to have a full working site in just hours. After each test, we identify the key problems and change the design (in minutes) to try to rectify the issues we saw. Then we try it again with new users. It's amazing how quickly we can evolve to a design that really addresses what the users need. What are your favorite data collection tools? Specifically, do you use any programs or special software to collect data? The most important data collection tool is the human brain. Everything else is just to augment it. In most projects, we don't use any fancy tools -- just paper and pencil. We write our observations on post-its or index cards, one per card. Then we can put all our observations on the wall and sort and compare. In some of our web studies, we've put together some fancy logging software for very specific purposes. For example, in one study, we were interested in why users chose each link to click on and what made it the best choice, in their minds. We built proprietary logging tools that allowed us to capture specific information about each click. We redesign these tools for every study, because we're always looking for new things. Tools like this are research tools, not design tools. Designers don't need anything this fancy. It's just a distraction and doesn't yield any interesting value. What are the typical costs associated with usability testing? What are the budgeting considerations? You can do usability testing on a shoestring (less than $100 per test) -- we've been doing it for years. You can also spend $200,000 on a project with us. How much you spend depends on what you are trying to do. For most designers, they just need to find users and watch them. Preferably, you want to watch them do things they are interested in doing. We've learned that users act differently when the topic is of deep interest to them than when it is only of passing interest or they are doing it as a favor to you. The expensive projects come when there is a huge investment into a site that no-one really understands what the user's missions are. For example, one of our clients has 31,000 pages of content on their site. They have won many awards for design. But users complain all the time that key information is missing -- information that is on the site, but apparently they can't find it. Adding new technology, such as better search engines, hasn't decreased the rate or intensity of the complaints -- if anything, it has steadily increased whether the technology has been changed or not. So, we're helping the team understand the core issues about their design. It turns out, after extensive testing with a variety of techniques, including using an eye-tracker to see exactly where the user gazes on each page, we've found that the problems come from inherent flaws in the designs of the pages. We have another client who has 2,000 new people sign up for their portal service every week, however they've had the same number of visitors every day for the last six months. And while the site has 10,000 pages of fresh content updated every day, 83% of all visitors never click on any links on the home page. Since the site makes its money through advertisements, this is a big concern for them. So, we're studying what it takes to make their site more valuable for the people who sign up, so they will return and use the site on a regular basis. How are users usually recruited for a usability test? How do you "find" them? And, how many are needed? Again, this varies depending on the project. Sometimes we recruit people right off the street. We have a database of several hundred potential users that we collect in a variety of ways. When we get a project, we start by matching the target profile with what's in our database. If that doesn't work, we have all sorts of tricks for recruiting users. For example, a favorite is to use real estate brokers. Real estate brokers know everything about everyone. They are constantly walking through people's most personal belongings, so they know who has kids, who has disposable income, who has what occupation, who uses computers, and many other fascinating little details. If we're looking for people with a specific profile, the brokers are a good place to start. As far as the number, that also depends on the project. We've had projects where four users provided more information than we needed. We did a project this summer where we tested 91 users in 3 days. For most of our client work, we test until we reach "the point of least astonishment" -- that point where the team is no longer surprised by what they are seeing. Usually that happens between 6 and 10 users. We typically schedule 8 for that reason. Sometimes we feel we had one too many tests, sometimes we wished we had just one more when we were done. Wrap Up What are your favorite books? Favorite Web sites? Clement Mok's Designing Business is a great book about the importance of design and what it really is all about. I'm also a fan of Edward Tufte's books -- great demonstrations on what information design is all about. I haven't found any web sites that really are that impressive to me. Amazon and Yahoo come to mind, but only because they are so simple and single-minded in their design. (They both work with 2.x browsers! -- no JavaScript, ActiveX, or Shockwave!) But then again, maybe I shouldn't. After all, would you notice a really great voice mail program? Probably not, as it should be so good that you don't even realize it's there. My guess is that really good web sites are the ones that you don't realize helped you. But I don't know for sure, because I've never met one. Web Site Usability: A Designer's Guide is your data-rich book about Web site design and development. Could you tell us more about it? It's our first attempt at trying to publish some of the wealth of information we've collected by studying users working with web sites. While we've come a long way in our understanding of how the web seems to work since we first wrote the book, it still holds up pretty well. We still haven't found many instances where content should play second fiddle to navigation or graphics, despite what the designers want to think. And we've found that many of our original findings, such as less whitespace increases user success at finding their target information, are still true, several hundred users later. It's now being published by Morgan Kaufman Publishers and it's been selling really well. We're very happy with it. Please describe your Eye for Design newsletter. What does it cost to subscribe? Can folks get a free issue? This is one of our greatest secrets. Eye for Design is now in its 5th volume. Every 2 months we publish our latest results from our consulting and research work. Recent issues have talked about seducible moments, designing for branding, different ways to test documentation, how to recruit difficult-to-find users, and other pearls of wisdom we've acquired through the years. But don't tell anyone. If everyone were to get a copy, then they'd all know how to do better design -- we wouldn't want it to get out. The list price is $79/year. On our web site, there's a page where you can get the next issue for free without any obligation. Final comments? Wisdom to share? Shameless plugs? What, the previous two questions weren't shameless enough? Well, let's see. . . We've got our conference, User Interface 2000, that will be November 8-10 in Cambridge, MA. We've got our courses on paper prototyping, documentation, and web design. We've got UIEtips, our free e-zine that you can also sign up for on our web site. But, I won't mention those things, 'cause I don't want to be pushy and commercial. As to final wisdom, one of my favorite quotes is from Woody Allen: "There is no scientific evidence to prove the fact that life needs to be taken seriously." That pretty much says it for me. Thanks for encouraging my behavior. Jared, thank you very much taking time out of your busy schedule to explain your Web usability ideas. Great stuff! * Feel free to contact Jared for further information: Jared M. Spool jspool@uie.com User Interface Engineering 800 Turnpike Street, Suite 101 (978) 975-4343 North Andover, MA 01845 fax: (978) 975-5353 USA http://www.uie.com This interview was conducted via email by John S. Rhodes * Recommend Jared Spool's interview to a friend * Read another popular interview: Interaction Design: The Guru Speaks * Feel free to create a link to this interview, but please do not copy or redistribute it in any form without my permission. If you want to know when new interviews go online, subscribe to the WebWord.com Usability Newsletter! _____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Contact John S. Rhodes, WebWord.com Editor and Webmaster URL: http://www.WebWord.com/interviews/spool.html © 1999 by John S. Rhodes. All rights reserved. Do not reproduce or redistribute any material from this document, in whole or in part, without explicit written permission from John S. Rhodes. _____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________