WebWord.com > Interviews > The Usability of Usability (25-July-2001)

 
If you want to know when new interviews go online,
subscribe to the WebWord.com Usability Newsletter!

The Usability of Usability

An interview with Jared Spool, Founding Principal of User Interface Engineering

Conducted via email by John S. Rhodes (25-July-2001)


The Past

In your last WebWord interview, from about two years ago, we talked a lot about data. What usability data have shocked you the most over the last two years?

I think the thing that has shocked me the most is how wrong we've been about our base assumptions.  For example, it is often stated as if it was almost a law of nature that the faster pages download, the more usable the site was.  But when we actually compared the usability of sites to their download times, we didn't see any correlations.  None, zero, zip.  If this "fact" was true, we should've seen something. 

To go farther, we found that when we asked users to rate the speed of a site, that didn't correlate to the actual download time either.  Instead, the perceived speed of the site correlated strongly to whether they completed their tasks!  This tells us that, when users are complaining about download time, they probably aren't actually talking about the download time, but about their ability to complete tasks.  

We've had similar findings with the number of users needed for testing.  Again, common belief in the usability profession is that with 8 users, you'll find 85% of the problems.  In a recent column, Jakob Nielsen went so far as to say that 5 users was all you really needed and that you'll spot all the serious problems almost immediately.  

Yet in our recent research, we found that we didn't even get above the 50% mark after 18 users and were seeing serious problems all throughout the testing.  If the "85% theory" was true, we should've have learned everything after the first 8 users, with the remaining 10 only repeating problems we'd seen before.  But, it didn't happen as it was supposed to. 

The problem is that when our base assumptions, like download time or number of users needed for testing, are put into questions, we loose everything we have to hold onto.  It's like finding out that gravity doesn't really exist -- what's actually holding us on the planet? 

So, to answer your question about what has shocked me the most, I'd say it is the realization of how much we really don't know. 


Obviously you can't change the past but looking back over the last 4-5 years, what would you change about your usability work? What would you change about the usability profession?

(Actually, I can change the past. It's a little trick I learned as a kid. Well, I think I learned it as a kid -- I'm not sure.  The
past is sorta fuzzy that way. :)

I'm not one for regrets, because I believe that people will usually make reasonable decisions with the available information.  If, in retrospect, it seems like a decision was made poorly, it is often because the parties involved didn't have all the information they needed. 

Given that philosophy, I'd say that the biggest thing I'd try to do differently going forward is look more closely at our assumptions and where we get these "maxims of good usability."  In the early days (I'm talking 1977-1990 now), we spent a lot more time on solid empirical research.  Then, with the success of the Mac, we got on this evangelism bent.  

So, now the entire field finds itself in the role of preachers of a religion of sorts versus researchers of best practices. And we're now beginning to see pushback. 

My sense is that people don't want to be lectured to.  They want to get their problems solved. Repeating the same mantras over and over again doesn't help them solve their problems.  Real data on what works and what doesn't would help them solve their problems.

I take a lot of responsibility for where we are today.  We spent years promoting the "Well, it depends" attitude without ever answering the questions we were asked.  While it seems helpful at first, it quickly turns out to be a negative -- people don't want to just be told "it depends"; they want to know what to do differently. 

The biggest example of how wrong we've been falls in our response to business.  Recently, a reporter interviewed me for a story she was doing on why we haven't really seen any real improvements in the usability of web sites over the last few years. She had interviewed the "usual suspects" of gurus and consultants in the industry by the time she'd gotten around to me. 

The first thing I said to her was: "No matter what anyone else has told you, it's not the fault of the company."  Her response was "That's funny! Everyone else said it was the fault of the company"  She went on to say that this cadre of intellect had told her that if only these silly companies would wake up and understand that usability was important, the world would be a better place.  That's our [the usability profession's] position: all these companies are idiots and need to change.

It's ironic: For years, usability folks have cringed every time a designer says "the users need to make major changes to their behaviors if they want to use my product."  Yet, what do usability professionals do?  They make businesses undergo major changes to their behaviors if they want to produce usable products! 

Usability is not usable!  It doesn't work.  It doesn't produce the results we promise it will.  And we get angry when people stop paying attention to us. 

As a profession, we need to spend a lot more resources on basic research.  We need to stop thinking that there are pat, one-size-fits-all solutions to every problem.  And we need to align ourselves with the business goals more directly.  We need to make our own work usable. 


What things were done right over the last several years? What things were done wrong? Which web sites capitalized on usability? 

As far as I can see, the things that were done right in making web sites more usable didn't happen in the usability community.  It's interesting to note that the most usable sites in our studies, such as Dell, eBay, Amazon, and Edmunds have no real usability efforts to speak of. eBay has a relatively new team that is only a handful of folks, Dell also has a small team compared to the size and impact of their site.  The others: practically no usability team. 

Now contrast that with the organizations which have the largest usability groups, such as Microsoft and IBM. From a usability perspective, their sites don't do nearly as well.  There are substantial usability problems that permeate both companies sites seemingly without resolution, even though the teams work very hard.  

So, what does this mean when the most usable sites don't have large, high-powered usability groups and those companies with big groups don't seem to have an impact on their sites?  To me, it means two things:

  1. You don't need to have a usability group to make a usable site.  In fact, there is no evidence to suggest you even need a single usability professional on site to make a difference. 

  2. What we know about managing a usability group apparently doesn't work.  If anyone could make it work, it would be folks like Microsoft and IBM. I know those folks -- they are really smart people. I believe that they could do a bang-up job, if we knew what we were doing.  But we don't, so they don't. 

So, what was done wrong is understanding what it really takes to produce usable web sites.  We still don't really know.  In our studies, the best sites still fail users 57% of the time.  

(Is 57% an acceptable number?  Well, if your car failed you 57% of the time, you wouldn't get to work 3 out of every 5 days.  Is that acceptable to you?  Complacency doesn't get us very far.) 

So, which web site capitalized on usability?  None really. We've never seen a site that succeeds more than 50% of the time, let alone 60% or 70% of the time.  We don't know what one looks like.  We certainly don't know what it takes to build one.  Oh, we can pretend we do.  But we really don't, because we never have seen one. 

And what's worse is that the sites that have come closest didn't use what we've been preaching.  What does that say about what we know? 


The Present 

What are your hot research projects right now? What do people want? What do they still need? 

We've got a couple of things we're working on.  It's not our policy to discuss research-in-progress, but I can give you some ideas. 

We're very excited about our work in e-commerce.  From a usability perspective, e-commerce is one of the simplest types of web sites.  This is primarily because both the goals of the user and the goals of the business are easily identified and are met at the same time: when a product is purchased, everyone is happy. 

So, we see e-commerce as a "laboratory rat" -- a place where we can do some in-depth work, looking for cause and effect.  Everything we learn about e-commerce turns out to be valuable for many non-e-commerce sites.  

For example, our recent work on categories vs. search showed us that customers are more likely to keep shopping when they use categories and purchase more when they use categories. We also learned that the most likely thing that causes people to use search instead of categories is that the categories are poorly designed.  On sites where the categories were well designed (and we've compiled a list of things that make "well designed categories"), users never used search.  (This happened on 21% of the sites we tested.)  This means that e-commerce designers can change their designs and get more revenue from their site. 

Well, this particular piece of research could also be relevant to non-e-commerce sites.  While the point where users meet their goals is harder to identify in non-e-commerce sites (What is the goal of someone coming to CancerNet? How do you accurately measure if they've achieved it?), we can see how better categories might lead people to find content they wouldn't otherwise find on the site.  Since lots of our clients are telling us that they want people to "discover the value of their site", this seems like a hot direction to follow. 

So, we're looking at all the data we've collected from our e-commerce research and trying to mine it for all the juicy nuggets. (If you want to know more about our e-commerce research, look at our latest whitepapers. We'll also be presenting our latest results during the UIE Research Forum on October 4, 2001 at the User Interface 6 East conference in Cambridge.) 

We've also been looking into the world of advertising-paid sites. Here, there's a constant tension between bringing in revenue for the site, keeping the users happy, and keeping the advertisers happy.  It's a 3-way constraint system that has to be managed. 

For example, many advertiser-paid sites get paid when the ad is retrieved from the server and displayed on the screen. Interestingly, they are not paid based on users actually seeing the ad -- just when it is retrieved from the server.  If the users never scroll down to see it, the advertiser still pays the site even though they didn't get any value from the ad. They are will to pay more, however, if the site can guarantee that the ad will be seen. 

Now, a site could generate a lot of ad revenue by having 10 pop-up ads appear after every click, but that would disrupt and annoy the customer into never returning to the site.  Or the site could make a lot of money by displaying a dozen or so ads at the bottom of each page, underneath the content in a place where users would be unlikely to scroll.  But that would quickly piss-off the advertisers and cause them to put their money elsewhere. 

So, the question then becomes, is there a balance that can be reached?  How do designers measure whether a specific design is within the tolerable limits of both the users and the advertisers, while still generating the required revenue?  

And what is the role of the content in this situation?  We know that if you have content that is highly desirable and unavailable from any other source, users are more likely to put up with annoyances than for content that is commonly available.  How do you measure where your content lies on this spectrum, so you know what the boundaries of tolerability are?

These are the primary questions we're interested in around the advertiser-paid space. 

And we've got lots more stuff happening in the areas of branding and web-based applications.  We'll have something to talk about soon. 


Privately you told me that you can now directly tie design elements to bottom line effects in certain contexts. Can you explain this? What implications does this have on web design? 

Yes, well, this is really exciting stuff. While we were conducting our e-commerce research, we noticed a cool phenomenon which we could take serious advantage of.  To understand this phenomenon, I need to give you a little background on the experiment the was the basis of the research -- something we call the 7-11 Milk Experiment. 

Imagine we had a way to instantly identify when someone has run out of milk.  We pick them up in our car and drive them to the nearest 7-11.  And just to make sure everything goes smoothly, we give them the money to buy milk. How likely is it that that 7-11 will sell milk in that context?  Probably close to 100%, right? 

Well, that's what we did online.  We identified people who needed products, we brought them to sites that had those products, then we gave them the money to buy the products.  The result: people only purchased 30% of the time.  Much worse than the 100% we would expect to see. 

Now, what's important here is that we've subsetted the shopping experience.  We're not trying to measure every possible combination of shopping, just a single subset: people who know what they want and have the cash to buy it.  We should see very high purchase rates in this scenario. In fact, this subset represents the most likely scenario of purchase. 

Now, here's where the real breakthrough comes in: we found that, during the study, people made purchases that weren't on their shopping list. That fact alone isn't that interesting, what's interesting is the reason: it was because of the design of the site. 

It turns out that only 8% of the impulse purchases we saw were because of price or special promotions.  The rest were because of the design of the site.  And it wasn't small potatoes either.  The impulse purchases averaged 42% of the revenue from those sites. (To put this in perspective, Staples.com generated $451 million last year. 42% of that revenue is $189 million. I wouldn't mind having that kind of revenue.) 

How did the design of the site affect impulse purchases?  Well, as I mentioned before, the categories on the site played a huge role.  On the home page of a retail site, you basically have 3 choices when shopping: 1) use the search engine, 2) choose a featured product, or 3) use the category links provided.  In our study, if you chose option 3, you were three times more likely to make an impulse purchase than if you didn't.  

(Oh, by the way, not a single impulse purchase came from choosing option #2.  This finding puts into question the practice of dedicating lots of home page real estate to featured products.) 

We found that subtle differences in the design of the home page would be a huge predictor in the impulse sales on the site.  Designs that worked hard to guide users into the categories were far more successful in generating this revenue than sites that essentially scared people into using the search engine.  (Our whitepaper on impulse purchases goes into far more detail on this.) 

Well, it turns out that we've identified about 40 of these "predictors" of revenue for retail sites.  And this gives us a huge new capability: we can now analyze sites for the presence or absence of these predictors and use that to help measure and diagnose problems on the site. 

To that end, we've just launched a new service, which we call our Compelled-Shopping Analysis (CSA).  This service, which is currently available for retailers (though we're working on expanding it to other types of e-commerce and non-e-commerce sites), allows us to benchmark the design of the site based on the revenue it produces and how the users react to it. 

CSA is different from other types of measures, like SUMMI, QUIS, or WAMMI because it doesn't base it's metrics on the opinions of users -- it actually measures the success people have at shopping. We don't care if someone loves the site, but doesn't buy.  We're only interested in the shopping experience. 

So, with CSA we're now measuring how sites are actually doing.  And we're building up a library of data that allows us to identify best practices amongst retailers in a variety of contexts.  This is extremely exciting because we really get the full advantage of our research capabilities. 

(If you are interested in these best practices, you might check out our course, Designing for Dollars: Discover How People Buy Online. We're offering this course September 24-25 in San Francisco, MA.)


Challenging the Usability Community

I have made it clear that I think usability will "go away" (see Trouble in Paradise: Problems Facing the Usability Community). I don't literally think that the tools, methods, and techniques will go away, but I do think the profession is facing some very serious challenges. How do you feel about the usability community? What would you change? What is your open message to the usability profession? 

As I've indicated above, I agree that there are some real challenges. I think a huge problem with the Usability Community is that we're trying to be a "community".  Usability is an attribute -- some things are more usable, some things are less.  

There are other attributes, but they don't have communities.  Fun doesn't have a community.  Integrity doesn't have a community.  Why do we insist on having a community? 

The companies that are doing the best at usability don't seem to have a community.  Instead, they have a culture.  A culture where people perform the activities needed to create usable sites just because the culture dictates that's how things are done. 

We need to understand how to build more cultures like these.  We need to understand how they've come about, how they flourish, and how they become interwoven into the patch quilt of the business. This is where the advances will come from -- not by promoting a community. 


In light of the previous question, why weren't you at the (Usability Professionals' Association) UPA conference this year? You are well known, you write well, and people seem to like your work. What is the scoop? People are curious! 

Well...  I did submit a couple of proposals.  They were rejected.  No reason was given for the rejection, so I couldn't tell you why.  It's possible that I'm a crummy author or don't have anything new or novel to say to the UPA community, despite the fact that I've spoken almost every year in the past and I'm often one of the highest rated speakers at the conference. 

After I complained to the conference chairs about my rejection, they did offer me a speaking slot -- but unfortunately it was too late in the process and I was unable to accept. 

Despite what I've written above, I'm a big supporter of UPA. My company was a Gold Sponsor this year, and a Silver sponsor the previous year.  My staff serve on UPA committees.  We recommend the UPA to our clients. 

It's too bad.  I would've like to have been there.  I guess there's always next year.


Wrap Up, Wrap Down 

What is tetrahydrofuran? How did you figure it out? What tools did you use? How did your usability knowledge and experience influence how you figured it out? 

Long day, huh John? 


Most people I know are trying to lose weight and get in better shape. Does this have any impact on the usability profession? Does it impact web design? When you think about it, these questions aren't so strange. We are human. Should web designers seek to understand human emotions and desires? What is your perspective? 

Definitely, a long day. 


What is your favorite usability quote?

I've grown very fond of this quote lately, for whom I don't know who to attribute it to: 

Good Judgment comes from Experience. Experience comes from Bad Judgment. 


Comments? Shameless plugs? Wisdom? 

Man, don't you think I've said enough?  I've certainly plugged almost everything I can.  And I doubt I can impart any more wisdom. 

I say we're done for now. 

Thanks. 

Jared

- o - o - o -
Jared M. Spool
User Interface Engineering
242 Neck Road
Bradford, MA 01835 USA
(978) 374-8300  fax: (978) 374-9175
jspool@uie.com   http://www.uie.com


Editor's Comments

Excellent! What else can I say? 

-- John S. Rhodes 


What next?

  • Feel free to create a link to this interview.

 

If you want to know when new interviews go online,
subscribe to the WebWord.com Usability Newsletter!

 

Home | Moving WebWord | Cool Books | Hot Web Sites
Reports
Newsletter Archive | Interviews | About WebWord.com


Contact John S. Rhodes, WebWord.com Editor and Webmaster

URL: http://webword.com/interviews/spool2.html

© 2001 by WebWord.com. All rights reserved.
Do not reproduce or redistribute any material from this document,
in whole or in part, without explicit written permission from WebWord.com.