Our Blog
The “science” of user research
For the past few years we have been trying to convince clients the “user testing” isn’t all its cracked up to be. Christopher Fahey has a great 5 part series called “User Research Smoke and Mirrors” pts 1 ,2 ,3 ,4 ,5 .
We often mention to clients that we see user research being used inside organizations as a political tool. A way to get things moved forward “the research told us” is easier to sell than “this is the right thing to do”.
We often hear clients talk about the “lab” where they are ushered into a sitting room with one way glass. The theater of it is very impressive. But it’s like going to the zoo to study gorillas.
I don’t deny that things like persona rooms and labs have a significant impact on clients because they look so “scientific”.
We have had great success with clients claiming the emporer has no clothes on this subject. We believe in research and science as long as it’s valid. There is a time and a place for it.
But most clients should save their money and trust the firms they hire to guide them through a design process.
I’ve found that expert design analysis/assessment isn’t enough to provide the insight required when doing a large scale refinement or re-design project. I feel like I’m operating in a bubble when I just talk to the customer and my colleagues about possible improvements.
Only when talking to customers directly do I discover what’s actually important to regular folk. They have a whole different perspective on how things should work, look and behave; all of which are essential to enhancing the system.
I agree that user experience and usability testing are highly political vehicles. I know for us that these deliverables will give clients the firepower they need to sell the idea up the food-chain. Getting CEO buy-in is huge.
There’s also something to be said for getting customers involved in the testing process. It can really build buzz and positive feelings towards the organization, that they’re taking steps in the right direction.
Finally, I much prefer remote online testing versus lab testing. Not being face to face has some disadvantages, but they’re outweighed by being able to do screen sharing (via a product called Ethnio) with video and voice recording. Cheaper, faster and more comfortable for the participants too.
I totally agree. What I am reacting to is the improper use of testing and data. Also this notion that testing is scientific or absolute. It is a great input just like 3rd party research and personal experience.
We sometimes get usability studies that give absolute recommendations that are so biased as to be dangerous, but since “testing told us” it must be true. That is what is bad.
Yep, for sure. This type of testing is by no means scientific or absolute, at least in the traditional sense.
It comes from the gut, it comes from learning how to read and interpret people’s reaction. The output is subjective input and advice, a sort of “if you wan to do it well, there’s a darn good chance this will do the job”.
Sorry I put this on the wrong thread:
This observation seems to fall into a similar, albeit less researched, vein as your comment above.
http://www.seomoz.org/blogdetail.php?ID=1230
Do you guys find yourselves fighting this fight a lot?
We have found that many clients have a requirement to do user testing as part of the project, which is great. But many times its not needed.
A few years ago user testing was extremely valuable because we were still learning a lot about how people used the Web. Today we have a few factors that make user testing not the priority it was.
1) We have a more experienced Web user population.
2) We have identified a lot of heuristics and best practices
3) Designers have gotten better at intuitively understanding what good Web design is versus bad design
I believe that for many projects just using best practices and user centered design practices (personas, some third party research) you can deliver a very good product.
In the Web 1.0 delivery mechanism we used to take 18 months to build large scale sites. Refreshes would occur in another 18 months. In that world testing was essential.
Modern web development is much more agile and nimble so the ability to optimize and change is much more realistic. We are proponents of the idea of release early and release often. In that world user testing occurs in the real world, not a lab.
The seomoz post is a bit too angry in tone, but I agree with it. Basically, he is saying “too many cooks in the kitchen spoils the broth”.
To me, the key to this discussion is context. The value of usability (or any methodological tool for that matter) depends on what we’re designing, who will use it, how they’ll use it, etc. Seems obvious, but I find this perspective often gets lost.
If we’re designing an online e-com solution where there are tons of best practices to draw from (perhaps some of these were originally derived from user testing), there’s no point to reinvent the wheel. Experienced designers working iteratively and in a scenario-driven fashion should be up to the task.
If we’re designing something more specialized-say, an expert system for streamlining internal processes at work-usability testing doesn’t go far enough. What we need here is a more participatory approach in order to fully appreciate the situational factors that will ultimately determine whether or not the design solution succeeds. There’s a ton of domain knowledge that we need to tap into, and testing users isn’t the way to do it; collaborating with them is.
If we’re designing something genuinely novel-for example, a new scheme for visualizing or manipulating information-here’s where I think user testing makes sense. We use it to validate our assumptions, suggest new directions for further exploration, and generate/codify new knowledge that maybe we’ll be able to use for free next time!