If we’re designing an online e-com solution where there are tons of best practices to draw from (perhaps some of these were originally derived from user testing), there’s no point to reinvent the wheel. Experienced designers working iteratively and in a scenario-driven fashion should be up to the task.

If we’re designing something more specialized-say, an expert system for streamlining internal processes at work-usability testing doesn’t go far enough. What we need here is a more participatory approach in order to fully appreciate the situational factors that will ultimately determine whether or not the design solution succeeds. There’s a ton of domain knowledge that we need to tap into, and testing users isn’t the way to do it; collaborating with them is.

If we’re designing something genuinely novel-for example, a new scheme for visualizing or manipulating information-here’s where I think user testing makes sense. We use it to validate our assumptions, suggest new directions for further exploration, and generate/codify new knowledge that maybe we’ll be able to use for free next time!

A few years ago user testing was extremely valuable because we were still learning a lot about how people used the Web. Today we have a few factors that make user testing not the priority it was.

1) We have a more experienced Web user population.
2) We have identified a lot of heuristics and best practices
3) Designers have gotten better at intuitively understanding what good Web design is versus bad design

I believe that for many projects just using best practices and user centered design practices (personas, some third party research) you can deliver a very good product.

In the Web 1.0 delivery mechanism we used to take 18 months to build large scale sites. Refreshes would occur in another 18 months. In that world testing was essential.

Modern web development is much more agile and nimble so the ability to optimize and change is much more realistic. We are proponents of the idea of release early and release often. In that world user testing occurs in the real world, not a lab.

The seomoz post is a bit too angry in tone, but I agree with it. Basically, he is saying “too many cooks in the kitchen spoils the broth”.

This observation seems to fall into a similar, albeit less researched, vein as your comment above.
http://www.seomoz.org/blogdetail.php?ID=1230

Do you guys find yourselves fighting this fight a lot?

It comes from the gut, it comes from learning how to read and interpret people’s reaction. The output is subjective input and advice, a sort of “if you wan to do it well, there’s a darn good chance this will do the job”.

We sometimes get usability studies that give absolute recommendations that are so biased as to be dangerous, but since “testing told us” it must be true. That is what is bad.

Only when talking to customers directly do I discover what’s actually important to regular folk. They have a whole different perspective on how things should work, look and behave; all of which are essential to enhancing the system.

I agree that user experience and usability testing are highly political vehicles. I know for us that these deliverables will give clients the firepower they need to sell the idea up the food-chain. Getting CEO buy-in is huge.

There’s also something to be said for getting customers involved in the testing process. It can really build buzz and positive feelings towards the organization, that they’re taking steps in the right direction.

Finally, I much prefer remote online testing versus lab testing. Not being face to face has some disadvantages, but they’re outweighed by being able to do screen sharing (via a product called Ethnio) with video and voice recording. Cheaper, faster and more comfortable for the participants too.