Limited access to television, icy temperatures, and no amount of distance, held anyone back from watching Tuesday’s inauguration of the 44th president, Barack Obama. Several sites proudly played host to the most web-friendly presidential inauguration to date, spiking internet traffic levels to 54% above normal.

CNN streamed the live ceremony alongside an impressive Facebook tool, which let users update their status, and toggle between viewing live status updates of their own friends, or the general public (consisting purely of Obamania). You could call it a success, an average of 4,000 users updated their status every minute, and CNN reported over 1.5 million status updates through their portal alone.

Meanwhile, not one minute was wasted updating the White House website, literally. By 12:01, the site was Obamafied, and the blog page already flourishing with updates and recaps.

At the Obama-friendly T+L, we were lucky enough to get a mid-morning break to pile into the boardroom and watch history unfold. It was especially exciting for us that the ceremony that captivated millions of people around the world had a much higher priority online, compared to in the past.

We even had some friends from the Globe and Mail join us for a story.

I’m a third of the way through Steve Mulder and Ziv Yaar’s insightful guide to to creating personas, The User is Always Right. As part of that process the authors provide concrete examples and rules for conducting user research for the initial phase of design for web.

I’m finding that so far, along with some great nuggets of insight, a good portion of Mulder and Yaar’s process refreshingly mirror our own here.

“…Traditional usability testing often isn’t as helpful for creating personas…When you give specific tasks to users, the test becomes about the user goals you have chosen instead of the users’ unique goals.”

One example of that is the caution the authors’ express when using usability tests as a means of qualitative research for persona development. In trying to understand users’ goals to ultimately craft the best experience, traditional usability testing tells us successes or failures of a given task at hand. Rather than uncovering what we don’t know, this type of research usually tests what we suppose is right or wrong with what exists versus informing what needs to exist.

That being said, user testing is effective when the conditions are right. Field, ethnographic and quantitative research are other means of research that provide great insights in our experience and that Mulder and Yaar speak to at length in the book.

If you’re interested, the pitfalls of user testing is something T+L have always been cautious of and something David has spoken about in detail more recently.

I’m looking forward to finishing the book and providing more feedback as to how it compares and contrasts to our process here.

Had the opportunity to attend this month’s Torchi event, featuring two guests from Microsoft. Lisa Anderson, MS Surface User Experience Director, talked about the fundamental shift from command line interfaces to GUIs, to what she called Natural User Interfaces (NUIs, I guess). Some really cool theoretical thinking in her presentation but it would have also been nice to get into a more concrete discussion about where they’re headed with Surface. I guess we’ll just have to wait for the SDK and interface guidelines spec. Lisa talked about how they’re trying to make the interface “disappear” by leveraging intuition and allowing interaction through direct manipulation. But there must be at least some standardized interface elements built into the Surface and I’m really keen to learn what these are and how they work.

Jansen Harris’ discussion about his work heading up the Office User Experience Team was also really interesting and much more concrete. No matter how you feel about Microsoft’s past performance when it comes to innovating the user experience, it’s hard to deny that they did a great job with the latest Office suite (and this has been borne out both critically and in terms of revenues generated). 3 things Jansen mentioned that stuck with me:

1. Early-on they established a set of design tenets to guide decisions at a high level over the course of the project. One that really stood out to me was “Straight-forward is better than clever.” I know I’ve often been guilty of breaking that rule :)
2. Developers took the first couple of months basically rebuilding the programatic foundations for Office specifically so that the UX team could afford to make mistakes and revise as they progressed along.
3. The Office 2007 team was comprised entirely of people who passionately believed in the project. If your heart wasn’t in it (and there were apparently those who fundamentally disagreed with the idea that MS could or should re-engineer the UX for Office), they found something else in the company for you to work on.

In general, I was impressed by how much grunt-work the team put into validation and evaluation throughout the design process. Rather than testing for testing’s sake or gathering data just to justify pre-ordained decisions, they used evidence to answer very focused, well-defined questions.

Quick plug: ToRCHI events happen monthly and are usually worth coming out to. Great guests and good discussion.

The team at Clearleft launched a new application called Silverback. Essentially, it’s user testing software that allows you to capture screen actions as well as audio and video of participants.

There’s a free trial, or you can just check out the video demo to get a sense of it’s capabilities. Pretty slick.

I can imagine they had some challenges with this application. Not because of the complexities involved in designing or building it, but because doing something outside of your core business practice takes serious commitment (they’re an experience design firm, not a software developer). No doubt the two have similarities but It mustn’t have been easy leaving paying jobs on the table. I applaud them for making the leap – it’s a great idea and a solid execution.

We’ve gone down these roads roads before with little success. Our PVR report was well received but when we tried to actually build it we couldn’t commit the resources to it. We did it a second time with Paruba. Again, long term commitment was an issue. We’re actually about to venture into it again, albeit with a different approach. Hopefully with the same success as Clearleft.

Expanding the horizons and expanding the parameters,
Expanding the rhymes of sucker MC amateurs

~ The Beastie Boys, The Sounds of Science, 1989

I’ve always thought it’d be cool to be a scientist—a real scientist, with the lab coat and the beakers and whatnot. You could win friends and influence people (and pwn enemies) anytime, anywhere.

Sometimes I get the impression that I share this secret ambition with web and UI designers at large. After all, making design decisions that are “only” based in a team’s collective experience, thoughtfulness, observation, trial and error, etc. leaves those decisions open to critique. But grounding/couching your work in some sort of rigorous-sounding, quantifiable, testable result: that’s science…you can’t beat that!

Now, don’t get me wrong. Testing your design in appropriate ways can be invaluable (I’ll talk more about this in another post). But I really take issue with the idea that User Testing, per se, leads to great design. I’ve seen just the opposite happen.

I think this is the case because we’re trying to appropriate a tool that loses its power and actually becomes counter-productive when used outside of the context it was designed for. We’re borrowing from the experimental design paradigm in cognitive science, which is a scholarly discipline closely related to the applied field of HCI. But have you ever seen an actual experiment in cog sci? When I performed and ran a few of these back in school, they usually worked something like this:

• Sit someone down in front of a computer in a small room (maybe there’s a video camera or some kind of monitoring equipment set up)
• Have them stare at a dot in the middle of the computer monitor and hit A, B, or C as soon as they recognize some sort of visual stimuli presented to the screen
• Record the timing and number of errors
• Repeat the experiment with a bunch of different people, changing up one of two “explanatory variables” that you’ve guessed will have an impact on performance.
• Run statistical analyses on the results and use these to draw conclusions.

Now this kind of experiment is obviously very narrowly focused in its scope, and necessarily so. There are a bunch of reasons why, but the two main ones are these: reliability and validity. For an experiment to be reliable means that repeating it over and over again yields the same result. For an experiment to be valid means that it gives cogent answers (even if they’re only partial answers) to the questions you asked in the first place.

Interactive experiences like websites are complex phenomena. They don’t naturally lend themselves to the kind of experimental protocol described above because user performance varies greatly from person to person and from session to session. There are so many potentially confounding variables in play that reliability suffers. (The model experiment above tries to eliminate this problem by paring down the user’s task to a few basic actions.) You’re measuring 10th order effects and it becomes nearly impossible to establish causal connections between design characteristics and user performance.

If we do streamline things so that we’re just measuring one or two explanatory variables vs. 50 (say, by temporarily removing elements from the design), the experiment becomes more reliable, but less valid. That is to say, the results—while repeatable—can’t really be generalized to answer the type of questions that we want to ask in the first place (questions like ‘is this design easy to understand and use’), because we’re not truly testing the design.

Sometimes usability researchers will employ something called a “talk aloud protocol” to try and tap into the cognitive processes underlying user performance in a given scenario. This involves asking users who are testing a given design to explain what’s going through their heads as they move through some sort of task flow.

Again, I have real problems with this pseudo scientific approach to evidence-based design. For one, the act of talking about what you’re doing changes the nature of that experience. But more importantly, most people can’t accurately report on why they do what they do—that’s why there are such a fields of inquiry as cognitive science and psychology in the first place!

I don’t want to be overly cynical here, but do want to caution usability professionals and interaction designers in general: user testing can be helpful, but also misleading. It can also be a powerful political gambit or rhetorical expedient. If you’re going to test something, make sure you’re asking the right kinds of questions. User testing can be used to tune or optimize design, but cannot and should not substitute for creativity or thoughtful trial and error.

Andy Rutledge takes a very critical look at the Poynter Institute’s latest eye tracking study. The Poynter Institute is one of the more established, and generally, acclaimed sources for online journalism.

For the past few years we have been trying to convince clients the “user testing” isn’t all its cracked up to be. Christopher Fahey has a great 5 part series called “User Research Smoke and Mirrors” pts 1 ,2 ,3 ,4 ,5 .

We often mention to clients that we see user research being used inside organizations as a political tool. A way to get things moved forward “the research told us” is easier to sell than “this is the right thing to do”.

We often hear clients talk about the “lab” where they are ushered into a sitting room with one way glass. The theater of it is very impressive. But it’s like going to the zoo to study gorillas.

I don’t deny that things like persona rooms and labs have a significant impact on clients because they look so “scientific”.

We have had great success with clients claiming the emporer has no clothes on this subject. We believe in research and science as long as it’s valid. There is a time and a place for it.

But most clients should save their money and trust the firms they hire to guide them through a design process.

Eyetracking is becoming somewhat of a hot topic in the UX blogsphere these days. It’s got just the right feel of objective validity mixed with technical novelty. There’s also some really cool visualization techniques out there to help us make sense of the raw data.

Some have gotten right into the action, turning eyetracking data into readymade insights for interaction designers.

Jared Spool recently weighed in on the subject on his company’s blog, articulating what we think is a more balanced and down-to-earth view of the situation.

New analytical paradigms often have an instant appeal in our industry—I suppose, because things seem more scientific. It’s worth bearing in mind that same problem always crops up when we overlook the gap between experimental disciplines like cognitive psychology (where you’re looking at first order effects) and applied disciplines like interface design (Nth order effects). There’s such a huge leap there!

Take Fitts’ law —it’s super sexy because it’s predictive and mathematically well-formed; but the design implications are limited: put stuff on the periphery of the display (where the hit area is effectively infinite) to make it easier to acquire targets. This is why the locking the toolbar to the top of the screen (Mac OS) is more efficient than locking it to the window (MS Windows). Cool insight, but did we really need an equation to figure this out?