Last year, I gave a presentation at MeshU that took a behind the scenes look at how we arrive at design decisions. We’ve since taken clients through variations of this presentation, which is always evolving because it corresponds to such a perennial and fundamental question in our field.

I’ve always appreciated it when fellow UXD practitioners talk candidly about what works and doesn’t work for them. Insights and methods pioneered by others have helped us improve our process here at T+L a lot. Maybe our spin on things will be helpful to you and your team.

Here’s the presentation

During a recent pitch, one of our clients asked us to come back and explain how we “bridge art and science” when making design decisions. I think this is an intriguing way to pose the question (and it speaks to how clients are becoming more engaged and sophisticated in what they’re looking for from a design shop).

Let’s start by defining our terms (Slides 1 through 20). Here are a few conventional ways to differentiate between art and science when it comes to user experience design (please note that I’m speaking in stereotypical terms here): In terms of focus, an artistic process is concerned with issues of look-and-feel, whereas a scientific approach focuses on deeper, more systematic issues like underlying architecture. In terms of methodology, art relies on intuition and experience, whereas science depends on rigorous investigation and analysis. In terms of validation, an art-led process often rests on subjective or personal evaluation, whereas a process that’s grounded in science relies on rigorous testing using quantitative metrics.

Although perhaps a helpful starting point, this model doesn’t tell the whole story. In fact, we can flip things around (Slide 8) and take a look at the other side of the coin: over the last 6 years, some of the most influential academic research in HCI has demonstrated the importance of emotional, relational and aesthetic affordances in design; when you factor-in experience, intuition is often an important check on incomplete or distorted data that might otherwise mislead; and there is certainly an “art” to testing in appropriate and productive ways.

In other words, rather than this binary opposition of art vs. science, a better model is perhaps something more like a continuum. There are two extremes we want to avoid. At one end, we have deterministic design—the idea that we can be entirely predictive, almost in a Newtonian physics kind of sense, mapping out causality for everything, no matter how complex or layered. At the other end we have open-ended design—where decisions are more or less arbitrary. The first alternative has turned out to be pretty unrealistic; the second is just a cop-out. We’ve tried to strike a balance between these two extremes and have gravitated towards evidence-based design as our happy medium.

Evidence-based design (EBD) is a term that comes from the medical world. Evidence-based medicine is the “conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.” Transposed into our context, we like the idea of EBD because it acknowledges the fundamental reality that design is about making choices, and that the goal is to do this on an informed basis. We don’t have to maintain the pretence of a magical process that guarantees optimal results, but neither are we given free license to design whatever we want: our decisions become accountable to the best available evidence whenever possible.

EBD is not just about gathering up data at the beginning of a project: it’s about infusing design decisions with data-driven insights throughout the entire process. Slides 16-20 visualize this goal in terms of a typical agency process (actually, our process). In practice, there’s typically a gap between the define (or research or discovery or whatever you want to call it) and design stages. This is a fundamental challenge—you can have the best researchers on the planet, but if their observations and insights don’t carry over to impact and influence design decisions they’ll do you little good.

So how do we bridge the gap? For us, the secret has been to build our process around a rhythm of open exploration and refinement. Slides 22-28 visualize this approach (thanks to Brendan Schauer et al. from Adaptive Path for the inspiration). We oscillate between a ‘go wide’ mode where we investigate, explore and experiment, and a ‘refine down’ mode where we focus and prioritize. When this happens over and over again, we get these critical points of inflection that keep us grounded (evidence-based) and moving forward (design).

OK, let’s take a look at a real life example. Slides 29-48 provide a glimpse behind the scenes of our work on thestar.com redesign. We start by going wide and essentially scavenge for as many design inputs as possible. These vary project by project, but at a minimum we’ll need to get a handle on demographics, psychographics, behavioural profiles for target users; a concrete understanding of what is and isn’t working on the current site; business goals, needs & requirements; and a comprehensive competitive analysis.

All we have at this point is raw data—we need to turn this stuff into evidence and insights we can actually use. Borrowing from Subject to Change, we believe that inputs become evidence when they are durable (you can kick them around and mash them up), actionable (there are clear implications for design), and impactful (they actually start to change the way designers think about the problem itself). So how do you get there? This is where the art part comes in…

In the beginning, we aggregate stuff in one place, organize loosely and share it among the team. Tools like wikis are good for this exercise. Next we start to look for patterns and find corroborating data—multiple pieces that tell a coherent story. We then construct and test narratives for things we heard from stakeholders. These narratives help us get at the context behind the requirements. We then create focusers—models (visual, conceptual, personal etc.) that help us focus and filter. Two focusers that we often use are personas (architypal users) and design principles (mini mission statements). Focusers are important because they allow us to find the signal in the noise, develop a common language and conceptual framework, and make our assumptions explicit. Because they’re so elemental, they can inform design decisions on a case-by-case basis throughout the entire process.

Sketching has always been central to our process, but we’re starting to formalize this step and make it more collaborative and inclusive. In the case of The Star, we generated about 150 initial sketches (going wide) for 10 key templates (which we refined down in detailed IA). Prototyping is another technique that allows us to explore and test ideas and assumptions. Critical and complex elements in the UI make great candidates here.

Finally, we need to develop a plan of attack for knowledge gaps or areas of uncertainty. For example, with The Star we noticed some anomalies in the analytics data. When we took a look at the competitive set (with a little help from Image Spark), we didn’t see very much alignment in how other media outlets were using certain areas on the page. So, we developed a fairly wide-range of alternative modules for split/multivariate testing.

So there you have it: a relatively geeky look at the case for, and our approach to, evidence-based design. I hope that some of these ideas and examples ring true with you. What do you think? I’d love to hear what others are doing in their design practices and processes.

Last year, when Brandon essayed about Sketchboards on the Adaptive Path blog, Derek and I became instant fans.  Since then we’ve played around with variations of the method at T+L—sort of hedging it with our existing process—so I was really looking forward to this session.

First we talked about the sketch part in 2 stages:

1. Exploratory sketches: quickly saturate the design space by generating a number of rough options. Use word-play, inspiration libraries (hurry-up, imgspark!), conceptual models to drive this process.
2. Refinement sketches: take the most promising ideas and add a more detail and weighting through different types of marks, labeling, etc.

 
Then we talked about the board part:

1. Get a large sheet of paper or whatever you’re going to use to lay things out on and give it some structure: for example, use stages in a user flow.
2. Add referential inputs like personas, scenarios, requirements, design criteria, inspiration, etc.
3. Lay-out sketches, review, annotate, iterate and decide what you’re going to prioritize in wireframes.

A few personal reactions:

• Although things can go at a fairly fast pace, you’re definitely trading scope for time.  Brandon proposed a 5 day sprint that would maybe output 3 or 4 wireframe-able templates.
• I was impressed with how natural and intuitive the whole process felt. Sketching happens on an informal basis anyway—this just put’s some structure around it and invites dialog.
• The main thing I love about Sketchboards is that they encourage and facilitate conceptual externalization. They force you to get your ideas out and onto the page, making them tangible, accessible to all and amenable to critique (before you fall in love with them!).

Swimlanes is an early-process documentation method created by Yvone Shek and the folks at nForm.  As the image above demonstrates (here’s a closer look), multiple perspectives on a given use-case or design scenario are laid out in separate tracks, or “swim-lanes.” The idea is to capture and visualize implications of high-level requirements over time and in a parallel fashion. 

This is good because multiple stakeholders (business people, designers, project mangers, technologists) can see and give feedback on what they need to make happen/accomodate to, leading to a potentially more balanced and inclusive discussion.

To my mind, the key challenge here is making such a sophistic and integrated document like this flexible and agile. (We talked about the speed factor a lot, but I’m more concerned about the rigidity factor.) For more details on Swimlanes, check out Yvone’s original post here.

Next up: Five Sketches, Or Else! design process

Photo credit: mastermaq

Had the opportunity to attend this month’s Torchi event, featuring two guests from Microsoft. Lisa Anderson, MS Surface User Experience Director, talked about the fundamental shift from command line interfaces to GUIs, to what she called Natural User Interfaces (NUIs, I guess). Some really cool theoretical thinking in her presentation but it would have also been nice to get into a more concrete discussion about where they’re headed with Surface. I guess we’ll just have to wait for the SDK and interface guidelines spec. Lisa talked about how they’re trying to make the interface “disappear” by leveraging intuition and allowing interaction through direct manipulation. But there must be at least some standardized interface elements built into the Surface and I’m really keen to learn what these are and how they work.

Jansen Harris’ discussion about his work heading up the Office User Experience Team was also really interesting and much more concrete. No matter how you feel about Microsoft’s past performance when it comes to innovating the user experience, it’s hard to deny that they did a great job with the latest Office suite (and this has been borne out both critically and in terms of revenues generated). 3 things Jansen mentioned that stuck with me:

1. Early-on they established a set of design tenets to guide decisions at a high level over the course of the project. One that really stood out to me was “Straight-forward is better than clever.” I know I’ve often been guilty of breaking that rule :)
2. Developers took the first couple of months basically rebuilding the programatic foundations for Office specifically so that the UX team could afford to make mistakes and revise as they progressed along.
3. The Office 2007 team was comprised entirely of people who passionately believed in the project. If your heart wasn’t in it (and there were apparently those who fundamentally disagreed with the idea that MS could or should re-engineer the UX for Office), they found something else in the company for you to work on.

In general, I was impressed by how much grunt-work the team put into validation and evaluation throughout the design process. Rather than testing for testing’s sake or gathering data just to justify pre-ordained decisions, they used evidence to answer very focused, well-defined questions.

Quick plug: ToRCHI events happen monthly and are usually worth coming out to. Great guests and good discussion.

Thinking from the potential customer’s perspective is a Zen-like exercise. It requires you to drop your problem-solving role completely, and spend time—at least two hours, or maybe two weeks—engrossed in the world of this person. Stop thinking of them as a “user” of the thing you provide. Think about how and why they accomplish what they want to get done, not how or why they might use your stuff.

While there are lots of posts I’ve read throughout the years that champion designing with empathy, Young is able to do it with some great examples.

Read the complete article here

Expanding the horizons and expanding the parameters,
Expanding the rhymes of sucker MC amateurs

~ The Beastie Boys, The Sounds of Science, 1989

I’ve always thought it’d be cool to be a scientist—a real scientist, with the lab coat and the beakers and whatnot. You could win friends and influence people (and pwn enemies) anytime, anywhere.

Sometimes I get the impression that I share this secret ambition with web and UI designers at large. After all, making design decisions that are “only” based in a team’s collective experience, thoughtfulness, observation, trial and error, etc. leaves those decisions open to critique. But grounding/couching your work in some sort of rigorous-sounding, quantifiable, testable result: that’s science…you can’t beat that!

Now, don’t get me wrong. Testing your design in appropriate ways can be invaluable (I’ll talk more about this in another post). But I really take issue with the idea that User Testing, per se, leads to great design. I’ve seen just the opposite happen.

I think this is the case because we’re trying to appropriate a tool that loses its power and actually becomes counter-productive when used outside of the context it was designed for. We’re borrowing from the experimental design paradigm in cognitive science, which is a scholarly discipline closely related to the applied field of HCI. But have you ever seen an actual experiment in cog sci? When I performed and ran a few of these back in school, they usually worked something like this:

• Sit someone down in front of a computer in a small room (maybe there’s a video camera or some kind of monitoring equipment set up)
• Have them stare at a dot in the middle of the computer monitor and hit A, B, or C as soon as they recognize some sort of visual stimuli presented to the screen
• Record the timing and number of errors
• Repeat the experiment with a bunch of different people, changing up one of two “explanatory variables” that you’ve guessed will have an impact on performance.
• Run statistical analyses on the results and use these to draw conclusions.

Now this kind of experiment is obviously very narrowly focused in its scope, and necessarily so. There are a bunch of reasons why, but the two main ones are these: reliability and validity. For an experiment to be reliable means that repeating it over and over again yields the same result. For an experiment to be valid means that it gives cogent answers (even if they’re only partial answers) to the questions you asked in the first place.

Interactive experiences like websites are complex phenomena. They don’t naturally lend themselves to the kind of experimental protocol described above because user performance varies greatly from person to person and from session to session. There are so many potentially confounding variables in play that reliability suffers. (The model experiment above tries to eliminate this problem by paring down the user’s task to a few basic actions.) You’re measuring 10th order effects and it becomes nearly impossible to establish causal connections between design characteristics and user performance.

If we do streamline things so that we’re just measuring one or two explanatory variables vs. 50 (say, by temporarily removing elements from the design), the experiment becomes more reliable, but less valid. That is to say, the results—while repeatable—can’t really be generalized to answer the type of questions that we want to ask in the first place (questions like ‘is this design easy to understand and use’), because we’re not truly testing the design.

Sometimes usability researchers will employ something called a “talk aloud protocol” to try and tap into the cognitive processes underlying user performance in a given scenario. This involves asking users who are testing a given design to explain what’s going through their heads as they move through some sort of task flow.

Again, I have real problems with this pseudo scientific approach to evidence-based design. For one, the act of talking about what you’re doing changes the nature of that experience. But more importantly, most people can’t accurately report on why they do what they do—that’s why there are such a fields of inquiry as cognitive science and psychology in the first place!

I don’t want to be overly cynical here, but do want to caution usability professionals and interaction designers in general: user testing can be helpful, but also misleading. It can also be a powerful political gambit or rhetorical expedient. If you’re going to test something, make sure you’re asking the right kinds of questions. User testing can be used to tune or optimize design, but cannot and should not substitute for creativity or thoughtful trial and error.