13 September 2010 by Barbara Ballard
Everything has changed! It’s a new world! We have to re-learn everything about design!
So goes the tone of the majority of articles and presentations about mobile user experience that I’ve seen or read in the past three years (at least). Mostly, I ignore them – the actual content is good so I can live with a little grandstanding. Yes, context is crucial (we’ve been saying this for over a decade). Yes, device capabilities and interaction should influence design (again, I started talking about this in 1999).
But like some solid ideas get re-invented repeatedly, so do some not-so-solid ideas. Long-term mobile usability practitioners know that people interact differently with a device in their hand than with the same web site with the same pixel count on a computer screen. And this has been true for longer than we have had access to sensors like cameras, GPS, and accelerometers.
So it pains me to see an otherwise good article, like usability for mobile devices over at UX Matters, assert that a simulator or emulator is a good way to perform a usability test. Actually, the authors did this immediately after suggesting that testing in context is also good. So let’s set the record straight.
Guidelines for usability testing web sites and apps on mobile devices
Test on actual devices. If your app or site runs on different device types, then test on the most common devices found in your audience. Why?
People interact with things differently when the device is held in their hands rather than sitting on a desk.
Pixel sizes are different, not to mention bezels and the proximity of keyboards, so your visual design is perceived differently.
With an emulator, you run the risk of the software being not quite right (or too right) and your participants’ experience is changed.
Network connections differ between device and computer. Not only do connection speeds differ, but the latency of the connection differs. Faking a perfect network connection does your data no good at all.
Core UI elements don’t always work the same way. What if they need to install a component. Doing that on a handset is often totally unrelated to adding to the emulator.
Simulators are even worse. A lot of people don’t get the difference between simulators, emulators and remoting (like Device Anywhere). This just increases the chances of variations between actual devices and the test environment.
Unless you have participants shaking a computer, you lose any interactions that are not mimicable by a mouse, or even a touchscreen computer or tablet.
Don’t expect participants to learn a new device. This means either use the participant’s device (very risky given battery, connectivity, virus, and billing considerations) or use a device the participant is familiar with. You can add this to the screener, for certain classes of users, so you know in advance what they use.
Test in the field only when you have to. Studies suggest you won’t find any more usability issues, so do it only when you have something like in-person payments or location to perform. This is not saying you can’t discover information with ethnographic research, but we’re talking about usability testing now.
Consider paper prototyping, or digital variants. You can make the system behave in any way you want. And you can even let users pick up the paper. Or tape it to a phone or make a plywood “device” to make this easier and more realistic. Yes, you can even print paper at the right scale for your notional phone.
Capture video without limiting the user. There are ways to strap cameras down that let participants move normally while you capture on-screen interaction, their gestures and button presses, and their reactions.
Need eye tracking? You don’t always, but there are now very small, head-mounted solutions that can work with mobile device testing, and even cue to arbitrary environments.
For a little more about this, check out this very old presentation on Mobile Usability Testing or come to Design For Mobile 2010 next week in Chicago. I’ll be talking even more about this, so come join us!