Followup on usability testing... and a bit of a break

I'll be taking a bit of a break for a week as I take some time to relax, but in the meantime you might want to take a gander over at Lightspeed Venture Partner's blog. They have a great followup to their recent usability testing article which I commented on here.

In this post they go into depth on what sorts of testing are appropriate at various levels of development. Most of our work for entrepreneurs has focused on the Ideation phase of development, where LSVP recommends that you explore new ideas and opportunities, take the founders vision and add in the background developed during initial testing. Specifically Knowledge Athletes (aka KA one of our most successful clients - our first) has performed the following types of testing in their product roll out from LSVP's list:

• Ethnographic field studies (KA's "in school trials")
• Focus groups (KA has focus groups consisting of high school students at all levels, college students and corporate clients)
• Diary studies (the unique nature of KA's product is that it IS a diary so users can comment and track their usage of the product IN the product itself)
• Surveys (So far online user surveys have created a number of interesting new ideas for the product)
• Data mining (As KA has created user experiences TONS of data is now available on how the product is used and when it has been successful. This data is leading not only to new development ideas, but also paradigm shifts in teaching methods)

Th original paper on this topic by Christian Rohrer of xdstrategy.com has a great chart that shows where studies work best and compares data source vs approach vs context of use. You can learn more about usability testing at this University of Texas site.

Alternatively Jakob Nielsen's book on usability testing (linked below) is widely regarded as being one of the best in the industry.

Usability testing in entrepreneurial software

Jeremy Liew of Lightspeed Venture Partners has an excellent blog post on Usability Testing and how it applies to building entrepreneurial software. In his post he mentions that it's not just about rapid prototyping and turnaround (which he approves of) but also says that at some point you have to step back and do surveys, and psychological profiles of users and see exactly HOW they used the site. This is a critical step in building software, and obviously (since Jeremy commented on it) important to getting venture funding as well.

At OS-Cubed, we believe that usability testing should be scoped into the project itself, and the amount invested in it should be scaled according to the entrepreneur's budget, but it should be conducted throughout the development process as you're building your demo or proof of concept and rolling out your RAD prototype. In the early stages usability testing does not need to be either expensive or costly. You can do much of the testing yourself with the demo as it's rolled out. Noah Kagan of Mint lays out the steps for you in his excellent post. They involve basically: determining what to test, finding a representative user base, and then testing by observing. Noah recommends using Webex products, so far we've been lucky enough to have user bases that are representative right here in Rochester.

As Noah points out - doing this can be frustrating for the observer. The observer needs to sit back and let the user do it - without intervening or showing them how.

In our experience most of the user testing we've done with KaJour our product for Knowledge Athletes has shown up issues in the scalability section - which you'd expect in demo software. We'd actually anticipated these issues but didn't have the budget to build production level. We consciously chose with the entrepreneur to build more features, rather than make the features we had more robust. The users have also come up with unique and interesting ways of using the product that we didn't anticipate and neither did the Knowledge Athletes staff. Through our user testing we've found new markets, new ways of thinking about what we're doing and developed new teaching paradigms for the community we're serving (at the moment mostly high school and college students with this product).

Although Jeremy points out the obvious benefits of using usability testing to be sure your software is stable and simple to use, he doesn't mention that really listening to your users can allow you to create new and better uses for your software, and take your product in an entirely new direction. To do that you need to truly listen to your users and find out what it is they need - what pain you are solving - and how you could do it better.