Failing early. Lessons learnt from usability testing

• Problem Discovery - when expert knowledge hides customer problems
• Testing - comparing staff and community user groups
• Insights & Outcomes - what the testing revealed

The project aimed to replace a paper-based fundraiser registration process with an online service. Requirements and design decisions had been heavily influenced by subject matter experts and internal stakeholders. Community fundraisers, the primary audience, had not meaningfully evaluated the experience before late-stage testing.

The team wanted confidence that users could complete registrations without assistance. My task was to identify risks before launch and determine whether the service matched real-world user needs.

Community NGO operating models are distinct from big industry leaders. They rely on older volunteers who are familiar with previous models, and organisations need to proactively support adoption of new models. Government needs to support compliance via organisations and volunteers.

My role

I planned and facilitated usability testing, analysed behavioural and attitudinal findings, compared expert and non-expert user groups, prioritised issues and delivered recommendations to stakeholders.

I observed two user groups with a think-aloud usability test protocol. Group one was staff administrators; group two were community organisation fundraisers.

Staff user group

Staff participants said the design was clean and simple. They were observed completing tasks with a reasonably high success rate and rated the system 65 on the System Usability Scale:

• “It’s simple and basic, nothing that I think needs to be changed.”
• “Easy to use overall.”

Community fundraisers group

Community users became disoriented by a process flow which didn’t match the way they needed to collect information. Specialist jargon caused confusion and incorrect information. Community users became increasingly frustrated with poorly sequenced navigation and rated the system 28.3 on the System Usability Scale, described as unacceptable.

I observed them failing key tasks, losing work done without a save function, or giving up. None of the external fundraiser participants completed registration successfully without assistance:

• “I’m lost — haven’t I been here already?”
• “Where am I now? Are these traps that you set for us?”
• “I’m really angry now. This is the worst thing I’ve seen in years.”
• “I find this impossible… I’d rather print the form off and spend two hours completing it on paper.”

What the testing revealed

• Process flows needed to match real-world information gathering
• Users needed to know what information they needed before starting
• Inaccessible overlays needed to be replaced
• One point of contact needed despite multiple business touch points
• The form should not ask for same information twice
• A save function was needed to prevent data loss
• Internal jargon needed simpler plain language
• A clear progress bar was needed

What changed

• 36 usability issues were documented and prioritised
• Critical usability risks were identified before launch
• Testing created evidence for substantial redesign and refactoring work
• The findings demonstrated the danger of relying on expert stakeholders as user proxies

Reflection

The most important lesson was that expertise can become a design risk when it replaces user research. Internal stakeholders understood the policy environment so well that they unintentionally designed for themselves rather than for community users.

This project reinforced a principle that has shaped my work ever since: if the target audience has not been observed attempting real tasks, then many of the most important assumptions remain untested.