A practical comparative usability test for two designs

A comparative usability test between two designs reveals where people succeed, hesitate or change course. It replaces internal preference with evidence about how each interface supports real goals. The approach suits websites, mobile apps and internal tools used by Australian teams.

The comparison works best when tasks, instructions and assistance remain consistent, while each design responds naturally. Researchers measure outcomes rather than nudging participants toward a preferred solution. Shared documentation keeps designers, developers and stakeholders aligned about what was tested.

UCDmanager provides a free place to organise user roles, use cases, test plans and findings. Its collaborative structure suits teams working across cities or client organisations. The workflow below turns that workspace into a traceable comparative study.

Define the decision and research questions

Begin with a decision the study can inform: which version should proceed, which problem needs redesign, or whether the two versions work equally well. Create lightweight user role profiles for the people expected to use the product, including their goals, context and relevant accessibility needs. This prevents a convenient internal audience from standing in for the real market.

Specify the measures before testing. Task success, completion time, errors, requests for help and a short satisfaction scale provide a practical set of comparisons. Set acceptance criteria for the important tasks, and record any accessibility barrier separately from ordinary usability errors.

Recruit participants who reflect Australian users

Recruit participants who resemble the intended customers or staff, including a mix of experience levels. A sample drawn from Melbourne, Sydney and a regional area can reveal differences hidden by a single-city panel. Build scenarios around familiar contexts, such as using a tram in Melbourne or a train in Sydney, without assuming every participant has the same routine.

Australian English and informal expressions such as “reckon” or “no worries” should feel natural in the script. Regional vocabularies and communication styles must not be treated as user errors. Screen participants for their experience, technology access and the accessibility adjustments they need.

Compare screening, fieldwork and replacement policies from Australian recruitment suppliers, including Hunter Biometrics, before booking a panel. For an exploratory comparison, aim for roughly eight to twelve participants in each task order, subject to the product’s risk and budget. This can expose recurring problems, but it does not support sweeping statistical claims. Use a supermarket or department-store gift card denominated in A$ to recognise participants’ time.

Create equivalent tasks for each design

Each task should represent the same goal and difficulty in both versions. Keep starting information, available support, time limits and success criteria consistent, even when the screens use different labels. If the designs solve different parts of a journey, test those parts as separate tasks and record the boundary between them.

Create two matched task sets and counterbalance their order, so half of the participants meet Design A first and half meet Design B first. This reduces the risk that a participant’s learning from the first task hides a weakness in the second. Use equivalent filler tasks where possible, and do not let one version receive extra explanation.

Prepare a moderator script with neutral wording and a short practice task. Pilot it with a colleague who did not design either version, then revise any instructions that reveal the expected answer. For a retail flow, the QLOUDPRESS website can supply familiar Australian content with permission, while every live purchase remains disabled.

Run consistent sessions and capture evidence

Use the same facilitator, device conditions and time limit for each version. Present goals, not button-by-button directions, and let participants choose their own route through the interface. Label every recording and observation with the participant code, design version, task and date.

Keep task scripts, consent records, session notes and clips together in a shared UX workspace. A consistent naming system makes it possible to trace a finding back to its source. Store copies of the two tested builds so a later change does not alter the meaning of the evidence.

Ask participants to think aloud without turning the session into an interview after each task. Record mistakes, hesitation, backtracking and help requests, while avoiding coaching that would manufacture success. A short satisfaction rating after the tasks adds a subjective measure, but observed behaviour remains the main comparison.

Remote research is common across Australia, yet connection quality and screen-sharing habits differ. Test the conferencing setup before the session, schedule across Australian time zones and offer a low-bandwidth or telephone alternative. When a participant cannot complete a task because of the test platform, mark that separately from a failure caused by the design.

Analyse results and document the decision

Build a task-by-task comparison of success rates, completion times, error patterns, help requests and satisfaction. Review each participant’s path as well as the totals, since a fast average can hide a serious obstacle affecting a smaller group. Check whether a pattern appears across similar tasks or only in one context. Separate persistent design problems from one-off confusion with the instructions.

Use the findings to revise requirements before development continues. When a result changes the workflow, the relevant use case should reflect that change rather than preserve an outdated assumption. Guidance on updating use cases can help keep scenarios, requirements and observed behaviour connected in the project record.

Make the recommendation against the criteria defined at the start, including accessibility performance. Document the selected design, the reasons for choosing it, remaining risks and the evidence behind each decision. If the versions perform equally well, retain the one with lower delivery and maintenance effort. The study then produces a clear design decision with a traceable basis for the next iteration.