When purchasing assistive-technology testing, ask for reproducible tasks in named environments rather than a statement that the product “supports screen readers”. Useful evidence shows what the tester attempted, on which device, operating system, browser, technology and settings, and where a barrier occurred. It helps engineering repair the issue without turning limited success into a universal compatibility claim.
Expert testing differs from research involving actual users. The former examines specified behavior through relevant skills and tools. The latter adds people’s lived experience and usage strategies and requires recruitment, consent and suitable arrangements. A team member switching on a screen reader is not participant research.
Start with the task that could fail
Describe an outcome: find a suitable appointment, choose a time, correct contact details and understand whether the booking is confirmed. Then identify observations such as reaching a control, understanding its name and state, navigating away and back, and perceiving the result. Reading the page title does not establish that booking is possible.
Specify starting data, account, role and language, the success condition and what counts as tester assistance. If the journey includes an external step, state whether it is covered and how it will be reached safely. Do not use customer accounts or create real bookings simply because the live system is convenient.
Assistive Technology Testing covers agreed journeys and combinations. Before committing, confirm access to the required technologies and testers who can use them competently, or identify preparation still needed.
What belongs in the environment matrix?
Record the device, operating-system version, browser version, assistive-technology version, language, consequential settings, execution date and product build. “Latest” is not enough to reproduce a finding after an update. A result in one browser must not automatically be extended to another.
Select combinations using the audience, supported devices, risks and available evidence, not merely the supplier’s equipment. If usage evidence is unavailable, explain the selection rationale and what needs later validation. Adding a combination expands coverage; it does not make a limited set representative of every circumstance.
The matrix should also show unavailable environments before execution begins. That allows the buyer to approve a narrower scope or arrange access rather than discovering the gap after receiving a reassuring summary.
Worked hypothetical: a booking test matrix
Suppose three environments are agreed: desktop browser with keyboard only, desktop with a specified screen reader and mobile with a specified screen reader. Actual software names, versions and devices must be entered before testing. This is a proposed scope, not a claim that tests have been performed.
The four tasks are finding the service, choosing an appointment, correcting a contact number and checking confirmation. Three environments multiplied by four tasks gives twelve task executions per language. If both Arabic and English are included, the planned total is twenty-four. This arithmetic describes coverage, not duration or assurance that every exceptional state is included.
In one illustrative execution, the tester can open the appointment list but cannot determine the selected time after returning from an error message. The report distinguishes reaching the element from understanding its state. In another, the message can be read but returning to the form places the person far from the field requiring correction. These should not collapse into a generic “screen reader broken” finding.
After repair, the same steps are repeated on the new build with known test data. If engineering changes a shared component, the team identifies other potentially affected tasks and agrees their inclusion in retesting. One successful rerun does not automatically close every related row.
The test lead also records assistance. If a facilitator points out a hidden action or changes the environment to get past an obstacle, the original task did not simply pass unaided. That intervention can be useful diagnostically while remaining visible in the outcome.
Produce defect records engineering can use
Each record should contain the environment, build, starting condition, commands or gestures, point of failure, actual result, required result and effect on the task. An authorised recording may help demonstrate behavior, with unnecessary information removed.
Speech output need not be word-for-word identical across technologies. Document whether the required name, state and meaning were available and understandable. A qualified specialist can identify the relevant technical reference when making a standards-related finding. Do not assign the cause to the product, browser or user’s proficiency without investigating.
WAI notes that interaction problems can involve content, browsers, assistive technologies or usage. Involving users in accessibility evaluation. That supports documenting the environment and investigating causes; it is not a reason to dismiss an issue without evidence.
Where a finding cannot yet be reproduced, preserve the original conditions and uncertainty. “Not reproduced in this attempt” is more accurate than declaring the reported barrier nonexistent.
Keep adjacent services distinct
A Digital Accessibility Audit can establish a broader standards-evaluation scope. This task matrix focuses on execution with declared combinations; it does not replace complete standards coverage, confer certification or provide a legal guarantee.
User Experience Design can improve a journey that remains confusing after technical repair. Software Quality Assurance can maintain traceable retest cases across agreed releases. Participant research, ongoing maintenance and every device should not be assumed unless explicitly included.
Procurement should distinguish those outputs. A task-level report can be valuable without pretending to be an organisation-wide accessibility assessment. Conversely, a broad audit label is not enough if the buyer specifically needs reproducible screen-reader execution on a defined customer journey.
Acceptance criteria for the result
Every planned execution should have a clear status: completed, blocked or untested, with the reason for the latter. Defects should link to tasks and environments, reruns should be comparable and remaining gaps should be visible. Inability to open a test environment must not become a pass. Assistance that enabled completion must not disappear from the record.
Agree how repairs are submitted and how changes to the matrix are approved. This prevents a retest from becoming an undocumented new evaluation or an old environment from being silently substituted.
Frequently asked questions
Is keyboard testing the same as screen-reader testing?
No. Some steps overlap, but the interaction modes differ. Record both explicitly rather than substituting one for the other.
Is one screen reader enough?
It can be part of a justified initial scope, but it does not establish compatibility with others. Coverage depends on the audience, product, risks and agreed resources.
Must disabled participants be recruited for every check?
Not for every expert check. Their participation supplies different evidence. If participant research is requested, name and plan it separately, including consent and accommodations.
What happens after a browser update?
The previous result remains tied to its environment and date. Agree what needs rechecking, especially critical tasks or issues affected by the update, rather than assuming permanent validity.
Source
- W3C WAI: Involving Users in Evaluating Web Accessibility, reviewed 26 October 2026.
