Why automated testing does not replace a manual accessibility audit

Automated accessibility tools are useful. They are fast, consistent and can run during every release. The problem begins when their output is presented as a complete accessibility audit.

A scanner evaluates code against predefined rules. A person evaluates whether the interface is understandable, operable and suitable for completing a real task.

What do automated tools find well?

• Some colour contrast failures.
• Form controls without programmatically associated names.
• Images without an alternative-text attribute.
• Empty links and buttons.
• Duplicate identifiers.
• Some invalid ARIA values and role combinations.
• A missing page language or document title.

These findings matter, particularly when the same defect repeats in a shared template. Automation is also useful for detecting regressions before release.

What can a scanner not assess reliably?

Is focus order logical?

A tool can identify interactive elements without understanding whether their sequence supports the user's task.

Does a dialog behave like a dialog?

The role may be present while focus remains behind the modal or fails to return to the trigger after closing.

Is a form error understandable?

A scanner may confirm that a label exists without knowing whether the error explains the expected format and how to continue.

Is alternative text meaningful?

An alt attribute can exist while containing useless text such as “image123”.

Can the critical task be completed?

Individual pages may score highly while a product option, date picker or payment dialog still blocks checkout.

Why can a high score be misleading?

A numeric score is useful for internal monitoring but is not an accessibility certificate. One serious focus trap can block a keyboard user completely despite hundreds of passing checks.

How should automated tests be used?

1. Run them during development. Test components, errors and authenticated states as well as public pages.
2. Include them in CI. Catch obvious regressions before release.
3. Group repeated failures. One shared-component defect should not become hundreds of separate tasks.
4. Validate findings manually. Particularly when context affects the recommendation.
5. Test complete journeys. Add keyboard, responsive and selected screen reader scenarios.

What should manual testing cover?

• Keyboard navigation and visible focus.
• Menus, dialogs, disclosures and custom controls.
• Form instructions, validation and post-error focus.
• Text resizing, responsive layout and reflow.
• Meaningful headings, landmarks, links and content.
• Selected screen reader scenarios.

Conclusion

Automated testing and manual auditing are complementary methods. Automation provides speed and repeatability; human assessment provides context and verifies real interaction. A usable website needs both.