Building an Accessibility Regression Testing Strategy with WCAG Reports

Wiki Article

Introduction


Automated accessibility testing can help website teams identify technical problems that may create barriers for users. It can examine selected pages, detect certain rule failures, and organise results around recognised accessibility criteria. Yet automated testing has important limitations. A page may pass the available automated checks and still provide a confusing experience for keyboard users or people relying on assistive technologies.


AegisRunner's WCAG Accessibility Reports are designed to document automated findings for WCAG 2.2 Level A and AA. The reporting workflow uses a browser-based page inventory, structured sampling, and axe-core checks in a fresh browser session. Its approach also makes clear that the resulting document is automated evidence, not a certification or legal determination.



Why Websites Need Accessibility Testing


Websites serve people with different abilities, devices, and interaction preferences. Some visitors use a mouse or touchscreen, while others rely on keyboard navigation, screen readers, or additional assistive technology. If controls are difficult to identify, text is hard to distinguish from its background, or relationships between content elements are unclear, users may experience unnecessary barriers.


Accessibility testing gives teams a way to identify and investigate these problems. It can form part of development quality assurance, release reviews, and ongoing website maintenance. Testing should not be restricted to a single launch milestone because pages, components, and interaction patterns can change over time.



What Automated Accessibility Testing Can Detect


Automated testing uses defined rules to examine page structures and identify certain patterns associated with accessibility failures. AegisRunner describes using axe-core during its evidence check on selected public pages. The resulting findings are mapped to relevant WCAG 2.2 A/AA criteria.


Its example report demonstrates how results can be grouped by criterion, level, and failure count. The displayed categories include Non-text Content, Info and Relationships, Contrast (Minimum), and Name, Role, Value. These categories help teams organise the technical findings for investigation.



Examples of Criteria in the AegisRunner Report



The example shows 16 automated failures across its examined sample, distributed across the four listed criteria. These values illustrate the report format and are not a prediction of what another website will produce.



Why Automated Results Are Not a Complete Accessibility Assessment


Automated tools can identify certain technical conditions, but they cannot reliably judge every aspect of an interactive user experience. An image may have alternative text, but a human reviewer may still need to decide whether that text communicates the correct meaning. A control may expose technical information but remain confusing when operated with a keyboard or screen reader.


Content quality, interaction behavior, and complete user processes require evaluation beyond automated rule execution. WCAG conformance applies to full pages and complete processes. Therefore, passing automated checks on a sample of pages does not establish that an entire website conforms to WCAG.



Combine Automated Checks with Manual Testing


A more complete accessibility testing approach uses automation and human review together. Automated checks can identify certain technical problems efficiently, while manual evaluation examines the experience of using the website. Both forms of testing contribute different evidence and should be treated as complementary.



Manual Review Areas



The exact manual testing approach should reflect the website's functionality and the user processes it supports. A simple information page and a complex account workflow may require different testing scenarios.



Coverage Is Essential to Interpreting Accessibility Results


A report should explain which pages were discovered, selected, and audited. AegisRunner places coverage information before its findings so readers can understand the scope before examining criterion counts. Its example shows 48 discovered pages, 12 selected pages, and 12 audited pages.


This distinction is important because a report may examine only a sample of the discovered website. A page that was not selected or could not be reached has not received the same automated examination as a page that was audited. Teams should review exclusions and unreachable pages before interpreting the result.



Reproducible Sampling and Evidence


AegisRunner describes a structured and reproducible random selection process from a completed scan inventory. The method and seed are recorded so that the same inventory can reproduce the sample. The evidence check then runs in a fresh browser session using axe-core on the selected public pages.


The resulting printable HTML report records coverage, detected failures, WCAG criteria, exclusions, and limitations. It separates automated failures, automated passes, and results that require human judgement. This structure helps prevent an automated pass from being mistaken for a complete manual evaluation.



What Happens When a Page Cannot Be Examined?


Incomplete coverage can weaken the meaning of an accessibility report. AegisRunner describes a fail-closed approach in which no report artifact is produced if important pages cannot be examined. The interface explains the missing roles, failed pages, or other reasons that prevented report generation.


accessibility testing automation

This approach encourages teams to address coverage problems before sharing a report. It also makes an important distinction: the absence of a finding is not evidence of accessibility when the relevant page was never successfully examined.



Using Accessibility Testing Throughout the Release Cycle


Accessibility testing should be included in ongoing quality work as websites evolve. A change to a shared component may affect several pages. A new form may introduce labeling or interaction issues. A redesigned section may alter contrast or content relationships. Reviewing accessibility after meaningful changes can help teams identify these issues before they become established parts of the website.


Teams can use automated reports to maintain a record of detected findings and manual review to assess broader usability. The evidence can help developers understand where problems were identified, while QA and accessibility specialists can determine what further checks are necessary.



Important Product Scope and Standards Details


AegisRunner states that new reports target WCAG 2.2 Level A and AA. The page also explains that EN 301 549 v3.2.1 currently maps WCAG 2.1, so clause 9 references appear only where applicable and newer WCAG 2.2 criteria are labelled not mapped.


For signed-in customer websites, the page states that testing behind a login is currently available only on controlled demos, with customer activation pending. Until then, the reports cover pages that do not require sign-in. Teams should confirm current availability before planning a project that depends on authenticated-page testing.



Conclusion


Automated accessibility testing is valuable for identifying certain technical failures and documenting how they relate to WCAG criteria. However, it cannot evaluate every content, keyboard, screen-reader, interaction, or complete-process requirement on its own.


AegisRunner combines automated WCAG 2.2 A/AA evidence with visible coverage information, exclusions, and manual-review limitations. Used alongside human evaluation and ongoing regression testing, this approach can help website teams maintain a more informed accessibility improvement process.

Report this wiki page