Why an automated AAA score isn't an accessibility audit
Thirty years in kitchens taught me one thing about signs in windows: anyone can stick up “five stars”. The rating that counts is the one the inspector gives after they've had a look in your fridge. Websites are no different.
The short answer
- A scanner score tells you the scanner found nothing it knows how to find. It doesn't tell you disabled people can use your site.
- A site claiming AAA from an automated checker failed Level A and AA checks within minutes of manual testing.
- Keep using scanners, but treat a clean result as the start of testing, not the end.
The site that scored AAA
I recently came across a public-sector venue website whose accessibility statement says it has “an AAA level rating” from an automated checker. AAA is the top level of the Web Content Accessibility Guidelines (WCAG), and very few sites even attempt it. I ran axe, another free automated tool, over the same pages and it found nothing either. On paper, a clean bill of health.
Within ten minutes of looking at it as an auditor, a few things caught my eye. Digging further with a colour contrast analyser, a keyboard and a screen reader confirmed my suspicions: failures at Level A and AA, the levels the site actually needs to meet. I'm not naming the site. The point isn't that one team got it wrong. It's that the tools let them believe they'd got it right.
What automated tools can and can't see
Automated tools check code against rules. They're good at things a
computer can measure: a missing alt attribute, a form
field with no label, text colour against a flat background colour.
They can't judge meaning or experience. A tool can see that an image has alt text, but not whether that text describes the picture. It can't see text sitting on a photo, tab through a page, or listen to what a screen reader says.
The UK Government Digital Service tested this. They built a page with 142 deliberate accessibility barriers and ran it through 13 automated tools. The best tool found 40% of the barriers and the worst found 13%. In an earlier round with 10 tools, 29% of the barriers weren't found by any tool at all.
What a manual check found
Every issue below passed the automated scan. Each one affects real people.
| What I found | Who it affects | WCAG criterion | Why the scanner missed it |
|---|---|---|---|
| White headline text on a light photo, measuring about 2:1 at its weakest point (large text needs 3:1) | People with low vision, or anyone on a phone in sunlight | 1.4.3 Contrast (Minimum), AA | The background is an image, so there's no colour value in the code to check |
| An orange focus outline on a yellow header, measuring about 1.7:1 (needs 3:1) | Keyboard users, who can't easily see where they are on the page | 1.4.11 Non-text Contrast, AA | Focus styles only appear when someone tabs to an element; a scan looks at the page at rest |
| An image carousel that stays silent when you press Next with a screen reader. You hear the click sound, then nothing: no slide number, no description, no sign anything changed | Blind and low-vision screen reader users | Closest match is 4.1.3 Status Messages, AA | The announcement area exists in the code, so the scanner sees nothing wrong. It can't press the button and listen |
| Alt text that repeats the image file name, such as “First Floor Library CGI Visual” | Screen reader users, who get no idea what each picture shows | 1.1.1 Non-text Content, A | A tool can check that alt text exists, not whether it describes the image |
The carousel is the clearest example. Its code was nearly right: labelled buttons, a hidden heading, and a live region ready to announce each slide. The live region was simply never filled in. To a scanner that's a pass. To a blind visitor it's a button that appears to do nothing.
Tools also flag things that aren't problems
Scanners don't only miss issues. They raise false alarms too, and tools disagree with each other.
On the same site, WAVE flagged 12 “possible heading” alerts in the footer, one for each line of the opening hours. Each line, such as “Monday: 10am – 5pm”, is a short paragraph, and WAVE guessed it might be a heading. It isn't. The block already sits under a proper “Opening hours” heading, and a screen reader reads each line clearly. Axe, run on the same page, didn't raise them at all.
So one tool missed real barriers, and another flagged a non-issue a dozen times. A team working only from tool output could spend hours “fixing” the opening hours while blind visitors still couldn't use the carousel. Knowing which results matter takes a person who understands both WCAG and how assistive technology behaves.
A tool score is not a WCAG rating
WCAG conformance is all or nothing at each level. A site meets AA only if every page passes every A and AA criterion; one failure means it doesn't. A scanner score measures how many of its own rules passed, which is a different thing.
Putting a tool score on your accessibility statement as if it were a WCAG level causes two problems:
- It misleads disabled visitors. Someone reading “AAA” will reasonably expect the site to work for them, then hit barriers the statement said weren't there.
- It can put you on the wrong side of the rules. UK public-sector sites must publish a statement saying whether they are fully, partially or not compliant, and list known issues. A statement that says “compliant” while listing failures, or rests on a tool score, doesn't meet that standard. Public-sector sites are now monitored against WCAG 2.2 AA, so statements still quoting 2.1 are out of date too.
Private businesses aren't covered by those regulations, but the Equality Act 2010 still applies. A confident claim you can't back up is worse than an honest “we're working on it”.
What to do instead
Keep using automated tools. They're quick, free and catch real problems. Just treat a clean result as the start of testing, not the end.
- Run an automated scan to clear the obvious, mechanical issues.
- Unplug your mouse and press Tab. Can you see where you are at every step, and reach and use everything? Try the Tab Test.
- Check text on images with a contrast analyser, measuring the lightest part behind the text.
- Listen with a screen reader. Turn on VoiceOver or NVDA, use your main features, and ask whether each step makes sense with your eyes closed.
- Get an expert manual audit before you publish any conformance claim, and date your statement so readers know when it was last checked.
For a fuller comparison, read manual audit vs automated accessibility scanners, or see how much a website accessibility audit costs.
Sources
- Government Digital Service: Accessibility tool audit
- GDS Accessibility blog: What we found when we tested tools on the world's least-accessible webpage
Want to know what your scanner isn't telling you?
I test by hand with a keyboard, VoiceOver, NVDA and JAWS, and explain every issue in plain English for your developers.
Email Clearsight AccessThis guide is general information, not legal advice. Back to all Web Accessibility Guides.