How Setsquare decides

What you get

Every settled page in your suite is scanned against WCAG 2.2 AA, and the run ends with a summary:

  Setsquare — accessibility evidence

  Scanned 4 surfaces in 2 themes, assessed against WCAG 2.2 AA.
  1 rule failing on 4 elements. 600 checks passed.

  serious  color-contrast — WCAG 1.4.3
           4 elements on 4 surfaces: privacy · light, terms · light, cookies ·
           light, accessibility · light
           https://dequeuniversity.com/rules/axe/4.13/color-contrast

  4 of 8 scanned contexts had no violations at all.

and an evidence file at .setsquare/runs/<timestamp>/run.json — the full findings, their WCAG criteria, measured contrast ratios, the exact ruleset used, and performance timings.

Checks that need the page driven, not read

Some of what runs is not axe. A DOM-snapshot analyser can only answer questions about the document as it stands; several WCAG criteria are about what the page does. These run because we are already inside your test, with your page:

If you use Tailwind and 1.4.8 fires across most of your prose, it is one line of theme configuration rather than anything you wrote. Tailwind's type scale ships line heights below the 1.5 the SC asks for — text-sm is 20px on a 14px font, which is 1.43 — and line height inherits, so one utility on one container hands it to every paragraph inside, whatever size those paragraphs are. One utility, dozens of findings. The fix is in @theme, once:

@theme inline {
  --text-xs--line-height: 1.55;
  --text-sm--line-height: 1.55;
  --text-base--line-height: 1.55;
  --text-lg--line-height: 1.55;
  --text-xl--line-height: 1.55;
}

That raises the default and leaves leading-tight and friends working, so headings and single-line labels keep the leading they are meant to have — which matters, because 1.4.8 is about blocks of text and WCAG defines a block as more than one sentence. 1.55 rather than 1.5 so a clamped font size cannot land a block a hair under the bar. Worth checking before editing anything, because a run of these usually has one cause rather than one per block.

The three walks that drive the page — focus, keyboard and reflow — run at teardown only, and that is a real limitation rather than an implementation detail. Focusing an element fires your page's own focus, focusin and blur handlers; the keyboard walk presses Tab and Escape, which fires your key handlers too; the reflow check resizes the viewport, which re-runs your media queries. A handler that opened a menu or started a request has already run by the time anything is restored.

Scroll position, the previously focused element and the viewport are all put back; side effects cannot be. So a scan of a named state ({ state: 'menu-open' }) carries no evidence from these checks, while whatever your test left on screen does.

And twenty criteria are reported as no automated rule can decide this — separately from the criteria nobody has built a rule for yet, because those are different facts and an auditor's first question about an untested criterion is which of the two it is. Each one carries the reason and what would have to change for it to move, so the list is a position rather than a wall. Where presence is checkable and adequacy is not — a caption track exists, but whether it matches the audio is a judgement — the entry says so at that granularity rather than writing the criterion off whole.

That list is the concrete floor under "automated evidence is not conformance". A run that tells you twenty criteria no tool will ever check for you has told you something more useful than a green tick.

Alongside those, nine heuristics read the page for criteria no rule can decide: text painted as a heading (1.3.1), meaningful sequence (1.3.2), sensory instructions (1.3.3), weak headings and labels (2.4.6), and prerecorded media without captions or audio description (1.2.1/1.2.2, 1.2.3, 1.2.5); and with AAA, unheaded runs of prose (2.4.10), abbreviations (3.1.4) and location cues (2.4.8). 1.2.4 and, with AAA, 1.2.6–1.2.9 are answered only on a page with no media at all.

They follow one discipline: fire only above a margin, say what would resolve it, and decide only what the page itself can show. Where the question is about meaning — whether a reordered layout changes it, whether a heading describes its section — a heuristic asks a person, because someone who disproves one guess stops trusting all of them. Where the criterion names a mechanism or states a test the page either has or fails — none of 2.4.8's techniques anywhere on the page, an instruction that identifies its target by shape, colour or position alone — it reports a failure, with the caveat in the message.

2.4.5 Multiple Ways is not one of them: it is computed across the run from what the scans carried, and it asks rather than decides, because some of the routes the criterion accepts are ones it cannot recognise.

The heuristics are on by default, being read-only and unable to fail a build, and each is switchable on its own — heuristics: { sensoryCharacteristics: false } — so disagreeing with one costs you one rather than all nine. The three that read English words stay silent on a page in another language rather than guessing.

The last three below are the ones no per-page tool can have. They are statements about a set of pages, computed after the whole run from what each scan already carried — so they cost no extra page work, and they exist because the run holds every surface at once.

They are also the ones with the most dangerous failure mode, so: a run with fewer than two surfaces reports them as untested, never as passing. A one-page run cannot evidence consistency, and a clean pass on a run that had nothing to compare would be the most flattering possible lie. The summary says "these criteria compare pages against each other — this run had one surface" rather than folding them in with the criteria nothing can check.

Only links in navigation, header and footer landmarks are compared, because 3.2.4 is about repeated components: naming an inline prose link differently in two sentences is ordinary writing, not an inconsistency.

Five criteria are off unless you turn them on (listeners: true), and for a different reason again — 2.1.4 Character Key Shortcuts, 2.5.1 Pointer Gestures, 2.5.2 Pointer Cancellation, 2.5.4 Motion Actuation and 2.5.7 Dragging Movements. Every one is a question about a HANDLER rather than about your markup, and there is no way to enumerate a page's event listeners after the fact, so the only route is to wrap addEventListener before your own scripts run, for the whole life of the page. One wrapper answers all five, so turning it on costs the same whether you care about one of them or all of them. (keyShortcuts: true still works — it is the name this option had when 2.1.4 was the only criterion behind it.) Everything else here happens after a page has loaded and can be undone. Even proven inert — the wrapper records and delegates with your original arguments, so removeEventListener still matches, and the fixture suite asserts exactly that — it is a larger presence in your application than a scanner has any business having by default.

They only ever report needs a person, or a pass carrying a count: each of the five is satisfied by an affordance somewhere else — a way to turn a shortcut off, a button that does what the gesture does, an undo — and none of those is visible from a listener. 2.1.4 is satisfied if the shortcut can be turned off, remapped, or is only active when a component has focus, and all three are affordances no check can see.

3.2.2 is off unless you turn it on (onInput: true), and the default is the decision rather than caution: a change event is where an application saves, and writing to your data is a worse failure than any finding the rule could produce — worse than being a flake, because a flake is visible and a silent write is not. Turn it on where the suite runs against disposable data, which is most suites. Even then, only <select>, checkboxes and radios are touched and their values are put back; nothing is ever typed into a text field, no form is ever submitted, and no navigation is ever allowed to commit.

2.5.5's click probe is off unless you turn it on (clickProbe: true), for the same reason and with the same shape. Geometry gets 2.5.5 wrong on a control every design system ships — a 40px <input> in a 44px wrapper whose handler focuses it — and nothing in the DOM can tell that wrapper from one that is decoration, so the only way to answer is to click. It clicks only controls this run has already failed, once each, at teardown, with the default action of every click suppressed so nothing navigates, submits or toggles; focus is put back. It clears a control only when that control was activated, so a toolbar whose handler focuses something else does not excuse the icon button beside it.

Where geometry or stacking makes the answer a judgement — a translucent overlay over the focused element, an indicator drawn with a box-shadow — these report needs a judgement, never a pass and never a fail. A focus ring against a gradient reports could not measure, which is a different statement: that one is our debt rather than your work, and the two are separate sections for exactly that reason.

Indicators styled for :focus-visible only are judged. :focus-visible is a statement about how the user got here, so a suite that clicked anything leaves the browser in pointer modality where it matches nothing — the walk presses Tab first, so an indicator that a keyboard user would see is one we can see too.

14. Criteria checked beyond axe

Nothing to configure — these run already. They are listed because people reasonably assume a scanner is axe and nothing else, and then wonder where a finding came from.

Two things worth knowing about the focus walk.

They run at teardown only. Focusing an element fires your page's own focus, focusin and blur handlers; the keyboard walk presses Tab and Escape, which fires your key handlers too. A handler that opened a menu or started a request has already run by the time anything is restored. Scroll position and the previously focused element are put back; side effects cannot be. So the walk happens where your test is over and the page is about to be discarded — which means a scan of a named state carries no focus evidence, and whatever your test left on screen does.

They are bounded, and they say so. The focus walk focuses at most 120 elements. The keyboard walk presses Tab enough to go round your page's tab order twice in each direction, up to 1,000 presses, which is two laps of a page with up to 499 tab stops. Both give themselves a wall-clock budget, and both say how many of how many they covered, in the finding itself — a bounded walk reported as a complete one is a number that looks like a statement about your page and is a statement about part of it.

For 2.1.2 the bound is the feature rather than a defence: a trap is precisely the condition under which "tab until you reach the end" never terminates.

The last row needs two surfaces, and says so when it does not have them. A run with fewer than two surfaces reports 3.2.3, 3.2.4 and 3.2.6 as untested — "these criteria compare pages against each other; this run had one surface" — rather than as passing. A one-page run cannot evidence consistency, and a clean pass on a run that had nothing to compare is the easiest lie in this whole tool to ship by accident.

Only links in navigation, header and footer landmarks are compared: 3.2.4 is about repeated components, and naming an inline prose link differently in two sentences is ordinary writing.

Twenty criteria are reported as "no automated rule can decide this", kept apart from the ones nobody has built a rule for yet — an auditor's first question about an untested criterion is which of the two it is. Each entry in packages/schema/src/automation-limits.ts carries its reason and what would have to change, so it is reviewable rather than a wall, and entries where presence is checkable but adequacy is not say so at that granularity.

The heuristics are on and individually switchable. Each fires only above a margin argued in the source and says what would resolve it. Four of them can report a failure, where the criterion names a mechanism or states a test the page either has or fails; the rest ask a person. If you disagree with one:

export default withSetsquare(defineConfig({ … }), {
  heuristics: { sensoryCharacteristics: false },
});

That silences one and leaves the other eight, which is deliberate: a customer who disagrees with one heuristic must not have to turn off the lot. The three that read English words are silent on a page in another language rather than guessing at it.

Five criteria need turning on too, with listeners: true, and for a different reason: it is the only thing this SDK installs that runs before your own code. One wrapper serves 2.1.4 Character Key Shortcuts, 2.5.1 Pointer Gestures, 2.5.2 Pointer Cancellation, 2.5.4 Motion Actuation and 2.5.7 Dragging Movements — all five are questions about a handler rather than about markup, so none of them is reachable any other way, and turning it on costs the same for one as for five. (keyShortcuts: true is the old name and still works.) The wrapper records and then delegates with your original arguments, so removeEventListener still matches — a wrapped listener would be a different function object and your page would silently keep handlers it asked to be rid of. The fixture suite proves it by adding and removing a handler and checking the key does nothing.

They only ever report needs-a-person, or a pass carrying a count, because each is satisfied by an affordance no check can see — turn the shortcut off, remap it, offer a button that does what the gesture does, provide an undo. Not one of them can be a failure from a listener alone.

3.2.2 needs turning on:

export default withSetsquare(defineConfig({ … }), { onInput: true });

Off by default because changing a control's value is where an application saves, and writing to your data is a worse failure than any finding the rule could produce. Turn it on where the suite runs against disposable data. Only <select>, checkboxes and radios are touched and their values are put back; nothing is ever typed into a text field, no form is ever submitted (submit events are cancelled and form.submit() is patched, which is not the same thing), and no navigation is ever allowed to commit.

While it is off, 3.2.2 reports as a check that exists and did not run here — which is what it is, and is a different statement from "no automated check exists".

2.5.5's click probe needs turning on:

export default withSetsquare(defineConfig({ … }), { clickProbe: true });

Off by default for the same reason. It exists because geometry gets 2.5.5 wrong on a shape every design system ships: a 40px <input> inside a 44px wrapper whose click handler focuses it. The effective target is 44px, the DOM says 40, and there is nothing in the DOM to say otherwise — the handler is attached at the framework's root and getEventListeners is a DevTools API. The only thing that can see a delegated handler is a pointer action, so this performs one.

What it touches: only controls this run has already failed on a short pointer region, one click each, on one ancestor, at teardown, after the scan. The default action of every click is suppressed for the probe's duration, so nothing navigates, submits or toggles — while your own handlers still run, which is the point. Focus is put back. What it cannot stop is your own JavaScript: a handler that routes or POSTs will.

It only ever moves a row from failing to passing, and only when that control was activated — a toolbar whose handler fires and focuses something else does not clear the icon button beside it. Turning it on also makes the teardown sweep re-scan pages it would otherwise skip, so a suite pays one extra scan per unchanged page for the correction.

While it is off, those controls stay violations and the rule's own text says what it could not see.

Two things 2.1.2 will not claim. A cycle is only a failure if it cannot be left, so a modal dialog that releases on Escape passes — cycling focus is what a correct modal does. And a page containing an iframe reports needs a person, because focus moving into a frame is handled by that frame's own document and the parent gets no event for it: a trap inside the frame would look exactly like a clean walk from outside.

Where the answer is a judgement it reports needs a person, never a pass and never a fail: a translucent overlay over the focused element, an indicator drawn with a box-shadow (whose area computed style does not pin down), a ring against a gradient, or the browser's own outline-style: auto ring, whose painted thickness is not what outline-width reports.

One more, worth knowing if you use the modern pseudo-class: where :focus-visible did not match, "nothing changed on focus" is reported as needs-review rather than as a 2.4.7 failure. A programmatic focus does not always match :focus-visible, and an indicator styled for it only would not have applied — so that is a weaker claim than "a keyboard user sees nothing".

What it will not do

• Fail your build. A scan error warns; your tests keep their own verdicts.
• Phone home. No telemetry, no error reporting, no version checks — with or without a token. There is no network client in this package at all, and its test suite asserts that there never will be.
• Catalogue your media or images. Not in a local run, ever. A connected mode that records references to a page's media and images — URLs and metadata, never bytes — so the hosted platform can judge criteria no page can answer is decided and not yet built (D21). When it lands it records only when a token is present, and this section will list every field it carries before any run does.
• Write images. The contact sheet above indexes and links to the screenshots Playwright wrote; it copies none of them and this package writes none of its own. One capture is taken in memory to measure contrast against a background CSS cannot resolve, and discarded where it is read — nothing reaches your disk or your run file through us. paintedContrast: false turns even that off.
• Compare screenshots. Pixel comparison against a stored baseline is a platform capability. Locally you get Playwright's own comparisons, presented.
• Compare runs. No history, no baselines, ever, locally. The moment the question is "better or worse than last week", that is the hosted platform.