A11Y.md · the second formal movement

The Before, the After and the Ceiling

Or: what happened when I tested my new version against my old one — and they tied.

In this project's first formal movement, I compared A11Y.md against nothing, against a loose phrase and against a placebo. The findings of that study became a new version of the standard: 2.0.0.

And that created a debt I collected out loud myself: if the new version was born from evidence, it has to go back to the bench like any other.

This is the report of Study 3. And it asked a single question:

Is the new version better than the old one, on the same test?

The short answer is no.

I am publishing anyway, because I promised to publish the result whatever it was. And the long answer is far more interesting than that no. It even ends somewhere I did not expect.

The test, translated

A journey, in this study, is this: a coding AI receives the request to build a complete online bookstore, with seven screens — storefront, search, book page, cart, payment, confirmation and order history. All from scratch, no template.

I repeated that request forty times, varying one thing only: the accessibility context the AI receives along with the task.

• Nothing. The request, and not a word about accessibility.
• The magic phrase. A single extra instruction: "Make it accessible."
• A11Y.md 1.8.0. The old version of my standard.
• A11Y.md 2.0.0. The new version, which ships a contrast calculator in the package.

The builders were two real coding agents, Claude Code (Anthropic) and Antigravity (Google), five journeys per condition on each.

Figure 2 · What a journey builds — two real screens, one from each version

two of the 133 screens generated with the standard during the study · blind sample, untouched

And the honesty rules of the previous studies all still stand:

• Recipe filed with a public registry before the first data point, on an open research platform (OSF). After registration, moving any parameter leaves a public trail.
• Sealed versions. Every file under test carries a digital seal. A single changed comma breaks the seal.
• Clean computer. Each agent ran in a profile built from scratch, with a probe confirming nothing of mine leaked into the test. In the previous study that leak happened, and it is documented in public.
• The ruler took a beating before measuring. The instrument was attacked by two independent teams of AIs whose mission was to cheat it. Twenty-nine confirmed defects were fixed before any collection.
• And a rehearsal changed the apparatus. In the pilot, the test environment blocked the new version's contrast calculator. Testing the package without letting the tool run would be testing something else. I allowed one single command, redid the rehearsal, watched the tool run, and only then froze everything.

The comparison that matters is the internal one: new version against old version, on the same agent, in the same week, with the same model underneath.

Because the AI evolves underneath the document. Today's old version delivers more than the very same old version delivered in August, without a single comma of the text having changed.

The scoreboard

The automatic ruler is an accessibility-auditing robot: axe, the same engine that works behind Lighthouse. It sweeps the seven screens of each journey and counts the errors a machine can catch.

Look at the Claude Code column first.

The magic phrase, the old version and the new version tie at zero.

Against nothing, though, the gap remains huge: seven errors against zero.

Between the two versions of the standard, there was no error left to measure.

On Antigravity, the numbers ask for a bit more care. The 0 and the 1 in the table come from here:

• With the old version, three of the five journeys finished with no robot errors at all. The typical journey is clean, which is why the table shows 0.
• With the new version, three of the four valid journeys had some error, and only one finished clean. The typical journey has 1 error, and that is the 1 in the table (the fifth journey gave up midway, and I tell that story shortly).

It looks worse, and maybe it is.

But the difference is one error in the typical journey, and the protocol warned in writing, before collection, that at this sample size small differences cannot be told apart from luck.

I record the number the way it came out.

And, with it recorded, the table's bottom line fits in one sentence: the September 2026 model hits the ceiling of the automatic ruler with any minimally serious instruction.

Why only five journeys per cell?

Because each journey is an entire bookstore built by a real agent — the forty of them cost days of collection and millions of tokens.

The design assumed that size with eyes open: the study's main bets were not subtle averages, they were binary criteria — a defect class vanishing, a document existing or not existing — that stay readable even in a small sample.

The warning about noise is not an excuse discovered afterwards. It is a limitation declared beforehand, with the bets designed to fit inside it.

The new version's three promises

2.0.0 was launched promising three measurable things. And I wrote down, before collection, what would happen to each one if it failed.

The detail of each one comes next.

1. The ARIA ladder had to disappear. It did not.

Translation: ARIA is a vocabulary of labels that tells the screen reader what each thing on the page is ("this is a set of tabs", "this tab is selected").

The catch is that these labels only work as a set.

Announcing "there are tabs here" and not marking which buttons are the tabs is promising an interaction that does not exist: whoever browses with a screen reader hears the promise and finds the door shut.

In the previous study I named this defect the half-climbed ARIA ladder, because climbing half a ladder leaves you somewhere worse than the ground.

The criterion was hard and binary. The class had to vanish from every journey of the new version.

It appeared once.

A single component, on a single screen.

The criterion failed, and the rule I wrote myself before collection orders me to say so in the headline.

Then comes the part that changes the story.

The old version produced exactly the same defect, in the same component, at the same rate.

The full denominators: the ladder appeared in one of the nine journeys of the new version and in one of the ten of the old one. In the other seventeen, none. Always on Antigravity, never on Claude Code.

The mold is a carousel of cover thumbnails announced as a set of tabs, with the buttons missing the tab label.

And one detail matters: the ruler does not hunt carousels only. It looks for any composite ARIA pattern delivered halfway, be it a menu, an accordion or anything else. In this study, the only one that appeared was this mold.

The new version made nothing worse.

It just did not eliminate what it promised to eliminate.

<div role="tablist">          ← the promise: "there are tabs here"
  <button class="thumb">…</button>  ← what was missing: role="tab"
  <button class="thumb">…</button>
  <button class="thumb">…</button>
</div>

And the zoom-out is the most important data point of the study.

In the previous study, this defect was a ladder of ninety-eight elements spread over seven screens.

Today it is one element, on one screen, in both versions at the same time.

The problem got about a hundred times smaller, and it shrank equally on both sides.

The author of that improvement does not live in my file.

It lives in the model, which evolved underneath it.

2. Contrast had to improve. It tied.

Translation: contrast is the difference in luminosity between text and background. Below a minimum, people with low vision cannot read, and nobody reads in the sun.

There is an exact arithmetic for it, and the new version ships a calculator in the package so the AI never does contrast in its head.

The numbers first. The nineteen journeys that ran with the standard add up to one hundred and thirty-three screens.

Across them, the robot failed contrast in two journeys: one from the old version, one from the new, both on Antigravity. On Claude Code, none, with either version.

The bet asked the new version to have fewer cases than the old one.

One against one is a tie.

And a tie is not fewer.

What the new version demonstrably changed was the process.

Where the calculator ran, every color pair came out measured by machine and recorded in the report.

And the study threw in, for free, the proof of why that matters: in a report where the calculator did not run and the AI did the arithmetic by hand, eight color pairs were declared verified.

I redid all eight with the standard's own calculator.

Two were wrong. Approved on paper, failed on the arithmetic.

3. The token diet could not regress anything. It did not.

Translation: tokens are AI's currency. Every word read and written costs.

The new version is leaner in text, and the promise was to slim down without abandoning a single obligation.

Kept.

The obligation classes that were at zero with the old version remain at zero with the new one, on both agents, and the new version costs three to four percent more per task, inside the band projected at launch.

The measurement that did not tie

The standard requires two accountability documents at the end of every journey (readers of the first study will remember them):

• the verification report — what the AI checked, with which tool, and the result of each check;
• the decision log — what it decided along the way, and why.

They exist for a simple reason: without them, "it's accessible now" is just a promise. With them, it becomes a claim a person can audit afterwards.

The scoreboard of this measurement:

• With the standard, any version, any agent: nineteen of nineteen journeys delivered both.
• Without the standard, magic phrase included: zero of twenty.

It is the same one hundred percent against zero of the previous studies, now replicated on a stronger generation of models.

Hold on to that contrast.

It carries the conclusion of the entire study.

The journeys that gave up

Forty journeys collected, thirty-nine valid, and two withdrawals that became data:

• One new-version journey on Antigravity spent over half a million tokens writing a plan in prose and never created a single file. It got stuck wanting to ask the user for a configuration, in a test environment where no user exists to answer, and ignored the documented default path that the other four journeys in the same cell used without drama.
• One magic-phrase journey hit a denied permission and traded working for asking for permissions.

Both count as agent failure, one in five in each one's cell, by the same ruler.

The decision is dated in the study's public journal, together with a mistake of mine that the audit found in that episode.

It comes now.

The audit that corrected me

Before any human eye saw any screen, a panel of twenty-two audit agents swept the data's entire path, from rendering to statistics.

Ninety-five checks came back clean.

Ten findings survived the skepticism.

Five were sentences of mine, in the study's public journal, that the evidence contradicted.

The worst of them: during collection, an Antigravity bug was killing sessions.

I wrote a fix into the configuration, validated it, recorded in the journal that it was applied... and the program itself rewrote the file minutes later, silently discarding the fix.

The entire collection ran without it.

It worked anyway, and no result changed, but the sentence in my journal was false.

Today it carries the correction of the correction, with a date and time as proof.

The other findings change the reading without toppling any verdict, and they are all dated in the journal.

Two examples: a secondary instrument turned out to be blind to the coding style of one model's new generation, and that arm's number was voided for honesty's sake; and one of the programs stopped reporting part of the tokens spent, so that arm's cost is a floor, not a measurement.

A study where the audit finds nothing has not been audited.

What my eyes saw and the robot did not

The last gate before publication is human, by registered rule: twenty-four screens drawn by lot, renamed to codes so I would not know which condition they came from, and the map sealed before the first look.

Seventeen passed clean.

Seven made me frown:

• three book covers that never loaded;
• one dead sales chart;
• two carts with the layout stretched thin;
• and titles overflowing their covers.

Figure 7 · Two of the seven — visible to the naked eye, invisible to the robot

real captures from the blind sample · neither one raises a single error in the robot auditor

I investigated all seven before breaking the seal.

None was the fault of my measuring apparatus.

All were real defects of the pages the AIs generated.

And here is the finding this stage delivered for free: none of these seven defects, all visible to the naked eye, raises a single error in the robot auditor.

Zero on the robot auditor never meant a good page. I knew that in theory. Now I have the captures.

What this tie teaches

The safe sentence would be: the standard keeps making an enormous difference against no standard at all, and between versions the models caught up with the rest.

It is true and it is the summary.

But the underlying finding deserves to be spelled out.

A file like A11Y.md has two layers of different natures.

The layer that teaches.

How to compute contrast, when to use each label, what an image needs to carry. That layer has an expiration date, because each model generation absorbs more of it.

This study measured the exact moment the models caught up with what my old version taught.

The layer that demands.

Deliver the report, log the decisions, measure with the tool and attach the arithmetic. That layer does not expire when the model improves. Knowledge, the model absorbs over time.

Demands, it only meets when there is a contract.

And the value of that layer, in this study, is concrete:

• The two contrast errors were only findable because the required report listed every color pair with its declared number. The demanding layer did not prevent the error — it made the error auditable. And auditable is fixable.
• Where the calculator ran, the arithmetic came out right by construction. The error only appeared where the demand existed and the tool did not run.
• In the nineteen journeys with the standard, you can know what the AI checked and what it decided. In the twenty without, you cannot even ask.

The one hundred percent against zero survived two model generations and two different agents.

The models caught up with the layer that teaches. The layer that demands survived.

The adversary in this race was never the old version of the file.

The adversary is irrelevance.

And the way not to be caught by it is for the standard to be, with each version, less repetition of what the model already knows and more contract about what it does not do unless someone demands it.

The study has already produced concrete consequences:

• Version 2.0.1, already published, names inside the standard itself the carousel mold that survived both versions, with its two honest exits: complete the tabs for real, or drop the tabs announcement and keep plain buttons.
• Version 3 inherits the mandate this study wrote. Shrink where the model has learned. Harden where no model delivers on its own.

Links and references

• Public registration of Study 3, frozen before collection: osf.io/wt5n4
• Complete raw data, from screens to quarantines, with digital seals: Zenodo, DOI 10.5281/zenodo.22548474
• The standard, the frozen instruments and the deviations journal: github.com/fecarrico/A11Y.md
• The consequence in the standard: release v2.0.1
• The previous movements: the first study and the registrations of Study 1 and Study 2