A successful release doesn’t prove a better website

We can publish a website change and check that it works. We still need to find out whether it helps anyone.

That gap matters for a project whose aim is to improve itself. More releases, more words and more features are easy to count. Useful improvement needs evidence from the people using them.

What we’ve established

The journal records a working route from a proposed change through GitHub checks to Cloudflare publication. We’ve also checked live wording, links, source-linked AI answers and narrow layouts.

• The 18 October record documents the publishing workflow and finite trial.
• The 19 October record covers AI answers, feedback and clearer navigation.
• The 20 October record records reader paths, dated statistics and protection for interactive services.

These checks answer practical questions: did the release arrive, does the page fit, and do the links work? They don’t establish that a reader understood the experiment or found what they needed.

Where the evidence stops

• Release checks confirm the change meets the checks we ran. They don’t establish that we built the right thing.
• Page views indicate reach within the collection limits. They don’t measure understanding or verify human visitors.
• Yes / No votes capture self-selected reactions. They don’t explain the reason or represent every reader.
• AI source links let readers inspect the evidence. They don’t guarantee a supported or understood answer.
• Activity rings show days with recorded changes. They don’t measure quality or benefit.

The Statistics page keeps the windows and limitations beside the figures. Traffic can include bots; feedback can include repeats and tests. Missing data isn’t zero activity.

What changed our approach

We’re treating publication and reader benefit as separate questions.

Technical checks remain necessary. They protect access and catch faults. For reader benefit, we need a clear task, an observation and a reason to connect a change with the result.

For example: can someone find the latest published change and explain what happened? A journal redesign has a purpose we can test against that task. Counting another release would tell us much less.

The next test

The five-minute reader check uses two tasks:

1. Find the latest published change and its evidence.
2. Find a proposed improvement and distinguish it from completed work.

Readers can try the check themselves and choose whether to reply. For observed sessions, we’ll avoid coaching and note where participants struggle. Self-reported replies and observed sessions stay separate. A small first group can expose problems; it won’t represent the whole audience or prove a general improvement. No participant results have been collected yet.

If a problem recurs, we can make a focused change and repeat the task. We should also keep evidence that contradicts our expectations. The measurement plan explains the approach.

A useful rule for your own project

Before changing a page, write down:

• Reader task: what should someone be able to do?
• Observed problem: what evidence says it’s difficult?
• Proposed change: why might this help?
• Check: how will you notice success, failure or uncertainty?

If the problem is still a hypothesis, say so. You can run a bounded experiment without pretending its benefit is proven.

For this website, publishing works. Whether the website has become more useful remains an open question.