Key finding
We had two different facts to preserve: some checks completed successfully, and the scheduled review process stopped before completing its planned coverage. Reporting only the successful checks would hide the second fact. Reporting the whole website as failed would also be wrong. A process timeout is evidence about the process, not automatic evidence about market demand or even website availability.
Observation period and sample
This case uses two DatumHarbor review attempts on 12 and 19 October 2026. The review was intended to observe our public platform and the experiment inventory. The first attempt had a 150-second execution limit and the second a 60-second limit. Both were stopped at their respective limits before producing the full review record. This is a two-run operational case, not a controlled performance benchmark.
| Attempt date | Execution limit | Full sequential review | Separate evidence |
|---|---|---|---|
| 12 Oct 2026 | 150 seconds | Incomplete | Bounded technical observations recorded separately |
| 19 Oct 2026 | 60 seconds | Incomplete | Independent production validator: 27 checks passed |
Method and evidence
The review process fetched a sequence of sites and linked pages. Individual network requests had timeouts and retries, so a limit on a single request was not a limit on the full run. We recorded the process outcome separately from the individual HTTP observations. The sanitized first-party dataset contains the dates, execution limits and completion states underlying this case.
The second observation also included a separate production validator that completed all 27 of its checks. That validator covered a different, explicitly defined set of assertions. Its success did not finish the sequential review, supply observations for skipped work or establish a commercial outcome. We retained both records instead of combining them into a single unqualified green status.
Why partial success was easy to misread
A progress stream can contain many successful responses before a process stops. If the final report is written only at the end, the output may never include a completed inventory or a dated summary. A reader then has to distinguish between a check that passed, a check that failed and a check whose result was never obtained. Absence of a result belongs to the third category.
Our second execution limit was shorter than the first. That does not show that the underlying review became faster or slower: the budgets differed and the work did not finish. We cannot use these two stopped runs to estimate an improvement percentage, saved labor or average request latency. The demonstrated problem is missing completion, not a measured performance regression.
The completion protocol we derive from the case
- Declare the intended coverage before execution: properties, page sample and assertions.
- Give the full run a deadline as well as limiting each network request.
- Preserve completed observations as observations, with timestamps and sources.
- At the deadline classify unobserved assertions as UNKNOWN, not PASS or zero.
- Record whether the intended review completed. A separate validator's success cannot replace this field.
- When using a bounded substitute, describe its coverage and gaps rather than claiming it was the original full crawl.
This is a reporting and execution method derived from the failure. The two observations do not prove that every future implementation of the protocol will finish within a particular budget. A production implementation needs its own completion test and should expose when the budget is exhausted.
How we separate website health from market evidence
A successful public page response is a useful availability observation. A broken link can justify a technical repair. Neither advances an experiment to a market-signal state. The same boundary applies to successful review execution: a completed monitoring task proves that its defined checks ran, not that users want the product.
We therefore keep operational coverage, publishing activity, search exposure and commercial validation in different fields. A project can be available while its traffic remains UNKNOWN. It can also have an incomplete review while independent technical samples remain useful. Explicit labels preserve the value of the evidence without stretching its meaning.
Limitations
There were only two attempts, on different days and under different time limits. We did not randomize requests, measure a controlled baseline or attribute the delay to a particular server. We do not publish private infrastructure details or identify a child project as the cause. The case establishes a completion gap in our monitoring process; it does not establish a general network-speed result or a customer-facing outage.
Conclusion
Keep a passing assertion and a completed process as separate claims. In this case both review attempts were incomplete, even though a separate production check passed. The practical lesson is to preserve partial evidence, label the missing coverage and make completion an explicit acceptance condition.
Read the related method on sitemap growth without market validation, the experiment methodology and the public experiment inventory.