The decision this playbook helps you make
Before launching a business experiment, write down the decision that the evidence will change, the people and behavior you will observe, the maximum exposure you will accept, and the conditions for continuing, revising or stopping. Agree on those rules before reviewing outcomes. A useful protocol makes an unattractive result interpretable and prevents an attractive but irrelevant number from becoming permission to scale.
This playbook is for an operator testing a searchable information product, a small service, a directory or another lightweight business proposition. It assumes that the operator can publish a limited offer and observe at least one relevant response. It does not assume that traffic, demand or revenue already exists. Its purpose is to turn a vague intention to try something into a bounded decision with an accountable next step.
DatumHarbor's existing observations show why this discipline matters. Our October sitemap comparison established a publishing-surface change, while search and commercial outcomes remained unknown. A separate display case exposed numeric labels without supporting measurements. Those cases identify evidence gaps; they do not demonstrate a successful market test. This article adds a prospective decision protocol rather than claiming that the protocol has already increased conversion or prevented business losses.
Start with a decision rather than an activity
Publishing ten pages is an activity. Deciding whether a defined audience values a specific answer enough to take a meaningful next step is an experiment question. The difference changes the design. If the only acceptance condition is that ten pages exist, a successful build finishes the task even when nobody wanted the content. If the decision concerns demand, the protocol needs evidence about people, their problem and their response.
Write the decision as an allocation: after this test, will we invest another bounded amount of effort in this proposition, change one part of it, or stop the present version? Avoid deciding the entire future of a company from one small observation. An initial test might justify another interview, a revised offer or a second observation window. It rarely settles every issue of pricing, delivery capacity and acquisition economics simultaneously.
The UK Government Service Manual provides a useful public reference for distinguishing discovery from building. Its discovery guidance places understanding the problem before committing to a service and makes progression a decision informed by learning. We borrow that separation of learning and commitment, not its administrative process. The protocol below is our own application to small commercial experiments, not a government endorsement of a business model.
Define one falsifiable proposition
A proposition should identify a user, a situation, a proposed benefit and an observable response. For example, a hypothetical operator might ask whether independent repair businesses preparing supplier inquiries will use an evidence checklist and request help applying it. This example is invented for explanation. It is not a DatumHarbor customer study, and no number in the later calculations represents measured performance from that market.
Keep the initial proposition narrow enough that a negative result has meaning. A statement that businesses need useful information is too broad to challenge. A statement about one role facing one recurring decision allows the operator to examine the job, the existing workaround and the consequence of leaving it unresolved. Include the circumstances under which the problem occurs, because an occasional inconvenience may require a different product from an urgent operational failure.
Describe a plausible alternative explanation before launching. Readers might arrive because a headline promises a free template rather than because they need paid help. A professional might praise a guide out of courtesy but have no intention of using it. Recording these alternatives does not invalidate every response. It tells the reviewer what additional evidence would distinguish interest in the content from demand for the proposed service.
Separate assumptions into a dependency chain
A commercial proposition often relies on several assumptions: the problem exists, the intended audience can be reached, the proposed explanation is understandable, the offer fits a real workflow, someone can authorize a purchase, and delivery can be economical. One experiment need not test all of them. It should identify which assumption is currently most consequential and which later assumptions remain open even if the test succeeds.
Use a dependency chain to avoid buying evidence about the wrong stage. A guide can test whether an explanation is understandable without testing willingness to pay. An inquiry can reveal a delivery requirement without proving that the buyer accepts the price. A payment can establish a transaction without establishing repeat demand. Name the link that the current observation addresses and keep the other links explicitly unresolved.
In the protocol, write the strongest claim that a positive result could support. Then write the stronger claims it cannot support. For an initial inquiry test, a justified continuation might be to investigate delivery requirements with qualified respondents. It would not automatically justify hiring a team or purchasing expensive infrastructure. This advance boundary prevents an early signal from being stretched into a much larger investment decision.
Choose an observation unit and a qualification rule
The unit could be an identified organization, a consenting participant, a unique inquiry or an attributable completed task. Choose the unit that matches the decision and that can be observed lawfully within the existing workflow. Do not switch casually between page requests, people and businesses. One organization may generate several requests, while one shared account may represent several employees. The protocol needs a consistent rule for handling those cases.
For a service inquiry, define qualification in terms of the actual problem rather than flattering engagement. An illustrative rule might require a stated business context, a concrete task and permission to respond. Spam, internal tests, accidental submissions and requests unrelated to the offer would be recorded separately. Qualification should not depend on whether the operator likes the answer; negative feedback from a relevant prospect can be valuable evidence.
Keep enough information to support the classification without publishing personal details. A public evidence summary can report the rule and an aggregate result while private communication remains private. If the operator lacks an authorized collection channel, that is a prerequisite gap. A blank inquiry spreadsheet does not establish zero demand. The experiment must either obtain an appropriate channel or choose an observation that it can actually collect.
Specify the denominator before using a rate
A conversion rate requires a numerator and a denominator with compatible scope. Dividing qualified inquiries by all server requests would mix a business event with a technical request population that may contain assets, bots and repeated visits. Dividing inquiries received this week by visits from last month would introduce a different mismatch. State the intended population, period and exclusions before calculating a percentage.
Consider a deliberately hypothetical calculation: eight qualified inquiries from two hundred eligible, attributed visits equals four percent under that definition. The arithmetic is eight divided by two hundred, multiplied by one hundred. It does not establish a general conversion rate for the business. If attribution is incomplete, the published result needs that limitation. If the two hundred are requests rather than eligible visits, the proposed rate should not be reported using the same label.
A small experiment may sensibly avoid a conversion rate entirely. A carefully documented sequence of relevant conversations can reveal a workflow problem without supplying a representative population estimate. Explain whether the test seeks qualitative discovery, a directional operational measure or a quantitative comparison. Numerical formatting must not imply a sampling design that the experiment did not use. Precision in a spreadsheet cannot repair an ambiguous population.
Write a minimum evidence package
List the artifacts needed to review the decision: the offer as participants saw it, the eligibility rule, the observation period, the collection method, the recorded outcomes, exclusions and known interruptions. Include the version of any page or questionnaire that materially influenced the response. This package need not be an elaborate software system. A dated document and a small structured record may be sufficient if they are consistent and recoverable.
Preserve both positive and negative observations. A participant who cannot use the proposed output may reveal an implementation barrier that a complimentary response misses. Keep contradictory evidence attached to the relevant claim rather than averaging it into a vague confidence score. Distinguish a direct statement, observed behavior and the operator's inference. Each can be useful, but each supports a different kind of conclusion.
Define what will remain private and what can be safely shared. The public page should explain the method and support the conclusion without exposing customer identities, credentials, correspondence or sensitive commercial information. Sanitize a copy for publication rather than using the private working record as a web asset. Check the evidence package independently from the article, because an innocuous narrative can still link to an inappropriate raw file.
Set a budget that makes stopping possible
A stopping rule is ineffective when the operator has not defined the exposure being limited. Record a maximum duration, production effort and any authorized cash expenditure. These are planning commitments, not claims about how much time previous work consumed. A lightweight experiment should also identify the operational dependency that could expand its cost, such as manual research for each respondent or a delivery task requiring specialist review.
Use the budget to constrain the next decision rather than to force a positive result. Reaching the effort limit without enough interpretable evidence means that the planned test did not settle the question within that exposure. It does not prove that no market exists. The reviewer can stop the current approach, redesign recruitment or request another explicitly bounded test. Continuing automatically because the question is still interesting defeats the purpose of the limit.
Keep safety and authorization limits separate from ordinary business thresholds. A privacy issue, unsupported material claim or unauthorized deployment can require an immediate stop even when response numbers look attractive. A low response count may justify a planned review, but it does not justify bypassing those safeguards. The protocol should identify who may resolve each type of interruption and which work can continue independently.
Choose continuation, revision and stopping criteria
Write three branches with actions, not merely labels. Continue means a specified next investment after a stated evidence package is available. Revise means changing a named assumption or component and defining a new test. Stop means ending the current allocation while retaining the learning. These are operational decisions. They need not correspond to a permanent judgment about every possible version of the idea.
For the hypothetical inquiry example, continuation could require several independently qualified conversations showing a repeatable task and an identifiable decision maker. Revision could follow relevant responses that consistently reveal a different problem from the offered one. Stopping the current recruitment method could follow a completed observation window with no interpretable eligible responses. Exact thresholds must reflect the operator's decision and exposure; this article supplies no universal magic number.
Add an inconclusive branch. Collection failure, insufficient reach or incompatible records may leave the question unanswered. Calling every inconclusive test a failure penalizes honest measurement; calling it a success rewards storytelling. The useful response is a dependency-specific action: repair collection, alter recruitment or stop investing in the current test design. State which evidence would permit another attempt and what maximum additional exposure would be acceptable.
A worked protocol for a hypothetical information service
Imagine an operator considering a supplier-evidence review service for small repair firms. The proposed first step is a public checklist explaining which documents to assemble before asking a supplier for clarification. The experiment asks whether relevant firms have a sufficiently concrete problem to discuss an assisted review. It does not yet ask whether an automated product can replace professional procurement work.
The audience definition is independent repair firms facing an active supplier-document question. The observation unit is one organization, with repeated messages grouped under that organization when the available evidence supports doing so. The qualification rule requires a concrete task and permission to reply. The operator preserves unrelated requests separately. No real organizations or inquiries are represented by this example, and the example introduces no claimed market size.
The offer should state what the checklist covers and what it does not. If no paid service is ready, the page must not imply immediate delivery capability. The response channel should explain the next step accurately, such as a request to discuss the problem. The experiment must not manufacture scarcity, testimonials or successful customer outcomes. Honest scope makes the resulting response easier to interpret because the promise is known.
Suppose the operator plans a bounded review after an agreed observation period. If relevant organizations describe the same costly preparation problem and request a discussion, the next action may be a limited manual service design. If respondents mainly need a different kind of document, the next action may be a revised checklist. If recruitment never reaches eligible organizations, the result is a recruitment limitation, not a demonstrated absence of the underlying problem.
Handle changes without rewriting the original question
Experiments often require corrections. A broken response link should be repaired, but the interruption belongs in the record. A factual error should be corrected promptly, and the reviewer should consider whether it affected the offer participants saw. A major change in audience, promise or response channel can create a new version of the test rather than a seamless continuation of the original one.
Record the effective time of material changes and preserve the previous version. If outcomes before and after the change are combined, explain why the combination remains meaningful. Otherwise review the periods separately. Do not move the success threshold merely because the observed number sits just below it. A revised threshold may be reasonable for a future protocol, but it should not be presented as the rule that governed the completed test.
Separate an implementation repair from an improvement claim. Restoring a working form establishes that the collection channel is available again; it does not establish higher customer demand. A changed title may alter reach, but without suitable observation it does not prove why a later inquiry occurred. The protocol should preserve these distinctions so that necessary maintenance does not become an invented explanation of performance.
Review outcomes with an evidence-to-action table
At the review, take each material claim and ask what observation supports it, what alternative remains plausible and which next action is proportionate. A useful table has columns for the claim, source, scope, limitation and decision. It can be brief. The important feature is that a reviewer can trace the action to evidence rather than to the operator's enthusiasm or disappointment.
For example, a recorded discussion may support that one relevant organization has a problem, while leaving frequency across the market unknown. A completed paid pilot may support that one transaction occurred, while leaving margin and repeatability unresolved. A cluster of similar objections may justify revising the offer, while leaving the original audience definition intact. Make the narrow supported statement first, then describe the conditional interpretation.
When the evidence contradicts the preferred narrative, preserve it. The value of a small test is often the ability to avoid a larger unsupported commitment. That value should not be quantified as money saved unless a defensible counterfactual and cost record exist. A simple statement that the team stopped the current version for a documented reason is more useful than an invented estimate of avoided losses.
Distinguish exploration from a causal comparison
The protocol described here can organize discovery and limited operational learning. It does not automatically create a randomized controlled experiment. If the decision concerns whether one treatment caused a change, the operator needs an appropriate comparison design, assignment process and analysis plan. Seasonal differences, changing traffic sources and different participant populations can explain an apparent improvement without the proposed treatment being responsible.
For a small operator, a responsible first move may be to narrow the claim rather than to imitate a sophisticated statistical study. Report that a particular offer produced certain observed responses in a defined setting. Do not report that the wording increased conversion by a causal percentage unless the design supports that statement. A before-and-after chart can describe observations while still leaving the cause unresolved.
Likewise, do not calculate statistical certainty from a convenience sample without examining the assumptions behind the calculation. A handful of selected interviews can be valuable for discovering vocabulary and workflow constraints, but they do not establish population prevalence. The decision may still be useful: commission a better test, refine the audience or stop an unsuitable delivery method. Useful learning does not require pretending that every test estimates a market parameter.
Prepare the next measurement before expanding
A positive early response should identify the next uncertainty. If people request help, investigate what delivery requires. If a manual pilot works, examine repeatability and cost. If customers return, examine whether acquisition and fulfillment remain sustainable. This sequence keeps expansion attached to evidence. It also makes it possible to stop a promising-looking idea whose delivery burden exceeds what the operator can responsibly support.
Set a review date and a named recovery condition for missing measurements. Search data may become available through an authorized export; inquiry evidence may require a properly managed channel. Continue independent research whose facts are available, but do not silently substitute publication counts for missing outcome measures. The measurement-contract guide explains how to preserve that uncertainty in public reporting.
Keep the experiment history readable after the next version launches. Archive the original question, observed evidence, decision and limitations. The experiment directory identifies the assets under observation, while the methodology page explains the broader evidence principles. A new version should build on that history without converting an earlier unresolved claim into a proven result.
Frequently asked questions
How many responses are enough to continue?
There is no response count established by this playbook that is valid for every decision. A small number of relevant conversations may justify another discovery step but not a large investment. Define the minimum evidence needed for the specific next allocation, explain its limitations, and distinguish qualitative learning from estimating a population rate. If uncertainty is too large for the proposed commitment, choose a smaller next step or a stronger design.
Can a test continue when search data is unavailable?
Yes, if the proposition can be investigated through another appropriate and authorized observation. A direct conversation can reveal a workflow problem without a search export. However, the absence of search data prevents claims about search exposure or acquisition performance. Record those fields as unknown and avoid assigning organic-search credit to responses whose source cannot be established. The test question should match the evidence actually available.
Should the operator stop immediately after a negative comment?
A single comment should be interpreted in context. Determine whether the participant fits the intended audience, what the objection concerns and whether other evidence supports it. A safety or factual issue may require immediate action, while an isolated preference may justify further investigation. The protocol should state which events trigger an immediate interruption and which belong in the scheduled review.
What if the original threshold was poorly chosen?
Preserve the original threshold and explain why it was unsuitable. Review what the completed test can support under its actual design, then write a revised protocol for the next attempt. Do not retroactively claim that the revised rule governed the earlier decision. An honest protocol amendment can improve future work without erasing the limitations of the first version.
Does stopping mean deleting the published pages?
No. Ending an investment decision and removing a public URL are different actions. A page may retain useful evidence, references or reader value after a test stops. Review its factual accuracy and role before changing its publication status. Preserve historical conclusions with dates and limitations. Any removal or redirect should follow the site's existing authorization and technical process rather than being an automatic side effect of a business label.
What should a public experiment result disclose?
Disclose the question, observation period, sample, method, supported result, limitations and resulting decision. Include enough evidence for readers to understand the conclusion while protecting private records. Clearly label hypothetical examples and distinguish advice from measured outcomes. If the observation was incomplete, say so. An accessible page and a completed publication workflow establish delivery, while demand, revenue and repeatability require separate evidence.
Conclusion: make the next commitment reviewable
A good small-business experiment does not need an impressive dashboard. It needs a question that matters, an observable response, a bounded exposure and rules that connect the evidence to an action. Write the protocol before the results, preserve changes, and make inconclusive evidence an explicit outcome. Continue only into a defined next step whose uncertainty and cost are understood. Stop the current version when the evidence or the permitted exposure no longer justifies it, while preserving the learning that makes the next decision better.
References and observation scope
- UK Government Service Manual: how discovery works, accessed 12 November 2026. Supports the separation of problem discovery from commitment; this commercial protocol is DatumHarbor's own synthesis.
- DatumHarbor October observation sample: two dated technical snapshots, not a market study.
- DatumHarbor measurement display case: one dashboard observation, not a customer outcome.