Research infrastructure
Infrastructure for research that keeps itself honest
Software plus a set of rules for investigating something without fooling yourself.
- Our role
- Designed and built it
- Built with
- Python · SQLAlchemy · Content-addressed store · mypy --strict
- Where it got to
- Research system, ran a real engagement
The idea
Investigate a market, a hypothesis or a claim properly, and be able to show afterwards exactly how each conclusion was reached.
What it actually needed
Everyone intends to check their own conclusions. Almost nobody does it in the week the conclusion is convenient, and by then the sources have moved and the reasoning has been compressed into a sentence nobody can unpack. So the honesty could not be a habit. It had to be something the software does whether or not the researcher is in the mood.
What got built
- An evidence store that cannot drift
- Every source kept whole and addressed by a hash of its contents, so the stored copy is provably the thing that was fetched and not a version edited later to agree with the conclusion.
- Claims linked to what supports them
- A conclusion is connected to the specific evidence behind it instead of to a summary of that evidence, so the chain can be walked in either direction.
- Memory where nothing is deleted
- When something is superseded, that is recorded instead of applied silently, so the history of what was believed, and when it stopped being believed, survives.
- A stopping condition, agreed first
- What would end the investigation is written down before it starts, so research concludes on evidence, not on fatigue or on finding something agreeable.
- Published results, including the failures
- Findings come out as written deliverables, with the refutations at the same weight as the survivors.
The decisions that shaped it
Attack every hypothesis, not just the doubtful ones
The doubtful ones were never the danger. A claim is at its most dangerous at the moment it starts looking correct, because that is exactly when nobody feels the need to check it. That is why the discipline had to be mechanical instead of a matter of judgement.
Publish the result that went against us
One engagement produced a verdict that contradicted the commercial outcome we wanted, and one experiment returned a flat zero. Both were written up and delivered as they were. A research system whose inconvenient results quietly do not get published is not a research system.
In production
Ran a real engagement end to end. Every theory opened was attacked, the majority were shown to be wrong, and the refutations were published as written deliverables alongside the survivors.
What this is worth
Research that records what it got wrong at the same weight as what it got right. The expensive kind of wrong is the kind nobody wrote down.
Most research goes wrong in the same quiet way. You learn something, and a week later you remember the conclusion perfectly but have completely forgotten whether you checked it or inferred it. Both feel like knowledge. Neither arrives with a label.
CPOS is the software and the rules that stop that happening. An evidence store that keeps every source by a hash of its contents. A graph linking claims to the artifacts supporting them. Versioned memory, where nothing is deleted and supersession is recorded, not applied silently.
The rule that does the work
Findings that can be traced to a preserved source go in one section. Claims that merely sound traceable go in another, marked as such. Reasoning from the second as though it were the first is not permitted.
It reads like bureaucracy until you watch what happens without it.
What it found, including against us
Its first real engagement opened eleven theories about where a market opportunity was, and deliberately went looking for evidence against every one of them, not only the doubtful ones.
Four survived. Seven were refuted, several after they had already looked correct. The final verdict went against the commercial outcome we were hoping for, and that is what got written up and delivered.
Five experiments from that period are written up and published in full, including the one whose result was a flat zero. A negative result you can trust is worth more than a positive one you cannot.