Artifact
No Source Gets Used Until Its Review Is Written: A Clinical Knowledge Base
Neutrality is a myth. Everyone has presuppositions. So every source that comes in gets read, written up and weighted before anything is taken from it.
A thing I built. This page says what it is and what is true about it. Dan Stephens writes about psychology, AI systems and the method behind both at danieltstephens.com.
Neutrality is a myth. Everyone has presuppositions. Presuppositions exist in hierarchies. So a filter of some kind is needed.
This exists because of those four sentences. When I learned the pattern of building a knowledge base an AI reads from, the filter seemed like an obvious missing piece. If I am going to use a piece of information, I need to know two things about it. How much do I trust it, given what I already believe. And how do I use it, given what I already believe.
Those are different questions. A source I distrust is not useless. It is usable differently.
What it is made of
- A loading dock. Nothing is quoted from a source until a written review of that source exists.
- A written review for each source, covering what it claims, how strong its evidence is, and where it sits against what I already hold. I read and approve every one of them.
- A score on each review, and a ceiling: no note drawn from a source can claim more confidence than its source’s own adjusted score allows.
- A weighting rather than a door. A source that never got a written review can still be used, with less influence on the finished work than a reviewed one.
- A record of what I have read in full, and what I know only at working distance.
Where it stands
In daily use, and it is the slow part on purpose.
The gate is the bottleneck and it does slow things down. What it buys is rigor, integrity and confidence. A reader deciding whether to work this way needs both halves of that sentence.
Approving a review is not the same as having read the source cover to cover. I read and approve every review; I have not read every underlying source in its entirety, and the page says so rather than letting the stronger claim stand. In a couple of cases a preliminary review is what decided whether a source was worth my reading time at all, which turns the gate into a filter on my scarcest resource rather than only on what enters the collection.
Why there are far more pages than sources
A page here is one idea, not one source. That is why the two counts are so far apart.
Most pages are notes about a concept, a pattern, a protocol or a question, and a single source can throw off a dozen of them. Many more come from clinical experience and from working through a problem, and have no outside source behind them at all. The written reviews are the gate on what comes in from outside. They are not an inventory of what is inside.
By the numbers
- 2,891 pages in the knowledge base, 2,058 of them carrying a confidence value
- 41 written source reviews on file, each one read and approved before anything was taken from that source
- Every note capped at five times its source’s adjusted quality score, so the strongest source in the collection caps any note drawn from it at 0.773
- 0 sources rejected on score alone; the written rule reserves refusal for disqualifying problems
- 2 questions every source has to answer: how far do I trust it, and how may I use it