CPTSD Research Pipeline

A retrieval system over contested clinical literature. It maps a debate; it does not resolve one.

Each question takes about a minute. The sources in this corpus disagree with each other, and the answer is built to show that rather than smooth it over.

Built and operated by Julien Radoff — infrastructure, retrieval design and the pipeline behind it. julienradoff.com · GitHub · LinkedIn

What this is

A question-answering system over the clinical literature on post-traumatic stress, built around the fact that the literature does not agree with itself. An answer here sets out what the sources hold in common, where they part company, what remains unsettled, and — given the same prominence as the rest — what this corpus has nothing to say about.

Answers are built from retrieved passages and cited back to them. The system can still reach for something it knows rather than something it read — when it does, the review pass printed under every answer is where you find out. It is a reader rather than an authority, and it is checked rather than trusted.

How a question is answered

The corpus is held in four separate layers rather than one pool. They are not tiers of authority, and they are not a quality ranking. They are positions in an argument, and they are kept apart so that each one has to be heard rather than being outvoted by whichever has the most pages.

  1. The clinical establishment — DSM-5, ICD-11, treatment guidelines. Supplied to every stage, including the review pass, because it is the frame the other three are arguing with.
  2. Neuroscience and pharmacology — mechanism, and the trial evidence for it.
  3. The counter-voices — practitioners working outside the mainstream.
  4. The institutional critique — why the research looks the way it does: what gets funded, what gets published, what does not get asked.

A single pool would bury layers 3 and 4 under layers 1 and 2, because the establishment publishes more and is cited more. Giving each layer its own retrieval quota is what makes a disagreement visible instead of averaged away.

A question then moves through four stages:

  1. The question is rewritten into a search query, because what a person asks and what retrieves well are rarely the same string.
  2. Each layer is searched on its own, with its own quota, so a loud layer cannot crowd out a quiet one.
  3. A judge reads what came back from each layer and sends that layer back to search if the passages do not bear on the question. Retrieval is a loop, not a single shot.
  4. Synthesis, into the shape you see in an answer: agreements, positions, contradictions, unsettled questions, absences.

Then a critic re-reads the finished answer against the passages it was built from and against the diagnostic criteria, looking for claims the sources do not support. Its findings are published alongside the answer under Self-review, rather than used to quietly revise it. An answer that needed correcting is more informative than one that appears not to have.

The four model stages: rewrite, retrieve and judge, synthesise, critic — with the diagnostic layer supplied to both synthesis and criticism.
The four stages. The judge is the only loop; the critic deliberately has no path back to the answer. Full size.

What it runs on

Three hosts in AWS, provisioned with Terraform and configured with Ansible: one for ingestion, one for the vector store, one serving the site. Postgres holds the chunked corpus and is the system of record; Chroma holds the embeddings and is rebuildable from it. A FastAPI service sits behind nginx and TLS.

Images are built, tagged and pushed to a private registry by a single script, which checks the working tree against the remote first and declines to deploy when nothing has changed, verifies the built artifact actually contains the change it was asked for, and bumps the version only once the deploy has succeeded.

Monitoring runs on a separate Kubernetes cluster that probes this site from outside it — Prometheus and Alertmanager, with alerts for a dead probe as well as a failing one, on the principle that a system should not be the only witness to its own health.

Infrastructure: three application hosts in AWS, a build host, a disposable dev box, a private registry, and a separate Kubernetes cluster probing the site from outside.
Three application hosts, built from a separate build host, watched from outside. Full size.

How it is checked

Answers are graded against a hand-written answer key: a set of questions with the responses a careful reader of this corpus should give, written down before the system was asked them. A harness runs the whole set and reports where the answers drifted from what the sources support.

That harness exists to find faults rather than to confirm good behaviour, and it has. A retrieval distance threshold that turned out never to be applied. Prompt scaffolding leaking into text meant for readers. A reviewer that could not see the diagnostic criteria it was reviewing against, and so reported correct citations as fabrications. A tendency to state facts about diagnostic manuals this corpus does not contain, and then cite a source that says nothing of the kind.

Each one is written up with how to reproduce it and what the fix would cost, whether or not the fix has shipped. The self-review printed under every answer is that same discipline running live, in public, on the answer you just read.

Limits worth stating

Not medical advice. This is a research tool: it reports what the literature says and where it disagrees. It does not diagnose and does not recommend treatment.

Julien Radoff · production build unknown · deployed 2026-11-03