This project was an opportunity to experiment with LangChain and LangGraph by building a grounded Q&A tool over clinical guidelines. It uses guardrails, retrieval, structured extraction, a classification step, and an answer that changes shape depending on that classification.

The source code can be viewed on GitHub:

This project focused on Type 2 diabetes, drawing on two public sources: the ADA’s Standards of Care and the UK’s NICE guideline. They can agree, disagree, or stay silent on a user prompt, depending on how the comparison and classification part of the graph handles it.

Prompts such as “What is the first-line pharmacologic treatment for type 2 diabetes?” (and others in example-prompts.md) work in the live demo below.

If the embed doesn't load, open the demo directly.

The flow, at a glance

flowchart TD
    Start([User Question]) --> KW[urgent_check_keyword]
    KW -->|hit| ER[emergency_redirect]
    KW -->|no hit| LLMU[urgent_check_llm]
    LLMU -->|hit| ER
    LLMU -->|no hit| SC[scope_classifier]
    SC -->|patient-specific| SR[scope_redirect]
    SC -->|general question| RET[retrieve_per_source]

    RET --> RA[retrieve: ADA]
    RET --> RN[retrieve: NICE]

    RA --> EA[extract_structured_claim: ADA]
    RN --> EN[extract_structured_claim: NICE]

    EA --> CMP[compare_claims]
    EN --> CMP

    CMP -->|same| SYN1[synthesize: unified answer]
    CMP -->|scope difference| SYN2[synthesize: explain scope]
    CMP -->|conflict| SYN3[synthesize: present both + grounded rationale if stated]
    CMP -->|silent| SYN4[synthesize: note gap]

    SYN1 --> End([Answer + Citations])
    SYN2 --> End
    SYN3 --> End
    SYN4 --> End
    ER --> End
    SR --> End

Grounded factual recall walks the main spine down through compare_claims. Cross-source comparison is what the four synthesize branches are for. The scope guardrail and urgent/emergency detection each trigger a different redirect, short-circuiting straight to End.

Setting up the system

I chose to go through OpenRouter rather than a provider API directly, as I can change models easily to test different outcomes. For now, it runs on a free-tier model with the key’s credit limit set to $0 so we don’t bill anything accidentally. Embeddings run locally via sentence-transformers, so indexing needs no API key either, just CPU time.

Source documents are downloaded once as PDFs and read from disk, not scraped live. Chunking is content-aware: NICE’s text is already atomic per numbered recommendation, so short documents pass through unchanged; ADA’s long narrative sections get recursively split, each chunk carrying its parent section’s metadata so a citation can point back to it (code: chunking.py).

The chunks land in two separate vector indices, one per source, instead of one pooled index. A pooled index would hand the model passages from both sources together, inviting it to blend them into one confident answer even when they disagree. Separate indices let the graph retrieve from ADA and NICE independently and compare what each actually says.

A Streamlit chat UI sits on top: uv run streamlit run app/streamlit_app.py. This is my first time using Streamlit, and I found it very easy to setup.

Grounded factual recall

A prompt such as “What is the first-line pharmacologic treatment for type 2 diabetes?” produces a metformin recommendation cited to ADA section 13, plus a note that the retrieved NICE passage doesn’t explicitly address a first-line drug for this framing of the question. Both sides stay grounded, instead of the model filling in the NICE side from what it already knows about diabetes drugs.

Behind the scenes, the graph retrieves from the ADA index and the NICE index independently, then runs each source’s passages through an extraction step that turns them into a structured claim rather than free text: a recommendation, the population it applies to, an evidence grade where the source states one, and a citation identifier (code: extraction.py).

A comparison node then checks whether the two claims agree, disagree, or one is silent, and synthesis writes the answer to match. Every claim traces back to a specific citation, rendered under the answer: [13] [1.45.2] above, ADA’s section and NICE’s recommendation number. Checking a claim against the source text is the whole grounding argument. That makes citations the one piece of the frontend worth getting right.

Cross-source comparison

A prompt such as “How do ADA and NICE differ on managing chronic kidney disease risk in type 2 diabetes?” produces two intact, separately cited recommendations instead of one blended answer. ADA’s is broad: incorporate agents that lower cardiovascular and kidney risk regardless of their effect on blood glucose. NICE’s is narrower and more specific: metformin plus an SGLT-2 inhibitor, and only once eGFR is above 30.

The comparison node’s job is narrow on purpose: classify the two claims as same, scope_difference, conflict, or silent, nothing else. Letting the classifier also explain a disagreement is how a model ends up quietly deciding which guideline is “right” (code: compare.py). Synthesis handles that harder call instead, in a separate branch per classification. The conflict branch carries the constraint that matters most in the whole system:

"conflict": (
    "The sources genuinely disagree. Present both positions without picking a winner. "
    "Explain WHY they differ ONLY if a claim's stated_rationale field is non-null — quote or "
    "paraphrase that stated rationale. If both stated_rationale fields are null, explicitly say "
    "the sources don't state a reason for the difference. NEVER invent a rationale."
),

stated_rationale is set during extraction, only when the source text itself gives a reason. If neither guideline explains why they diverge, the answer says so plainly instead of the model reaching for a plausible-sounding explanation. A fabricated “why” reads as authoritative in this domain, so it’s a hard rule in the prompt, not a suggestion (code: synthesize.py).

scope_difference is a separate bucket for cases that look like disagreement but aren’t: different eGFR thresholds framed differently, say, not an actual conflict about what to do. Collapsing it into “conflict” would manufacture a disagreement; collapsing it into “same” would flatten a real distinction. Either way, the chat UI marks it with a “guidelines differ here” badge.

Scope guardrail

A prompt such as “Should I stop taking my metformin? My doctor prescribed it but I feel sick.” produces a reframe back to the general question and a pointer toward an actual care team, not an answer. It’s not urgent and not off-topic. It’s asking for individualized medical advice, not what a guideline says generally.

It can also be difficult to control over-triggering. “If I have CKD, does that change treatment recommendations?” is phrased with “I,” but it’s about a population, not a specific situation, so it should still get answered generally. The scope classifier’s prompt spells out that distinction with worked examples on both sides (code: scope_classifier.py).

A second layer backs it up: the synthesis prompt restates the system’s role on every call, general or not: “You explain what published clinical guidelines say. You do not provide individualized medical advice.” A borderline message that slips past the classifier is still self-limited at generation. Neither layer holds the line alone.

Urgent/emergency detection

A prompt such as “I’m experiencing pain in my chest that I think is due to low blood sugar. Can I take aspirin alongside my normal diabetes medication?” produces an immediate redirect to emergency care, no retrieval involved, even though the second half reads like an ordinary medication question. The guardrail sits at the very front of the graph, before any RAG machinery runs, and the answer renders distinctly in the UI rather than as a normal chat bubble.

The check runs in two stages: a fast keyword/pattern match first, then an LLM classifier that only runs if the keywords miss, to catch phrasing a fixed pattern list wouldn’t (code: urgent_check.py). This message reads “pain in my chest,” not the literal “chest pain” the keyword pattern looks for, so it’s the LLM classifier stage that catches it: the whole message, guideline question included, before any of it reaches retrieval. Both stages are deliberately biased toward over-triggering: a false positive costs a mildly annoying redirect, a false negative means an emergency gets a calm, cited RAG answer instead of “call 911.” The classifier’s own prompt says outright to bias toward “yes” if unsure, instead of aiming for the balanced precision and recall a classifier would normally target.

Checking that the guardrails hold

A demo only proves a handful of hand-picked examples work. The eval suite checks the same guardrail and disagreement-detection claims against a full question set.

The question set is hand-written, read directly out of the ADA and NICE source text, not generated. It’s split across six categories: grounded factual recall, cross-source comparison, structured extraction, the scope guardrail, urgent-symptom detection, and adversarial edge cases like a hypothetical reframed to try to slip past the scope classifier. Guardrail categories grade pass/fail deterministically, since “did it trigger or not” has a clean right answer. Comparison and recall are run-and-reported for now. Grading those well needs reference answers this project hasn’t built out yet.

What this was for

The diabetes framing gave the project a useful topic to explore LangChain. Medical Q&A works here because the answer space is limited to two named sources, not the model’s general knowledge, and guardrails decide what’s answerable at all before anything reaches an LLM. This is a learning project, not a clinical tool: the guardrails are a first pass, not a substitute for the regulatory, legal, and clinical review a real deployment would need.