Governance

How the agentic-AI ecosystem behind thescidoc.com is governed: autonomy tiers, human-in-the-loop pattern, the independent Critic/Evaluator, drift detection, and security posture. Posted publicly so colleagues, residents, IRB members, and curious readers can see how the system actually works.

The system in one paragraph

A set of about twenty scheduled remote agents running on Anthropic Claude Code Routines (CCR). Each agent fires on a cron schedule, queries authoritative sources (PubMed eutils, FDA, named clinical guideline organizations, the user's own PMID archive), and composes a Gmail draft. No agent ever sends. A human (Jon) reviews every draft and decides whether to ship. A separate independent Critic agent scores every digest weekly against a 7-dimension rubric and pushes verdicts to a private GitHub archive. A Master Orchestrator rolls everything up into one executive brief on Friday evening.

Autonomy tiers

Every agent is classified by what it's permitted to do. Tiers are explicit so a reader can see which agents are passive readers and which can act on the world.

Human-in-the-loop

Every patient-facing or colleague-facing artifact is composed as a Gmail draft, never sent. The clinician (Jon) reviews each draft, edits if needed, and manually hits send. The handouts on this site are AI-drafted but physician-reviewed; the weekly research roundup at /this-week is pulled live from the SCI digest agent's output, with a defensive scrub for private references.

This isn't a marketing claim. It's implemented in the agent prompts as a hard rule ("Create a Gmail DRAFT only. Never send.") and the Critic verifies it during weekly review.

Independent quality review — the Critic / Evaluator

A separate agent fires every Friday at 3pm ET and scores every digest produced that week against seven dimensions:

1. Structure — required sections present, in order.
2. Source quality — every paper cited with PMID, journal, year.
3. Numerical specificity — concrete effect sizes, n, p-values, sensitivity.
4. Voice and tone — matches the audience the agent was written for.
5. Guardrails — no PHI, no leaked secrets, no fabricated PMIDs, no individual medical advice in patient-facing pieces. Failures here are blocking regardless of other scores.
6. Explainability — can a clinician trace each clinical claim back to its source in under thirty seconds? This dimension exists to address the “black box” failure mode flagged by Collaco et al. (2026).
7. Uncertainty calibration — does the agent honestly flag thin or contested evidence rather than overstating? Tier labels (HIGH/MEDIUM/LOW) should match study design. Single-study claims should be noted as such. Contested topics flagged.

Verdicts (✅ ACCEPT, 🟡 NOTES, ⚠️ CORRECT, ❌ REJECT, 🔥 CRITICAL) and per-dimension scores are archived to a private GitHub repository for trend tracking. CRITICAL findings can trigger an automatic re-run of the failing agent (capped at two per Critic run); they never silently modify the failing agent's prompt.

Bounded self-improvement

Only one agent in the ecosystem (the Master Orchestrator) is allowed to edit another agent's prompt, and only under three named failure patterns: missing Gmail keep-alive, paper cap too high, missing 502 retry. Each edit:

• Backs up the before-state and after-state to GitHub before applying.
• Is capped at one edit per orchestrator run, ever.
• Is logged in the master brief for the clinician's visibility.

Any failure pattern outside the whitelist surfaces as a “manual fix needed” recommendation rather than an automatic edit.

Drift detection

A Topic Memory Index agent fires every Saturday and walks the prior week's digest archive, building two index files: a topic-recurrence index (which clinical topics keep appearing across agents) and an agent-voice index (top-twenty word lists per agent, with a Jaccard-similarity drift score versus prior weeks). When an agent's voice drift exceeds a threshold for two or more weeks, it's surfaced in the master brief for review.

Privacy and security

• No PHI in any digest. Patient-facing handouts use generic SCI clinical knowledge; clinical-team digests reference published literature only. Agents are instructed to refuse anything resembling a patient identifier.
• Avanthi brief is private. The Critic and Master Orchestrator confirm it fired but never read its content; it's never archived to the public-adjacent digest archive.
• Defensive site filter. The pages that pull from the digest archive run a regex scrub on every rendered markdown to strip any line referencing private content.
• Quarterly credential rotation. A scheduled reminder fires on the first of January, April, July, and October to rotate the GitHub Personal Access Token used by all agents and verify the current token is still valid. An audit log lives in the private archive at _governance/pat_rotation_log.md.
• Education only. Nothing on this site is medical advice. The footer on every page says so.

Reference

The seven-dimension Critic rubric, the autonomy-tier classification, and the credential rotation cadence are all designed against the framework described in:

Collaco BG, Haider SA, Prabha S, Gomez-Cabello CA, Genovese A, Wood NG, Bagaria SP, Gopala N, Tao C, Forte AJ. The role of agentic artificial intelligence in healthcare: a scoping review. npj Digital Medicine 2026; 9:345. doi:10.1038/s41746-026-02517-5.

That paper's scoping review found that none of the seven published agentic-AI healthcare systems had long-term memory or adaptive behavior over time, and that black-box reasoning, credential security, and overconfidence on thin evidence were the dominant unsolved problems. The governance posture above is a direct response to those findings.

Education only. Not medical advice. The system described here drafts artifacts a clinician reviews and ships; the agents themselves do not practice medicine. For emergencies call 911. Questions: see /about.