Ah yeah I wasn't thinking necessarily about creating guidelines with ML—more like highlighting/synthesizing relevant excerpts from existing guidelines (e.g. NICE, American College of Cardiology, etc.). But I didn't know that individual clinics had their own guidelines in addition to the "official" ones).

Noted! I'll work on fixing the underlying problem, and more importantly I'll add bigger/better disclaimers to make sure nobody is taking these answers too seriously.

Yeah I feel like right now the answers are in a weird spot...it's understandable/usable by motivated consumers/patients (like WebMD), but looking at primary research is more of a clinician/doctor thing (like UpToDate). Truthfully I'm more interested in making a better WebMD, since I think most health decisions (diet, exercise, sleep, supplements, OTC meds, etc.) are made without any MD input.

Hah I yeah that's from OpenAI (they've been having API issues ever since ChatGPT launched). I've switched the reranker to use a different provider (for now) so it should be working again.

Agreed! I think "add more/bigger/better disclaimers" is my biggest takeaway from this post.

Yeah definitely working on that for "most recent"—but I've read conflicting things about using "most cited" as a proxy for trustworthiness (there are studies in some disciplines showing that citations are negatively correlated with reproducibility).

Indeed! I'm interested in trying to make a health-specific version of these tools. Elicit/Consensus are general-purpose research tools, which means it's harder for them to add health-specific views (e.g. a table of treatments sorted by effect size, a list of symptoms and their prevalence, a list of side effects and their prevalence, etc.). Obviously I haven't built any of that yet, but I'm working on it.

I hear you! I definitely want to "do no harm" here—I think while I'm still testing things out I need to plaster a lot more warnings around the site like "THIS IS A PROOF-OF-CONCEPT, NOT MEDICAL ADVICE, DO NOT TRUST."

My ultimate goal would be to make the "curation" of the answer much clearer, so that this would be more of a research tool (like Pubmed) and less of a magic oracle.

Definitely not overly critical—the whole reason I posted was to get critiques! I think you're right that I can go further with explainability, and I also think that there are ways to use NER, etc., to give more interesting answers (e.g., a table of treatments sorted by effect size or adverse events). I'll keep working in this direction.

Super helpful! This broadly aligns with other feedback I've gotten, that I should be focusing on guidelines in addition to/instead of clinical research.

Yeah just saw that! Definitely going to give it a test drive in the next few days.