Accuracy and Safety of Three Large Language Models Answering Turkish Questions on Adult Immunization

Abstract

Vaccine misinformation contributes to falling immunization coverage, and people increasingly bring vaccine questions to chatbots. We evaluated three large language models on 72 Turkish questions on adult immunization, in two arms: 40 patient-focused questions and 32 mapped to CDC/ACIP, WHO or Turkish Ministry of Health recommendations. The 216 responses were blind-coded and scored independently by two blinded specialists against a guideline-anchored answer key. Binary accuracy was 95.1% for Claude Opus 4.8, 93.8% for Gemini-3.1-Pro, and 86.1% for GPT-5.5. Patient-facing questions, including the classic vaccine myths, were answered well (93.8–96.2%), but accuracy on technical guideline content fell to 76.6% for GPT-5.5, which was also the most readable model and referred the reader to a clinician in only 53.1% of guideline-arm responses. Responses were more often incomplete than wrong.