Mass General Brigham conducted a study evaluating generative AI chatbots in clinical differential diagnosis scenarios. The study found that these systems frequently misalign with patient context and can propose irrelevant or incorrect conditions. Researchers observed variability in performance across models and cases, highlighting risks of relying on AI for diagnosis without human oversight. These results underscore current limitations of generative AI in real-world medical decision making. The authors recommend cautious deployment, rigorous evaluation, and clear human-in-the-loop guidelines.

• Evaluate performance across diverse differential diagnosis scenarios
• Highlight limitations and biases in AI-assisted diagnoses
• Emphasize the need for human oversight and supervision
• Develop standardized evaluation protocols for clinical AI tools
• Integrate AI tools into clinical workflows with clinician review gates

AWS releases Part 2 of the Nova Forge SDK series, presenting a practical workflow for fine-tuning Nova models with data mixing. The guide covers data preparation, training with data mixing, and evaluation, providing a repeatable playbook. Data mixing lets you tailor models to domain-specific data without sacrificing general capabilities. The post notes that data mixing preserves near-baseline MMLU scores while delivering a 12-point F1 improvement on a dataset. The article frames data mixing as a practical approach to customize models while maintaining versatility.

• Prepare domain data and curated datasets
• Apply data mixing to preserve general capabilities
• Train and fine-tune Nova models with Forge SDK
• Evaluate performance using MMLU and F1 metrics
• Document a reusable playbook for customization

Cloudflare updates its AI Platform to provide unified access to multiple AI models via a single API endpoint. The Cloudflare AI Gateway enables interaction with third-party providers like OpenAI and Anthropic using the same AI.run() binding used for Cloudflare's own models. This move aims to reduce vendor lock-in and simplify switching between models and providers with minimal code changes. The update supports more flexible agent workflows and expands the ecosystem by providing a common access layer. Developers can now use a single integration to access both Cloudflare and third-party models.

• Consolidate access to models via a single API surface
• Enable switching between providers with minimal code changes
• Extend compatibility to third-party models through AI Gateway
• Maintain security and governance features alongside new integrations
• Simplify multi-model experimentation and deployment