14 July 2025 | News
Image Courtesy: Public Domain
Flatiron Health today announced that the novel findings from its research, “Using large language models for scalable extraction of real-world progression events across multiple cancer types,” have been presented at the American Association for Cancer Research (AACR) Special Conference in Cancer Research: Artificial Intelligence and Machine Learning, which took place July 10-12, 2025, in Montreal, QC, Canada.
The research, led by a multidisciplinary Flatiron team of clinicians, research scientists, and machine learning engineers, demonstrates the power of large language models (LLMs) to accurately and efficiently extract real-world cancer progression events from unstructured electronic health record (EHR) data across 14 cancer types. Study findings concluded that LLMs achieved F1 scores similar to expert human abstractors and produced nearly identical real-world progression-free survival estimates, underscoring the potential of AI to scale high-quality clinical endpoint extraction for oncology research and care. This research utilized on the recently published Validation of Accuracy for LLM/ML-Extracted Information and Data (VALID) Framework to assess the quality of LLM-extracted real-world data, uniquely evaluating both the LLM and an expert human abstractor against a duplicate expert human abstracted reference dataset to directly compare performance.
The LLM that Flatiron scientists optimized for this research was provided by Anthropic—the leading AI safety and research company that builds the Claude family of models.
“AI and machine learning are fundamentally transforming how we generate and use real-world evidence in oncology,” said Stephanie Reisinger, Senior Vice President & General Manager, Real-World Evidence at Flatiron Health. “This research exemplifies how Flatiron is harnessing AI and multimodal data to unlock new insights from oncology real-world data—accelerating clinical research, improving patient outcomes, and setting a new standard for evidence generation in cancer care.”
“We’re excited to share evidence that large language models can approach, and in some cases, may even exceed expert human performance in extracting critical and clinically nuanced cancer progression endpoints from EHRs,” said Aaron B. Cohen, MD, lead author, Head of Research Oncology, Clinical Data at Flatiron Health and practicing oncologist at Bellevue Hospital in New York City. “Scalable, high-quality extraction of such an important and complex endpoint like progression will open new doors for novel research, predictive modeling, and more personalized patient care.”
In addition to this work, Flatiron Health will also present “Fairness by design: End-to-end bias evaluation for LLM-generated data,” further highlighting the company’s leadership in applying AI and real-world data to advance oncology research. Specifically, this study demonstrates the VALID framework’s ability to evaluate both the quality and fairness of LLM-extracted data, highlighting the importance of ongoing bias assessment.
