Prima
A health system-scale vision language model for brain MRI decision support
Model Weights
Direct downloads, no access request required.
Overview
The global demand for MRI studies has risen steadily, straining health systems, prolonging turnaround times, and intensifying physician burnout -- challenges that disproportionately affect patients in low-resource and rural settings. Researchers at the University of Michigan used a large academic health system as a data engine to develop Prima, the first vision-language model built as an AI foundation for neuroimaging that runs directly on real-world, clinical MRI studies. They curated UM-220K, a dataset of over 220,000 MRI studies -- more than 5.6 million 3D sequences and 362 million 2D images -- each paired with its corresponding radiology report. Trained on UM-220K via a contrastive objective, Prima uses a hierarchical vision architecture that yields general, transferable MRI representations.
Key Capabilities
- Preliminary differential diagnoses for suspected conditions
- Radiologist worklist prioritization
- Clinical referral recommendations
- Transferable feature extraction for downstream research tasks (Alzheimer's prediction, autism screening, brain age estimation)
Validation
Prima was evaluated in a 1-year, prospective, health system-wide study spanning roughly 30,000 MRI studies. Across 52 radiologic diagnoses -- including neoplastic, inflammatory, infectious, and developmental lesions -- it achieved a mean diagnostic AUC of 90.1% ± 5.0%, outperforming other state-of-the-art general and medical AI models by a wide margin. Scale of training data proved essential to this performance. Prima also offers preliminary differential diagnoses, radiologist worklist prioritization, and clinical referral recommendations across diverse patient demographics and MRI systems, and its learned features transfer to tasks outside routine radiology reporting, including Alzheimer's disease, autism, and brain age prediction.
Explainability
Prima's predictions are explainable through LIME (Local Interpretable Model-Agnostic Explanations), which highlights the MRI regions, or volume tokens, that most influenced a given prediction. In clinical vignettes -- including a diffuse low-grade glioma that progressed to malignant transformation, a spontaneous brain abscess that resolved after drainage, and a pediatric shunt malfunction -- Prima correctly identified the imaging changes driving its updated predictions between a patient's initial presentation and their follow-up scan.
Fairness
The underlying health system shows real scheduling inequities in patient wait times -- by county population, patient location, and day of the week. Despite this underlying bias in the training data, Prima was shown to maintain consistent fairness across all of these dimensions.
Technical Architecture
- VQ-VAE-based volume tokenization, dividing each MRI volume into subvolume patches and encoding them into discrete tokens
- A hierarchical vision transformer -- a sequence ViT over individual series, then a study ViT that aggregates series into one study-level representation
- CLIP-style contrastive training aligning imaging with LLM-summarized radiology report text
- Transfer learning: the tokenizer and transformers are frozen, and a lightweight MLP is trained on top for each downstream task
FastGlioma
Visual foundation models for fast, label-free detection of diffuse glioma infiltration
Model Weights
Gated on HuggingFace -- request access, then download via huggingface-cli.
Overview
FastGlioma is an AI-powered visual foundation model for rapid, label-free detection of tumor infiltration during brain surgery. It processes fresh surgical tissue imaged with stimulated Raman histology (SRH) and delivers a result in roughly 10 seconds -- fast enough to inform decisions in the operating room.
Technical Approach
- Patch tokenizer: processes individual image sections from whole-slide SRH images
- Whole-slide encoder: a vision transformer that aggregates patch-level information into a single infiltration score
- Pretrained via large-scale self-supervision (~4 million images) on optical microscopy data, then fine-tuned for clinical glioma assessment using ordinal metric learning
Clinical Performance
In prospective testing across 220 glioma patients at multiple international centers, FastGlioma achieved an average AUC of 92.1% ± 0.9%, and outperformed existing surgical guidance methods -- including fluorescence-guided approaches -- in a head-to-head comparison involving 129 patients. It also showed zero-shot generalization to other adult and pediatric brain tumor diagnoses beyond its training targets.
Intended Applications
FastGlioma targets three adult-type diffuse glioma categories defined by WHO standards: astrocytoma (IDH-mutant), oligodendroglioma (IDH-mutant with 1p/19q-codeletion), and glioblastoma (IDH-wildtype). It outputs a continuous score from 0-1, mapped to four infiltration categories.
| Level | Score Range | Classification |
|---|---|---|
| 0 | 0-25% | Normal tissue |
| 1 | 26-50% | Atypical cells |
| 2 | 51-85% | Sparse infiltration |
| 3 | 86-100% | Dense infiltration |
Limitations
FastGlioma is not intended for surgical resection guidance around eloquent cortical or subcortical structures, pediatric patients, non-primary brain tumors, or non-neoplastic pathologic tissue.
DeepGlioma
AI-based molecular classification of diffuse gliomas using rapid, label-free optical imaging
Overview
Molecular classification has transformed the management of brain tumors by enabling more accurate prognostication and personalized treatment, but timely molecular diagnostic testing is often out of reach for brain tumor patients, complicating surgical and adjuvant treatment planning. DeepGlioma is a rapid (under 90 seconds), AI-based diagnostic screening system that provides molecular classification of diffuse gliomas by combining stimulated Raman histology (SRH) -- a rapid, label-free, non-consumptive optical imaging method -- with deep learning-based image classification, predicting the molecular features the World Health Organization (WHO) uses to define the adult-type diffuse glioma taxonomy.
Workflow
A patient with a suspected diffuse glioma undergoes surgery for tumor biopsy or resection. The SRH imaging system is portable and operates directly in the operating room via simple touch-screen instructions -- a freshly excised tissue specimen is loaded into a premade microscope slide with no tissue processing required, and a 3x3mm SRH image takes about 90 seconds to acquire. DeepGlioma is trained on a multimodal dataset: an SRH image encoder learned via weakly supervised, multi-label contrastive learning, and a genetic encoder trained on public diffuse glioma genomic data (TCGA, CGGA, and others) to capture known co-occurrence relationships between mutations. The two encoders are integrated via a transformer for multi-label prediction of diffuse glioma molecular diagnostic mutations, and because predictions are patch-level, spatial heatmaps can be generated to visualize which SRH image regions drove a given prediction.
Results
In a prospective, multicenter, international testing cohort of diffuse glioma patients (n=153), trained using University of Michigan data only (n=373), DeepGlioma achieved an average molecular genetic classification accuracy of 93.2% and correctly identified the diffuse glioma molecular subgroup with 91.5% accuracy. Individual ROC curves showed AUROC of 95.9% for IDH-1/2, 97.7% for 1p/19q-codeletion, and 85.7% for ATRX classification. Against the primary testing endpoint -- comparing IDH1-R132H immunohistochemistry (IHC) to DeepGlioma for IDH mutational status detection -- DeepGlioma achieved 94.2% balanced accuracy overall and 97.0% for patients 55 or younger, driven by a +10% gain in sensitivity from also detecting non-canonical IDH mutations that IHC alone misses.
