Multimodal foundation models for healthcare

Sophont builds open, universal medical AI that understands pathology, neuro­imaging, clinical text and more—empowering clinicians and researchers worldwide.

Read our medical AI manifesto →

Team

Partner With Us

Collaborate with us at the frontier of open medical AI. Whether you're a healthcare provider, research institution, or investor, let's connect.

Research Publications

fMRI brain visualization

Reconstructing the mind's eye: fmri-to-image...

P. Scotti, ... T.M. Abraham

NeurIPS (spotlight), 2023

Introduced a novel method to reconstruct seen images from fMRI brain signals by aligning brain activity with image embeddings and a diffusion model.

Read Paper → GitHub Code →
Abstract AI visualization

MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

P. Scotti, ... T.M. Abraham

ICML, 2024

Demonstrates that a shared fMRI-to-image model can be fine-tuned on just one hour of data from a new person, enabling high-quality brain decoding with minimal data.

Read Paper → GitHub Code →
Chest X-ray on a monitor

A vision–language foundation model for the generation of realistic chest X-ray images

C. Bluethgen, ... T.M. Abraham, et al.

Nature Biomedical Engineering, 2024

Presents RoentGen, a model that can generate realistic, high-resolution X-ray images from clinical text prompts, aiding in data augmentation and education.

Read Paper → GitHub Code →
Doctor interpreting medical scans

A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation

Z. Chen, ... T.M. Abraham, et al.

AAAI, 2024

This model generates preliminary reports from chest X-rays, significantly improving radiologists' diagnostic efficiency without compromising accuracy.

Read Paper →
Code on a dark screen

LLMs in medicine: evaluations, advances, and the future

T.M. Abraham

Blog post, 2025

Provides a comprehensive overview of how LLMs are evaluated for medical applications, highlighting the critical need for more robust, multimodal evaluation frameworks.

Read Post →
Abstract colorful generative art

DALL-E Mini

B. Dayma, ... T.M. Abraham, et al.

Blog post, 2021

Details the community-led creation of DALL·E mini, an open-source text-to-image model, explaining its architecture, training, and impact.

Read Post → GitHub Code →
3D visualization of a human brain

EduCortex: browser-based 3D brain visualization of fMRI meta-analysis maps

P. Scotti, et al.

JOSE, 2021

A lightweight, open-source JavaScript library for visualizing 3D brain maps directly in a web browser to share and explore fMRI meta-analysis results.

Read Paper → GitHub Code →
Digital pathology slide

Label- and slide-free tissue histology using 3D epi-mode quantitative phase imaging...

T.M. Abraham, et al.

Optica, 2023

A microscopy technique using deep learning to create virtual H&E stains of unlabeled tissue, providing real-time histology without chemical staining.

Read Paper →
Retina scan equipment

Progress Towards Decoding Visual Imagery via fNIRS

M. Adamic, ... P. Scotti, et al.

arXiv, 2024

This study explores using fNIRS, a portable alternative to fMRI, for decoding visual imagery, showing promise in classifying imagined visual categories.

Read Paper →
Abstract data visualization with flowing lines

Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers

K. Crowson, ... T.M. Abraham, et al.

ICML, 2024

Introduces the Hourglass Diffusion Transformer (HDiT), a novel architecture for generating high-resolution images directly in pixel space.

Read Paper →
Abstract representation of connected nodes

Trainees' perspectives and recommendations for catalyzing the next generation of NeuroAI researchers

A. Luppi, ... P. Scotti, & H. Gellersen

Nature Communications, 2024

Identifies key challenges in NeuroAI training and provides recommendations for fostering an interdisciplinary environment for the next generation of scientists.

Read Paper →
Microscope view of cells

NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery

R. Kneeland, P. Scotti, et al.

CVPR, 2025

Introduces NSD-Imagery, the first large-scale fMRI dataset dedicated to mental imagery, designed to advance brain decoding models for imagined visuals.

Read Paper →
Abstract neural network graphic

MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery

R. Kneeland, ... P. Scotti, et al.

Under review

Presents a robust multimodal architecture that adapts fMRI-to-image models from visual perception to mental imagery, improving decoding of imagined scenes.