MONAI: the open source PyTorch framework for medical imaging AI

AIHealth

On-premise clinical platform with local LLMs, RAG on FHIR/DICOM data, diagnostic support, remote follow-up. Architecture designed for the MDR pathway.

Discover AIHealth →

Digital Health

Medical software development compliant with CE and MDR regulatory standards. Clinical decision support systems, AI integration in clinical workflows.

Discover →

The need for a dedicated framework

As of 2020 medical imaging deep learning is a scientifically mature but infrastructurally fragmented discipline. The community works with PyTorch (dominant framework after TensorFlow 1.x’s eclipse in research) and ad-hoc scripts for every lab: each group reimplements medical data transforms (spacing, orientation, CT HU normalisation, MR bias field correction, patch-based training), specific loss functions (Dice, Focal, Tversky), clinical metrics (Dice, Hausdorff, surface distance). Code is rarely reusable across labs, duplication is enormous.

In parallel, NVIDIA — the main AI hardware supplier — has developed since 2018 Clara Train SDK as an end-to-end platform for medical imaging model training, integrated with the Clara ecosystem for deployment. Clara Train is powerful but proprietary and tied to the NVIDIA stack: it does not meet the requirement for an open, neutral ecosystem.

In mid-April 2020, a consortium of institutions announced a response: MONAI — Medical Open Network for AI.

The founding consortium

MONAI was publicly announced on 24 April 2020 as a collaborative open source project jointly led by:

• NVIDIA — deep learning GPU expertise, code donated from Clara Train
• King’s College London (KCL) — School of Biomedical Engineering & Imaging Sciences (BMEIS), with M. Jorge Cardoso and Sebastien Ourselin as key figures; historical open source contributor in medical imaging
• Massachusetts General Hospital / Brigham and Women’s Hospital — clinical centre of excellence, linked to the Surgical Planning Laboratory (which hosts 3D Slicer)
• Stanford University
• Vanderbilt University
• University of Warwick
• Nanyang Technological University (Singapore)

Other centres — including DKFZ (Maier-Hein’s nnU-Net group) — collaborate informally with MONAI, keeping their parallel projects while contributing relevant components.

The project is distributed under Apache 2.0 and hosted on GitHub (Project-MONAI/MONAI).

First release

The initial MONAI 0.1 version was released in April 2020; 0.2 is in preparation with further components. In the early weeks post-publication, the repository accumulated hundreds of GitHub stars and external contributors started opening pull requests with new transforms, networks, loss functions.

Architecture

MONAI is built as a Python library on top of PyTorch, with optional PyTorch Ignite integration for the training loop. Framework structure:

monai.data

Loading of medical datasets: native support for NIfTI, DICOM via ITK/SimpleITK, NRRD, MetaImage. DataLoader preserving metadata (spacing, origin, orientation) across the pipeline. CacheDataset, PersistentDataset for IO optimisation on large datasets.

monai.transforms

The transformation library is the practical value core of MONAI. Dedicated to the medical context:

• Spatial: Spacing (resampling to target voxel spacing), Orientation (RAS/LAS/LPS alignment), Resize, CenterSpatialCrop, RandSpatialCrop
• Intensity: ScaleIntensity, ScaleIntensityRange (HU in CT), NormalizeIntensity, HistogramNormalize
• IO-aware: LoadImage as pipeline transformation
• Augmentation: RandFlip, RandRotate, RandAffine, RandGaussianNoise, RandGaussianSmooth, RandShiftIntensity, RandScaleIntensity, Rand3DElastic
• Dictionary-based transforms: every transform has a “d” variant (e.g. SpacingD) that works on {image, label} dicts preserving the association

The typical pipeline is described as a list of composable transforms, with deterministic and stochastic compose support.

monai.networks

Out-of-the-box CNN architectures, specific or adapted to the domain:

• UNet (2D/3D) — standard implementation
• DynUNet — dynamic depth/features adaptation to input size, inspired by nnU-Net
• SegResNet — residual U-Net
• HighResNet — high-resolution residual network
• DenseNet variants
• AHNet, VNet, RegNet, Classifier architectures

Base modules (Convolution, ResidualUnit, UpSample) allow building custom networks by reusing primitives.

monai.losses

Medical-segmentation-specific loss functions:

• DiceLoss — classical for segmentation
• DiceCELoss — Dice + cross-entropy combination (typical in nnU-Net-style training)
• FocalLoss, TverskyLoss, FocalTverskyLoss for very imbalanced classes
• GeneralizedDiceLoss — class-frequency-weighted

monai.metrics

Standard evaluation metrics:

• DiceMetric — Dice coefficient
• HausdorffDistanceMetric — 95th percentile or max distance
• SurfaceDistanceMetric — average surface distance
• IoU, Sensitivity, Specificity, Confusion matrix
• ConfusionMatrixMetric for all derived metrics (F1, PPV, NPV)

monai.apps

Ready end-to-end applications that download public datasets (MedNIST for demos, BRATS, MSD tasks) and provide reference training pipelines.

monai.engines

PyTorch Ignite integration for structured training loop: SupervisedTrainer, SupervisedEvaluator, handlers for logging, checkpointing, early stopping.

Design philosophy

MONAI does not aim to be a competitor of nnU-Net, 3D Slicer, MITK. It is conceived as common infrastructure on top of which various specialist projects can be built. The community invites:

• Research groups — to use MONAI for their pipelines, contributing useful modifications back to the community
• Developers — to contribute transforms, networks, loss functions specific to their domain (neuroimaging, histopathology, cardiology, ophthalmology)
• Commercial vendors — to integrate MONAI as infrastructural base for their products, with the Apache 2.0 licence enabling this
• University hospitals — to adopt MONAI to build in-house AI capabilities on internal clinical data

The stated vision is for MONAI to become for medical imaging what PyTorch is for deep learning in general: the standard platform everyone builds on, leaving room for specialist solutions in higher layers.

Relationship with other projects

Complementary ecosystem:

• nnU-Net (MIC-DKFZ) — not absorbed into MONAI; remains a separate self-configuring framework. Collaboration on shared components is expected
• 3D Slicer — user-facing application platform; a SlicerMONAI bridge is in development to import MONAI models into Slicer
• MITK — similar, with integration under discussion
• NVIDIA Clara — Clara Train SDK continues to exist as MONAI distribution optimised for NVIDIA stack (Clara Deploy, DGX, tensor cores), with components shared with open MONAI
• TorchIO — parallel library developed independently (Fernando Pérez-García, UCL), focused on 3D transforms and augmentation, with significant scope overlap. Cooperation/integration under discussion

Expected impact

Expectations for the coming years:

• De-facto standardisation of the PyTorch pipeline for medical imaging
• Reproducibility — papers using MONAI are more easily reproducible since transforms and training loops are known
• Cross-task transfer learning — models trained in MONAI more easily shareable
• Vendor entry in deploying MONAI-based pipelines in certified clinical products
• Education — MONAI will become a didactic base for medical imaging AI courses
• Expansion to non-imaging domains — extensions to tabular data, signal data (ECG/EEG), perhaps clinical NLP in future

Initial limits

MONAI in 2020 being a young project:

• Documentation growing — tutorials and examples multiply weekly
• API consolidating — 0.1 may not be fully API-stable; 1.0 is expected during 2021
• Gap with nnU-Net for segmentation — nnU-Net remains the out-of-the-box performance gold standard; MONAI requires more manual configuration
• Production deployment — MONAI is primarily research-focused; the “deploy” piece will be a future sub-project (emerging announcements for MONAI Deploy for 2022)

In the Italian context

Italian medical imaging AI groups have started following MONAI:

• Politecnico di Milano — integration in cardiovascular and mammography pipelines
• Universities of Turin, Verona, Bologna
• CNR / IFC Pisa — retinal imaging projects
• IRCCS institutes — experimental adoption in radiomics and segmentation projects

Operational adoption will take some months to stabilise; first Italian MONAI-based publications expected by end 2020-early 2021.

Outlook

MONAI’s success in coming years will depend on:

• Speed of technical governance consolidation
• Quantity and quality of external contributions — a health measure for an open source project
• Integration with commercial stacks — MONAI must be adopted in both research and products
• Clinical coverage — extension beyond segmentation (detection, classification, image-to-image, generative)
• Coordinated sub-projects — MONAI Label for annotation, MONAI Deploy for deployment, MONAI Stream for real-time ingestion — all announced but not yet released

The model is promising: a multi-institutional consortium, open licence, top-tier hardware vendor backing, leading clinical and academic community. MONAI in 2020 is in the initial phase of what could become the definitive medical imaging AI framework of the decade.

References: MONAI (Medical Open Network for AI), launched April 2020, monai.io. Repository: github.com/Project-MONAI/MONAI. Apache 2.0 licence. Founding consortium: NVIDIA, King’s College London, Mass General / Brigham and Women’s, Stanford University, Vanderbilt University, University of Warwick, Nanyang Technological University. Based on PyTorch.