Caffe and the arrival of deep learning in medical imaging

AIHealth

On-premise clinical platform with local LLMs, RAG on FHIR/DICOM data, diagnostic support, remote follow-up. Architecture designed for the MDR pathway.

Discover AIHealth →

Digital Health

Medical software development compliant with CE and MDR regulatory standards. Clinical decision support systems, AI integration in clinical workflows.

Discover →

2012, the year that changes everything

September 2012 was a breaking point for computer vision. At that year’s ImageNet Large Scale Visual Recognition Challenge, a convolutional neural network called AlexNet — developed by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton at the University of Toronto — reduced the top-5 classification error from 25.8% (best 2011 result with traditional methods) to 16.4%. For a 1000-class, 1.2 million image dataset, such a leap was unprecedented.

The mechanism was known — convolutional networks (LeCun et al. 1989, LeCun LeNet 1998) — but the execution was new: GPUs (two NVIDIA GTX 580) for training, ReLU as activation, dropout as regularisation, data augmentation for generalisation. In two years from 2012, CNNs became state of the art for almost every vision task.

Medical research watched quickly. The ability to classify diagnostic images with models trained initially on ImageNet, then fine-tuned on smaller medical datasets, promised to transform the computer-aided diagnosis landscape. But uptake needed accessible software frameworks — and that is where Caffe entered the stage.

Caffe

Caffe — Convolutional Architecture for Fast Feature Embedding — was launched in September 2013 by Yangqing Jia, then a PhD student at UC Berkeley, as part of the Berkeley Vision and Learning Center (BVLC) under Trevor Darrell’s supervision. Initial release was public by late 2013; by end of 2014 Caffe is at version ~1.0 rc with an active international community.

Technical characteristics:

• C++ implementation with CUDA kernels for execution on NVIDIA GPUs
• Python bindings (pycaffe) and MATLAB bindings for interactive use
• Declarative model definition in prototxt files (Protocol Buffers) — architecture, layers, loss, optimiser
• Serialised weights in caffemodel format (binary Protocol Buffers)
• Training via solver (SGD, Nesterov, AdaGrad) with declarative configuration
• BSD 2-Clause licence — very permissive, allowing commercial use and inclusion in certified products

The reference publication — Caffe: Convolutional Architecture for Fast Feature Embedding by Yangqing Jia et al. — appeared in mid-2014 on ArXiv and was accepted at ACM Multimedia 2014.

Model Zoo

The BVLC Model Zoo — repository of pre-trained models distributed with Caffe — is one of the project’s most relevant contributions. As of 2014 the Zoo includes:

• AlexNet (Krizhevsky et al. 2012) trained on ImageNet (1000 classes, ~1.2M images)
• CaffeNet — slightly modified AlexNet variant, reference training
• GoogLeNet (Szegedy et al. 2014) — Inception architecture, ILSVRC 2014 winner
• VGG-16 and VGG-19 (Simonyan & Zisserman 2014) — deeper networks, excellent as feature extractors
• R-CNN for detection (Girshick et al. 2014)
• Models for style, face recognition, segmentation

Weights are distributed free, under BSD licence, downloadable with a simple wget. The Model Zoo becomes standard: anyone doing vision research starts from shared pre-trained models instead of retraining from scratch.

Transfer learning in medicine

The operational pattern for Caffe’s medical applications in 2014 is transfer learning. Typical flow:

1. Start from a model pre-trained on ImageNet (AlexNet, GoogLeNet, VGG)
2. Replace the final classifier (the last fully connected + softmax layer) with one adapted to the medical task (typically 2-5 classes instead of 1000)
3. Fine-tuning: retrain the model (usually with low learning rate for pre-existing convolutional layers and higher learning rate for new layers) on the target medical dataset
4. Validation: stratified cross-validation with appropriate clinical metrics (sensitivity/specificity, AUC ROC, Precision-Recall curves)

The insight that makes transfer learning effective: low convolutional layers learn general visual features (edge detectors, texture detectors, colour detectors) reusable cross-domain. Only the higher layers — more task-specific — need significant retraining.

The advantage is huge: a typical medical dataset has 10^3-10^4 images, far less than ImageNet’s 10^6; training a deep CNN from scratch on a few thousand examples is practically impossible (overfitting). Transfer learning enables obtaining performant medical classifiers with modest datasets.

Early medical cases

In 2013-2014 the first medical publications appear that use CNNs with Caffe or analogous frameworks:

• Histopathology: Cireşan et al. (2013), “Mitosis detection in breast cancer histology images with deep neural networks”, MICCAI 2013. One of the earliest demonstrations of deep learning on histology
• Lymph node detection: Roth et al. (2014), on CT datasets, with Caffe
• Diabetic retinopathy: early experiments emerging; the most influential work (Gulshan et al. JAMA) will arrive in 2016
• MICCAI Grand Challenges 2014: tumour segmentation, histology, see CNN competitive entries
• Dermatology exploratory
• Mammography with CNN as feature extractor
• Chest X-ray: early experiments towards what will become the most active development field in the following years (Stanford CheXNet 2017)

A recurring pattern of these publications:

• Public dataset of a few hundred to a few thousand images
• Architecture: AlexNet or variant with fine-tuning
• Training: single-card GPU, a few hours to days
• Results: often on par or superior to classical methods with manual feature engineering

The role of GPUs

The key enabler of medical deep learning is consumer GPU accessibility. As of 2014:

• NVIDIA GTX Titan (6GB memory) — ~USD 1000
• NVIDIA GeForce GTX 780 Ti — ~USD 700
• NVIDIA Tesla K40 — ~USD 5000, professional workstation

A university research group can afford a 1-4 GPU workstation for under EUR 10,000, sufficient for research on available medical datasets. Caffe exploits single-card GPUs well; multi-GPU parallelisation is emerging.

The arrival of cuDNN (NVIDIA cuDNN version 1, 2014) as an accelerated library for base CNN operations, integrated into Caffe, brings further performance improvements (2-3x training speedup vs. Caffe native implementations).

2014 ecosystem

Caffe is not the only framework available in 2014, but is the most widespread in vision:

• Theano (MILA, Montreal) — more flexible, research-oriented, less optimised for production CNN
• Torch7 (NYU, Yann LeCun group) — Lua, used by Facebook AI Research
• cuda-convnet (Krizhevsky) — original AlexNet implementation, not a general framework
• Neon (Nervana) — startup, efficiency-focused

Caffe stands out for: speed, rich model zoo, ease of use for those with image datasets and classification tasks.

The upcoming arrival of TensorFlow (Google, expected November 2015) and PyTorch (Facebook, 2016-2017) will reshape the landscape again, but in 2014 Caffe is the reference tool for medical deep learning projects.

Limits

Caffe 2014 has specific limits:

• Static model definition — networks are declared in prototxt; no dynamic graph (recurrent networks and variable-structure models are hard)
• Limited Python API — good for inference, less complete for advanced training
• Primitive recurrent networks (RNN) — Caffe is CNN-born; RNNs are a later, less integrated addition
• CUDA only — optimal performance requires NVIDIA GPUs; emerging but less robust OpenCL backend
• Scalability — multi-node distributed training is limited

For a 2014 medical project with static 2D or 3D imaging, these limits are manageable. For temporal-sequence tasks (ECG, volume time series, video) Caffe is less suitable.

Prospective applications in medicine

Expected development directions in medicine:

• Automated mammography screening — mammogram triage to highlight suspicious ROIs
• Computer-aided retinal diagnosis — diabetic retinopathy, macular degeneration
• Digital pathology — whole slide image analysis, cell counting, tumour scoring
• Chest X-ray — detection of common pathologies, triage
• CT/MR volume segmentation — tumours, organs, critical structures for radiotherapy
• Dermatology classification — pigmented lesion screening

The topic of medical device regulatory certification is starting to emerge. FDA does not yet have CNN-specific guidance (it will come later), but the first 510(k) submissions with deep learning components are in exploratory phase. In Europe, the new medical device Regulation under discussion (proposal 2012, ongoing updates) will interact with clinical AI development.

In the Italian context

As of 2014 some Italian groups are exploring Caffe for medical applications:

• Politecnico di Milano — cardiovascular imaging, mammography
• University of Turin — oncology applications
• CNR / IFC Pisa — retinal imaging
• IRCCS institutes — early collaborations with medical informatics groups

Availability of annotated Italian clinical datasets is the main limiting factor; use of international datasets (TCIA, DDSM, MIT-BIH) enables methodological research but validation on the Italian population requires local collection and annotation.

Outlook

Deep learning in medicine at the end of 2014 is in the “early adopter” phase: some pioneering groups are publishing first results; the vast majority of clinical practice continues with traditional methods. The coming years will see:

• Community growth and multiplication of publications
• Larger shared medical datasets — initiatives like Kaggle medical competitions, MICCAI challenges, datasets released by major hospitals
• Public benchmarks for method comparison
• Integration with clinical platforms — embedding of CNN models in PACS, RIS, diagnostic systems
• Framework evolution — Caffe 2, TensorFlow, PyTorch in the coming years
• Regulatory attention — FDA, EU MDR, medical device AI certification

The adoption curve of deep learning in medicine deserves systematic monitoring. As of 2014, it is in a phase of more accurate automatic pattern recognition than before — but integration into routine clinical practice is still an open trajectory.

References: Caffe (caffe.berkeleyvision.org), Berkeley Vision and Learning Center (BVLC). Yangqing Jia et al., “Caffe: Convolutional Architecture for Fast Feature Embedding”, ACM Multimedia 2014. BSD 2-Clause licence. AlexNet (Krizhevsky et al. 2012). GoogLeNet (Szegedy et al. 2014). VGG (Simonyan & Zisserman 2014). cuDNN (NVIDIA 2014). Cireşan et al., “Mitosis detection in breast cancer histology images with deep neural networks”, MICCAI 2013.