The project
ALA, Advanced Laboratory Automation, is a joint research-and-development project between the INPECO Group (world leader in clinical-laboratory automation) and the BioRobotics Institute at the Scuola Superiore Sant’Anna in Pisa, with a total investment of EUR 3.5 million and a team of 30 researchers.
The work package noze worked on, WP3, had a stated goal: early detection of melanoma. Kick-off on 15 March 2019, closure planned for March 2022 and then moved to July 2022 because of the COVID stoppage. The scope followed the P5 framing of medicine (precision, predict, prevent, personalize, participate), which in practice means one thing: the system had to work outside the clinic, not only on a dataset.
Two parallel tracks came out of that, with opposite requirements.
The consumer app gives its user the probability that a lesion is malignant, with an artificial neural network engine classifying the image. The path starts with signing the privacy policy and goes through a two-sided mannequin, on which the user marks where the lesion sits before photographing it.
The mannequin anchors every capture to a body position. That is the condition which makes two acquisitions of the same lesion months apart comparable, and it is also how the patient builds a map of their own lesions over time.
The clinician’s instrument sits at the other extreme: a multimodal acquisition bench with a digital dermatoscope and a spectroscopy module, connected to a web app that collects the examination and closes it into a traced data packet.
The clinical network
The clinical data came from a network built piece by piece across Tuscany. The Azienda Ospedaliero-Universitaria Senese approved the protocol and enabled the retrospective data flow; the Azienda USL Toscana Nord Ovest enabled its own flow through the Accordo Quadro framework agreement, with centres in Siena, Livorno, Lucca, Pisa, Massa-Carrara and Pontedera.
On the methodological side the group published a systematic review with meta-analysis in Frontiers in Medicine on 21 April 2021, Comparative Analysis of Diagnostic Techniques for Melanoma Detection (Blundo, Cignoni, Banfi, Ciuti), comparing non-invasive techniques alternative to dermoscopy in clinical settings.
What makes classification hard
A skin lesion classifier is not answering a binary question. The reference map is the tree-structured taxonomy of skin disease used by Esteva and colleagues in Dermatologist-level classification of skin cancer with deep neural networks (Nature, 2017), the work that set the benchmark for everyone who came after. It is a taxonomy organised clinically and visually by medical experts: melanoma is one node among many, and benign and non-neoplastic forms cover most of what a camera runs into.
Redraw of the tree-structured taxonomy of skin disease from Esteva et al., 2017, fig. 2a, which in the original shows a subset of the top of the taxonomy (the paper carries a corrigendum in Nature 546, 686). The difficulty of the task lives in the branches next to melanoma: an atypical nevus, a seborrhoeic keratosis or a solar lentigo share with it signs that the image alone does not separate.
Two operational consequences followed, and they shaped the project. The first is that the dataset has to cover the confounding classes, not only the two of interest. The second concerns how the result is shown: with a prevalence of 1% and a classifier at 95% specificity and 99% sensitivity, 84% of the positives generated are false positives. This is the base rate error described in the literature (Gigerenzer, Psychol. Sci. Public Interest, 2008), and in a consumer app it becomes a product problem before it is a model problem: the same literature shows that presenting per-class probabilities raises the accuracy of human raters from 63.6% to 77.0% (Tschandl, Nature Medicine, 2020). That is why the output is a probability with an explicit referral to the dermatologist, not a verdict.
The result returned to the user: a malignancy probability, on a continuous scale with colour thresholds, rather than a hard benign-versus-malignant call.
The three research lines
- Software, deep-learning-based skin lesion detection: super-human performance benchmark, efficient DL architectures via optimisation techniques (AdaNet, AmoebaNet, NasNet), classification robustness through training regimens, data augmentation and generation
- Hardware, integrated multimodal sensing and DL inference device: system layout, sub-module design, infrastructure integration, with a dermatoscopic smartphone add-on and a data collecting bench carrying a digital dermatoscope and VIS-NIR spectroscopy
- Full traceability of biological samples and clinical data along the entire laboratory automation chain
The approach was incremental: starting from the implementation of a state-of-the-art (SoA) model, then going beyond it with proprietary techniques, following an iterative development paradigm, from prototypes to stable code bases.
A fourth line, opened later, pushed miniaturisation further: a thermal inspection module with a hyperspectral camera and a controlled thermal stimulus, and a compact VIS-NIR spectrometer built on a hybrid grated-Fresnel optical element.
The working group
- INPECO Group — client, world leader in pre-analytical and post-analytical laboratory automation
- BioRobotics Institute — Scuola Superiore Sant’Anna — Computer-Integrated Technologies for Robotic Surgery Laboratory, Surgical Robotics and Allied Technologies Area
- noze — AI and cloud software architecture, data pipelines, containerisation, web app and mobile app
- Dermatology Centres of Siena and Livorno — clinical validation, dermatological datasets
noze’s role
Stefano Noferi (noze) took part in the project from 2019 to 2023 as AI and cloud software architect, initially through the BioRobotics Institute at the Scuola Superiore Sant’Anna and later in direct collaboration with the INPECO Group, with responsibility for the design and implementation of the entire software infrastructure.
1. Cloud and on-premise architecture for the AI pipeline
Design and implementation of the cloud (Azure) and on-premise systems managing the whole artificial intelligence pipeline: data ingestion, pre-processing, training, inference and model distribution. The architecture, designed with a modular, microservices approach, made it possible to collect pseudonymised data from the various dermatology centres and to centralise their processing, segmentation and classification, with a clean separation between Frontend GUI, Backend (ANN Model Class) and Patient-data API.
One backend serves three clients with different requirements: the clinicians’ web app, the Raspberry Pi multi-sensor tool and the consumer app. Engine for the neural networks, Management for users, devices, patients and examinations, Storage for images, anamnestic data and reports.
The access layer uses multi-flow authentication and authorisation, with REST APIs over HTTPS/TLS, OAuth2 and JSON Web Tokens.
2. Data management and retraining
Implementation of data management flows for three distinct types:
- Anamnestic data — patient clinical information collected via the Azure backend, processed by the neural engine for anamnestic risk assessment
- RGB images — frames acquired via the smartphone camera API, classified by the neural engine for image classification with lesion segmentation and classification
- Point data and multimodal signals — coming from spectrophotometers and other sources, stored in the RAW data archive
Continuous retraining pipeline for the models with new clinical data validated by dermatologists.
Collection also had to survive a mundane and very frequent condition: the clinic with no network. The backend is therefore asynchronous with automatic online-offline I/O, optimised local caching and persistence on PostgreSQL with JSON fields, so that an examination started without connectivity closes and reconciles once the network returns.
3. Containerised infrastructure
Deployment through Docker containers with Redis orchestration (Redis Director + Redis Cache) for caching and routing inference requests towards the neural models. SoA trunk architecture for patient data persistence and retrieval.
On top of that layer sits a cloud-native Kubernetes cluster with hot-swappable ML models and automatic continuous integration, training and delivery flows, managed through a web console for administrators.
4. Web and mobile interfaces
Development of the web app for dermatologists (React) for viewing classifications, reviewing AI-assisted diagnoses and managing training datasets. Development of the mobile app for patients (React Native) for guided acquisition of lesion images and consultation of reports. Backend in Python/Django with REST APIs.
The clinicians’ web app. The dashboard holds together the acquired volume and the two queues that matter: images and examinations awaiting review, with the dermatologist’s assessment feeding back into the training dataset.
Work carried out in collaboration with the dermatology centres of Siena and Livorno for clinical validation of the results.
The models
The vision pipeline was built as three separate stages, each with its own model: detection of the region of interest within the dermoscopic image, segmentation of the lesion inside the ROI, classification with an ensemble method.
For the first stage two families were compared, a Single Shot Detector with a VGG-16 backbone and an EfficientDet, trained on ISIC 2018 (2,594 images with masks, from which bounding boxes were derived in Pascal VOC format). For classification the reference points were the EfficientNet family architectures and the ensembles that won SIIM-ISIC 2020, on a dataset extended beyond 46,000 training images drawn from the ISIC 2016-2020 releases.
Around the models sits the scaffolding that decides whether a training run finishes: a custom data augmentation generator, online and on CPU, with albumentations support and thread-safe parallel generation; a cosine annealing scheduler with restarts, adopted after linear decay left training stagnating in local minima; the replacement of batch normalisation with group normalisation and weight standardisation; and a siamese network built on convolutional backbones for comparing lesions. Training ran on a dedicated GPU cluster, with times in the order of days per model.
Morphological analysis was paired with a quantitative description of the lesion using 19 border and colour features, meant to support the 7-point checklist used by the dermatologist with computed measurements rather than visually estimated ones.
Technologies and approach
Deep Learning with CNNs for classification and segmentation of dermatological images. Azure as the cloud platform, Docker and Kubernetes for containerisation and orchestration, Redis for caching and routing, Python/Django and REST APIs for the backend, OAuth2 and JWT for authentication and authorisation, PostgreSQL with JSON fields for persistence, React and React Native for the interfaces. Python/TensorFlow stack for the neural models. Modular architecture inspired by Industry 4.0 principles with centralised management of pseudonymised data, privacy by design and end-to-end sample traceability.