If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Christie Medical Physics and Engineering, The Christie NHS Foundation Trust, Wilmslow Road, Manchester M20 4BX, UKDivision of Cancer Sciences, Faculty of Biology, Medicine and Heath, University of Manchester, Manchester Academic Health Science Centre, The Christie NHS Foundation Trust, Wilmslow Road, Manchester M20 4BX, UK
Deep learning contouring (DLC) has the potential to decrease contouring time and variability of organ contours. This work evaluates the effectiveness of DLC for prostate and head and neck across four radiotherapy centres using a commercial system.
Materials and methods
Computed tomography scans of 123 prostate and 310 head and neck patients were evaluated. Besides one head and neck model, generic DLC models were used. Contouring time using centres’ existing clinical methods and contour editing time after DLC were compared. Timing was evaluated using paired and non-paired studies. Commercial software or in-house scripts assessed dice similarity coefficient (DSC) and distance to agreement (DTA). One centre assessed head and neck inter-observer variability.
Results
The mean contouring time saved for prostate structures using DLC compared to the existing clinical method was 5.9 ± 3.5 min. The best agreement was shown for the femoral heads (median DSC 0.92 ± 0.03, median DTA 1.5 ± 0.3 mm) and the worst for the rectum (median DSC 0.68 ± 0.04, median DTA 4.6 ± 0.6 mm). The mean contouring time saved for head and neck structures using DLC was 16.2 ± 8.6 min. For one centre there was no DLC time-saving compared to an atlas-based method. DLC contours reduced inter-observer variability compared to manual contours for the brainstem, left parotid gland and left submandibular gland.
Conclusions
Generic prostate and head and neck DLC models can provide time-savings which can be assessed with paired or non-paired studies to integrate with clinical workload. Reducing inter-observer variability potential has been shown.
It is important to have accurate organ at risk (OAR) contours for radiotherapy planning to ensure healthy tissue is spared. Techniques such as volumetric modulated radiotherapy or intensity modulated radiotherapy allow highly conformal dose distributions with steep dose gradients to be created so it is imperative that the contours are accurate. The accuracy of OAR contouring has also been shown to correlate with toxicity [
Interobserver variation in clinical target volume and organs at risk segmentation in post-parotidectomy radiotherapy: can segmentation protocols help?.
Variations in inter-observer contouring and its impact on dosimetric and radiobiological parameters for intensity-modulated radiotherapy planning in treatment of localised prostate cancer.
An alternative method is to use atlas-based auto-contouring which can reduce contouring time and improve consistency for sites such as head and neck, prostate, and lung [
Systematic evaluation of three different commercial software solutions for automatic segmentation for adaptive therapy in head-and-neck, prostate and pleural cancer.
Clinical validation of atlas-based auto-segmentation of multiple target volumes and normal tissue (swallowing/mastication) structures in the head and neck.
], as for small structures a minor error in deformable registration would make a bigger difference than for larger structures.
Recent advances in artificial intelligence have created further techniques in the form of deep learning. These use neural networks trained on large datasets of contoured images to improve contouring accuracy and once trained are quicker to use than atlas-based methods [
]. Although also limited by numbers of patients, deep learning is based on larger datasets than atlas-based methods, with larger datasets improving model accuracy without reducing the speed. Atlas-based models however, will reduce in speed when more datasets are added. Studies have shown that deep learning outperforms manual and atlas-based contouring [
] assessed DLC for 83 prostate and 26 head and neck CT scans from three centres but only ten scans were assessed for time-saving and individual images were assessed rather than clinical evaluation at each centre. Wong et al. [
] investigated DLC head and neck models on 217 and 58 patient CTs respectively but only over two sites.
This study aimed to evaluate a commercial DLC system for prostate and head and neck across four radiotherapy centres, including time-saving evaluation.
2. Materials and methods
2.1 Patient selection
The prostate study included 123 patient CTs from three radiotherapy centres. The head and neck study included 310 patient CTs from four centres. Each image came from a different patient and this was a retrospective study. Local approval was granted for this work and written informed consent was obtained from all patients.
Centres 2 (both sites), 3 (head and neck) and 4 (head and neck) selected consecutive patients. The time to outline OARs using the existing clinical method was recorded. A different set of clinical patients were outlined using DLC and the editing time recorded. Patients from centres 1 (both sites) and 3 (prostate) were consecutive, with times to outline OARs using the existing clinical method and to edit after using DLC recorded for the same patients. Centre 4 timings were collected using an SQL query of the ARIA database (Varian Medical Systems, Palo Alto, USA) for all patients that had a head and neck planning task competed within two weeks of an import/OAR outlining task being completed. All other centres recorded timings manually.
Where a paired method was used, every effort was made for the same observer to do both sets of contouring separated by a significant gap (4–6 weeks). For the unpaired methods, different patients were compared and were potentially contoured by different observers.
Each centre used its own local protocols for scanning and outlining. Full details are given in Table 1, Table 2.
Table 1Methods of assessing auto-contouring using deep learning contouring (DLC) for prostate structures.
Centre 1
Centre 2
Centre 3
Number of patients for timing
9
42 manual, 42 DLC
10
Number of patients for quantitative analysis
9
10
N/A
Same patients for timing and quantitative analysis?
Y
N
N/A
Timing Study design
Paired
Unpaired
Paired
Number of staff
2 (both outlined 9 each)
4
10 for manual, 1 for DLCExpert
Staff group
Radiotherapy Planners
Radiotherapy Planners
Radiotherapy Planners and clinicians. One clinician edited DLCExpert contours
Existing method
Manual
Manual
Manual
CT scanner/slice thickness
Philips Brilliance Big Bore (3 mm)
GE Discovery (2.5 mm)
GE Discovery (2.5 mm)
DLCExpert model name
Prostate_CT_NL006_MO
Prostate_CT_NL005_GN (bladder and rectum)Prostate_CT_NL010_NN (femoral heads)
Prostate_CT_NL006_MO
DLCExpert model description
Generic model based on data from a centre in the Netherlands.
Generic models based on data from a centre in the Netherlands.
Generic model based on data from a centre in the Netherlands.
Number of images for training model
437
242 (bladder and rectum)337 (femoral heads)
437
OARs
Bladder, femoral heads, rectum
Bladder, femoral heads, rectum
Bladder, femoral heads, rectum
Targets
None
None
Prostate, seminal vesicles
Editing software
Pinnacle v16 (Philips, NL)
RayStation v7 (RaySearch, Sweden)
Eclipse v13.7 (Varian Medical Systems, Palo Alto, USA)
Outlining protocol basis
CHHiP
CHHiP, RTOG atlases
CHHiP
Timing method
Manual- time to draw or edit timed by the staff member
Manual- time to draw or edit timed by the staff member
Manual- time to draw or edit timed by the staff member Aria (Varian Medical Systems, Palo Alto, USA)-time to draw CTVs
A commercial DLC system (DLCExpertTM, Mirada Medical ltd, UK) was used for this study. Each centre was given the freedom to implement DLCExpert as was applicable to their workload and patients. DLCExpert comprises two stages of multi-layer central neural networks (CNNs) to classify OARs on a CT scan. The first stage coarsely segments the volume into OARs. This is implemented via a 14-layer multiclass CNN. The output of the first CNN is passed into multiple specialised 10-layer CNNs, which each perform binary classification for different organs. After correction of any discontinuities, the result is a prediction of the full resolution contours [
Models used in this study were generic (excluding centre 1 head and neck model), meaning they had not been trained on data from any of the centres in this study. The generic prostate model was trained on CT scans contoured using outlining guidelines from a clinic in the Netherlands and the generic head and neck model was trained on CT scans contoured using guidelines from Brouwer et al. [
CT-based delineation of organs at risk in the head and neck region: DAHANCA, EORTC, GORTEC, HKNPCSG, NCIC CTG, NCRI, NRG Oncology and TROG consensus guidelines.
]. All centres were using Workflow Box (WBx) 2.0 (Mirada Medical ltd, UK) from September 2019 to June 2020 and WBx 2.2 (Mirada Medical ltd, UK) from June 2020 to November 2020. WBx is a DICOM based tool which enables images to be routed directly from the scanner where they are automatically outlined using DLCExpert and then routed to the planning system.
2.3 Contouring time
Each centre recorded the time to outline OARs using their existing clinical protocol. OARs were also contoured using DLC and the time taken to edit these was recorded. The time taken by DLC to contour was not included. This was automated when the image was sent from the CT scanner to the treatment planning system and hence was not contributing to contouring time. The methods for each centre are summarised in Table 1, Table 2.
Some centres implemented a paired study, where the same set of patients were timed using DLC and the centre’s existing clinical method. This data was assessed for statistical significance using a paired T-test. Other centres implemented an unpaired study, where a different set of patients were analsyed using DLC and the centre’s existing clinical method. This data was assessed for statistical significance using an unpaired T-test. Statistical analysis was performed in SPSS (V25, IBM).
2.4 Quantitative evaluation
Dice Similarity Coefficient (DSC) and distance to agreement (DTA) were calculated to compare the existing clinical method to contours created with DLC before editing. DTA was taken as the shortest distance from a point on one contour surface to the other contour surface. Mean DTA was used which was the mean of all the distances calculated. Each centre decided upon the calculation method of DSC and DTA. This included ADMIRE software (v3.21.2, Elekta AB, Sweden) and in-house scripts within planning systems.
2.5 Subjective scoring
Subjective scoring was not part of the original study but was recorded by some centres when performing their own clinical evaluations. It has been included here for completeness. For prostate DLC, centre 3 subjectively scored all patients and centre 2 subjectively scored the last ten sequential patients. For head and neck DLC, centre 1 subjectively scored all patients and centres 2 and 3 subjectively scored the last ten sequential patients. No scoring was recorded for centre 4.
Centres 1 and 2 scored 1–7 using Greenham et al. [
] based on agreement to an acceptable clinical contour where 1 = good agreement through to 7 = gross error.
Centre 3 scored 1–5 where 1 = clinically acceptable through to 5 = would be easier to start from scratch.
2.6 Inter-Observer variability
Centre 1 assessed inter-observer variability for head and neck outlining. Each clinician’s manual and DLC edited contours were compared with all of the remaining clinicians giving twelve comparisons in all. This was repeated for five patients, for brainstem, larynx, left parotid and left submandibular gland (SMG). Box plots of the mean DTA and DSC were compared for manual and DLC edited contours.
3. Results
All three centres found a time-saving for prostate patient contouring using DLC compared to the existing clinical method. The total mean time saved per patient for contouring of all prostate structures assessed across the three centres was 5.9 ± 3.5 min (23 % time-saving on average). Fig. 1 shows the distribution of results for all centres. The paired T-tests and unpaired T-test indicated that the time differences between the manual and DLC methods were only significant at the 5 % level for centre 3 (supplementary table S1).
Fig. 1Box and whisker plot of contouring times, dice similarity coefficient (DSC) and distance to agreement (DTA) for prostate structures from 3 centres using manual and deep learning contouring (DLC). The boxes indicate the interquartile range (IQR), the line indicates the median and the cross indicates the mean. The whiskers indicate the highest and lowest values within 1.5 times the IQR and data outside this range indicated by circles. Axes for DTA and DSC have been limited to exclude one centre 2 patient left femoral head result as the DLC contour was outside of the external volume.
At centres 1 and 2, the DTA and DSC comparing manual contours to DLC contours showed the best agreement for the femoral heads (median DSC 0.92 ± 0.03, median DTA 1.5 ± 0.3 mm) and the worst agreement for the rectum (median DSC 0.68 ± 0.04, median DTA 4.6 ± 0.6 mm) (Fig. 1 middle and bottom panels and supplementary tables S2 and S3). This reflected the subjective scores (supplementary table S4).
The total mean time saved for head and neck OAR contouring using DLC compared to the existing clinical method across the four centres was 16.2 ± 8.6 min (7 % time-saving on average) (Fig. 2). Where the existing clinical method was manual, (excluding centre 2) the average time-saving was 22.5 ± 8.4 min (27 %). The p-values indicated that the difference between the existing and DLC contouring methods were significant at the 5 % level for centres 1 and 2 (supplementary table S5). The distribution of all results showed a time saving for centre 4 (12.2 ± 8.2 min) and a time increase for centre 2 of 6.1 ± 1.3 min and smaller interquartile ranges for all centres using DLC (Fig. 2).
Fig. 2Box and whisker plot of contouring times for head and neck structures from 4 centres using manual and deep learning contouring (DLC). The boxes indicate the interquartile range (IQR), the line indicates the median and the cross indicates the mean. The whiskers indicate the highest and lowest values within 1.5 times the IQR and data outside this range indicated by circles.
The mandible, brainstem and right parotid scored highly for DSC and DTA across all centres where analysed (Fig. 3 and supplementary tables S6 and S7). DSC and DTA values were calculated prior to editing the DLC contour.
Fig. 3Box and whisker plot of dice similarity coefficient (DSC) and distance to agreement (DTA) for head and neck organs at risk from 4 centres using an existing clinical method and deep learning contouring (DLC). The boxes indicate the interquartile range (IQR), the line indicates the median and the cross indicates the mean. The whiskers indicate the highest and lowest values within 1.5 times the IQR and data outside this range indicated by circles. Axis has been limited to exclude one patient spinal cord result for centre 1.
The inter-observer variability assessment showed that DLC generated contours reduced inter-observer variability compared to manual contours for the brainstem, left parotid gland and left SMG, with lower median DTA and higher DSC for the DLC contours (Fig. 4 and supplementary tables S9 and S10). There were also smaller interquartile ranges for the majority of the contours using DLC (Fig. 4). The difference between the two methods was statistically significant at the 5 % level using a Wilcoxon signed rank test (supplementary tables S9 and S10). No statistical difference in observer variability between DLC and manual contours was observed for the larynx DTA.
Fig. 4Distance to agreement (DTA) for manual and deep learning contouring (DLC) edited head and neck organs at risk. Results are from centre 1 for different permutations of 4 observers compared to each other for 5 patients (pt).
This work assessed the use of DLC across four centres within the NHS where the contouring method would ideally be standardised. Prostate contouring showed a time-saving using DLC for all centres although this was only statistically significant for centre 3. This may be due to a larger time-saving for the prostate and seminal vesicles which were only analysed by centre 3. Kiljunen et al. [
] found a larger time-saving of twelve minutes (46 %) with a range of 1.5 % to 70.9 %. Their study included the prostate, seminal-vesicles and penile bulb in addition to the OARs analysed in this study which may increase the time-saving. Zabel et al. [
] found a time-saving of 8.5 min (44 %) using DLC for outlining bladder and rectum on 15 prostate CT scans. However, the manual times included contouring by a radiation therapist and editing by a radiation oncologist. The times in the current study from centres 1 and 2 include manual contouring by one person, hence manual contouring for these sites will be faster.
Time-savings using DLC for head and neck were seen for centres using manual contouring as their existing clinical method. Oktay et al. [
] found a time-saving of 93 % using DLC compared to manual contouring for head and neck CT scans. A maximum time-saving of 45 % was found in this current study. However, Oktay et al. timed an average manual contouring of 87 min for experts which was longer than the average time of any centres in this current study. Also, time-saving will likely increase with the number of structures contoured, with sites who outline more OARs gaining more time. Oktay et al. outlined nine structures using DLC whereas three out of the four centres in the current study outlined less than this. Hence it would be expected that the time-saving in the current study would be less.
The manual timings in this study were significantly lower than in other studies, meaning such a high percentage time-saving was not achievable. For example, Kiljunen et al. [
] reported prostate OAR manual contouring times of approximately 20 min for most centres with two centres taking over 40 min. In contrast, the longest manual prostate contouring time in the current study was 15 min, possibly because less structures were outlined. This current work has also shown that implementation significantly affects the times quoted. Centre 4 reported longer head and neck manual contouring times compared to the other centres and this was due to timing using ARIA. Care should be taken when interpreting and comparing results from studies analysing auto-contouring as there may be differences such as contour numbers, current outlining method, timing methods, staff expertise and knowledge that a measurement was being performed. Although larger time-savings imply the initial DLC quality was better, with variation across centres a conclusion cannot be drawn regarding this for the current study.
Time-saving using DLC for head and neck was statistically significant for centre 1 that had a combined model using structures selected from three models, two trained on their own data and one generic model. This suggests that DLC models developed with local data may provide a larger time-saving benefit. In addition, centre 1 tested more structures than the other centres which may contribute to the larger time-saving. Centre 2 was the only centre to observe a time increase using DLC, possibly because the centre uses a well-established atlas-based method and timing was only recorded for a limited number of structures. Staff also had significant prior experience adjusting atlas contours and did not have experience of DLC.
The DSCs obtained show the generic model performed well for the bladder and femoral heads (Fig. 1 and supplementary table S11). The DSC for rectum and bladder were slightly lower than other studies where generic models were used but the femoral heads were better than or comparable to other studies [
]. The largest variation between centres was for the rectum, potentially due to the small sample size or differing local contouring protocols. This again shows that care should be taken when interpreting results as the same DLC model can produce different results at different centres.
The majority of DSCs obtained for head and neck OARs were comparable to other studies. The DSC for the mandible in literature was between 0.89 and 0.94 and the left parotid between 0.77 and 0.84 [
] was tested on data from the same centre so was not a generic model.
Each centre chose their own method of contouring timing, resulting in paired and non-paired studies with different patient numbers. A sample size calculation performed showed 51 patients required in each group for an unpaired study and 26 patients if it was paired [
]. However, an unpaired study requires no additional clinical work through repetition of measurements. The calculation suggests a larger sample size is required in the current study but this was not possible due to the clinical workload.
The results from centre 1 suggest that inter-observer variability using DLC was improved, agreeing with other studies [
]. Kiljunen et al. found improved inter-observer variability for prostate OARs, with more benefit for large, challenging structures such as lymph nodes. Even in centres with existing efficient contouring methods, DLC could provide the benefit of improved consistency. Further assessment is needed across a larger multi-centre dataset which has currently not been carried out within the literature. However, this evaluation is very resource intensive. The inter-observer DSCs for manual contouring (supplementary table S9) were comparable to the DLC vs manual contouring DSCs for head and neck OARs (supplementary table S6). This suggests that a clinician is likely to agree as much with DLC as they do with another clinician.
Different software was used to calculate the DSC and DTA at each centre so the values cannot be directly compared. This again highlights the need for caution when comparing data in literature as the calculation of these is not standardised. However, they do give an indication of which contours were closer to clinical ones. It has been shown that a DSC larger than 0.65 should give a time-saving [
] as the time to edit the auto-contours would probably be quicker than manual delineation. Although no individual contour times were calculated in the current study, the majority of contours had a DSC greater than 0.65. However, it has recently been shown for lungs [
] have produced recommendations for implementation and quality assurance of artificial intelligence applications in radiotherapy. In addition to the overlap and distance metrics used here, they recommend to compare the overall volume and the dosimetric impact of the delineation uncertainty. The impact of a DLC contour that is minimally different to a clinical contour may not have a clinically significant impact on the evaluated dose. van Rooij et al. [
] showed that differences in DLC contours for head and neck patients did not produce clinically significant differences in radiotherapy plans.
In conclusion, clinical implementation of non-centre specific DLC models for prostate and head and neck OARs can provide time-savings. Time-savings become more significant when a larger number of OARs are contoured. A generic model can be implemented and tested in different ways, using paired or unpaired studies to fit in with clinical workload. Unpaired studies take less time as they prevent contour replication. DSC and DTA showed good agreement with clinical contours for the majority of structures. The potential for reducing inter-observer variability has been indicated but further work is needed to confirm this.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
We are grateful to all staff members who performed outlining and data collection for the study including Liz Adams and Dan Love. We would like to thank Mirada for their technical support. Robert Chuter is supported by a Cancer Research UK Centres Network Accelerator Award Grant (A21993) to the ART-NET consortium.
Appendix A. Supplementary data
The following are the Supplementary data to this article:
Interobserver variation in clinical target volume and organs at risk segmentation in post-parotidectomy radiotherapy: can segmentation protocols help?.
Variations in inter-observer contouring and its impact on dosimetric and radiobiological parameters for intensity-modulated radiotherapy planning in treatment of localised prostate cancer.
Systematic evaluation of three different commercial software solutions for automatic segmentation for adaptive therapy in head-and-neck, prostate and pleural cancer.
Clinical validation of atlas-based auto-segmentation of multiple target volumes and normal tissue (swallowing/mastication) structures in the head and neck.
CT-based delineation of organs at risk in the head and neck region: DAHANCA, EORTC, GORTEC, HKNPCSG, NCIC CTG, NCRI, NRG Oncology and TROG consensus guidelines.
00093-8&id=gr1.jpg){kind=link}
00093-8&id=gr2.jpg){kind=link}
00093-8&id=gr3.jpg){kind=link}
00093-8&id=gr4.jpg){kind=link}