Simulator Validation
A surgical simulator is only useful for training if it is proven to be realistic and to measure what it is meant to measure. Before our simulators are used as teaching tools, we therefore validate them objectively: technically, by comparing the artificial anatomy with human specimens (see Artificial Anatomy), and with surgeons of different experience in structured studies.
Types of validity
Face validity
Does the simulator look and feel realistic to its users?
Content validity
Do experts consider it appropriate for teaching the procedure?
Construct validity
Does its assessment distinguish between novices and experts?
Concurrent validity
Does the simulator’s score agree with an established assessment, such as a specialist’s rating?
Participants are grouped by experience, perform the procedure on the simulator, and rate it in standardised questionnaires on a 5-point Likert scale. Because our hybrid simulators record instrument paths, forces and timing, their performance scores can be computed automatically and compared with specialist ratings.
Validation studies
| Simulator | Participants | Validities shown |
|---|---|---|
| PatientSim (spine) | 43 | face, content, construct, concurrent |
| Cranial graft simulator | 17 | face, content, construct |
| MEDUSA (aneurysm clipping) | 40 | face, content |
Spine simulator for transpedicular access
The hybrid spine simulator PatientSim was validated at the Klinikum Wels-Grieskirchen with 22 novices, 12 intermediates and 9 experts, who performed a transpedicular access into an artificial L3 vertebra under simulated fluoroscopy. Most novices found the simulator attractive and realistic, and 92 % of the intermediates and 89 % of the experts would recommend it. The simulation-based performance score distinguished experts from novices (p = 0.001) and intermediates (p = 0.01), and it agreed strongly with the specialist rating (R = 0.86) [1].
Cranial graft lift simulator
The model-based simulator for lifting tabula externa grafts from the parietal skull was evaluated at Kepler University Hospital Linz with 8 novices and 7 experts in craniomaxillofacial surgery, assessed by 2 specialist raters. Realism was rated 4.2 and suitability as a teaching tool 4.8 out of 5. Completion times and graft sizes did not differ between the groups, but the raters graded the experts significantly better than the novices (46.9 vs. 36.4 points, p = 0.001), which confirms the construct validity [2].
Mixed-reality aneurysm clipping simulator
The MEDUSA simulator, which combines a physical skull with silicone brain lobes and virtual vessels with blood flow simulation, was externally validated with 40 neurosurgeons, from novices to experts with more than 15 years of experience, at two international events. In a 32-item questionnaire, mean ratings for realism and educational value ranged from 3.13 to 4.25; the blood flow simulation (4.25) and the use for preoperative planning (4.20) were rated highest (Figure 1) [3].
Figure 1 from [3], licensed under CC BY 4.0.
References
- (2018): Transpedicular Approach on a Novel Spine Simulator: A Validation Study. In: Journal of Surgical Education, vol. 75, no. 4, pp. 1127–1134, 2018, ISSN: 1931-7204, (Publisher Copyright: © 2018 Association of Program Directors in Surgery).
- (2018): Validation of a Simulator for Cranial Graft Lift Training: Face, Content, and Construct Validity. In: Journal of Cranio-Maxillo-Facial Surgery, vol. 46, no. 8, pp. 1390–1394, 2018.
- (2025): Development and external validation of a mixed-reality aneurysm clipping simulator. In: Neurosurgical Review, vol. 48, no. 1, 2025, ISSN: 0344-5607, (Publisher Copyright: © The Author(s) 2025.).