HyTra
HyTra (Hybrid Tracking in Soft Tissue for Surgical Simulators) develops a hybrid tracking system that combines a conventional global tracking system with soft sensors embedded in the artificial tissue. The aim is to follow surgical instruments and deforming tissue in a hybrid simulator far more precisely than either method can on its own.
Motivation
In hybrid simulators for minimally invasive surgery, real instruments and a physical patient phantom must be synchronised exactly with their virtual 3D counterparts. Electromagnetic or optical tracking systems are well suited to follow instruments in space, but they are not accurate enough for very small, moving and deformable target structures such as the heart of an unborn child. Classic tracking systems only see the instrument, not the reaction of the tissue: once a flexible needle is inside the body, the position of its tip can no longer be estimated reliably from outside, and the tissue itself shifts and deforms. Conventional rigid sensors cannot simply be added either, because they would change the behaviour of the soft tissue they are meant to measure. In delicate procedures such as fetal interventions, however, millimetres in easily deformable tissue matter, and so far simulators cannot measure them in real time. The virtual model then drifts away from what really happens inside the phantom, and trainees may practise movements that would be wrong in a real operation.
Hybrid tracking
HyTra fuses two complementary sources of information (Figure 1):
Global tracking
A commercial electromagnetic tracking system with sensor coils in the instrument, the ultrasound probe and the phantom provides a stable absolute position reference.
Local soft sensors
Soft, tissue-mimicking sensors made of carbon black silicone composite, embedded in the target area, measure where and how strongly the tissue is touched, deformed or damaged.
The soft sensors build on our Smart Artificial Tissue research [1], [2]. Contact positions are derived from measurable potential shifts in the conductive layer, and tissue deformations are reconstructed from conductivity distributions with inverse finite element methods. Physics-informed neural networks and deep surrogate models make this reconstruction fast enough for real time, even with limited training data. Only the fusion of both data sources gives a complete picture of the intervention: the global system’s spatial stability together with the fine local detail of the soft sensors.
Approach
The project starts with a simple binary soft sensor to confirm the feasibility of the concept and then develops it iteratively into a spatially resolving sensor, read out in real time by a microcontroller. In parallel, software is developed that merges the relative data of the soft sensors with the absolute positions of the global tracking system.
A physical demonstrator with an abstracted patient phantom then integrates the soft sensors and the tracking system. It is evaluated under realistic conditions and compared with existing tracking solutions to show the gain in accuracy objectively, both for instrument guidance and for tissue deformation. Its measurements are compared with known reference positions and defined deformations, using four criteria:
Accuracy
How closely measured positions match the reference.
Repeatability
How consistent the results are over repeated measurements.
Response time
How quickly the system reacts, as required for real-time feedback.
Robustness
How well the soft sensors withstand puncturing and repeated loading.
Benefits for training
With more precise tracking, trainees receive objective, immediate feedback on how accurately and gently they work. Rare and risky procedures can be practised realistically and repeatedly without endangering patients. Because the combined system shows both the instrument’s movement and the tissue’s response, cause and effect become traceable, which is essential for learning.
Expected results
- A proof of concept and a demonstrator of the hybrid tracking system.
- A module that speeds up the development of future hybrid surgical simulators and improves their accuracy.
- Findings on the integration of position and deformation data from soft sensors and on the accuracy that can actually be achieved.
Outlook
A follow-up project is planned to develop a complete hybrid simulator for fetal interventions, building on the Fetal Heart Phantom. The technology can also be transferred to other scenarios such as neurosurgery or paediatric cardiology.
Partners and funding
HyTra is led by AMPIONICS GmbH (Thomas Thurner), a spin-off of the Research Group for Surgical Simulators Linz, together with ReSSL (Andreas Schrempf) and RISC Software GmbH (Benjamin Esterer, Philipp Moser). ReSSL supports the design of the hybrid tracking system, the construction of the demonstrator and its validation; RISC Software develops the AI methods for position and deformation detection. Physicians of the Kepler University Hospital Linz advise on clinical scenarios. The project runs for twelve months from March 2026 and is funded by the Austrian Research Promotion Agency (FFG) in the programme “Expedition Zukunft Start”.
References
- (2023): Smart Artificial Soft Tissue - Application to a Hybrid Simulator for Training of Laryngeal Pacemaker Implantation. In: IEEE Transactions on Biomedical Engineering, vol. 70, no. 2, pp. 735–746, 2023, ISSN: 0018-9294, (Publisher Copyright: © 1964-2012 IEEE.).
- (2024): Dynamic Tactile Synthetic Tissue: from Soft Robotics to Hybrid Surgical Simulators. In: Advanced Intelligent Systems, vol. 6, no. 12, 2024, ISSN: 2640-4567, (Publisher Copyright: © 2024 The Author(s). Advanced Intelligent Systems published by Wiley-VCH GmbH.).