3D Multimodal Medical Imaging System Based on Freehand Ultrasound, Structured Light, and Stereo Vision Tracking

Abstract

The most used medical imaging techniques are Magnetic Resonance Imaging (MRI), Com- puted Tomography (CT), and Ultrasound (US). Typically, they provide 2D information even though the structure of a patient is in 3D. We can acquire volumetric data with US but this 3D US data does not provide information about the external topography of the patient. We can complement 3D US data with the external surface of the subject or region of interest for a better interpretation by the physicians. In this thesis, we propose a 3D multimodal medical imaging system that combines freehand ultrasound and structured light 3D reconstruction in a single coordinate system without requiring registration. With this method, we can acquire the internal structure in 3D with freehand ultrasound, and the external features with structured light. To the best of our knowledge, these techniques have not been combined before as a multimodal imaging system. We can use our proposed multimodal technique as a tool for preoperative and intraoperative tasks or even for diagnostic and treatment planning. Our proposed method shows the potential as a navigation system because we acquire the 3D pose of the ultrasound probe and the 3D surface topography in the same coordinate frames. This enables us to show the external surface with a model of a probe and with the US slices acquired in real-time and 3D. For pose estimation, we propose an approach based on deep learning, a planar marker of three circles, and a calibrated stereo vision system. We compare this approach with a method implemented using classical computer vision techniques. Finally, we carried out an evaluation study of different triangulation methods and their influence on the reconstruction error for structured light systems. We discuss what triangulation approach to use based on the conditions and required application.