The CT imaging datasets used in this study consisted of volumetric scans acquired with variable technical parameters. The number of slices per volume ranged from 205 to 850, reflecting differences in anatomical coverage and acquisition settings. Field of view dimensions spanned from smaller regions of 176–186 mm to extended coverage exceeding 490 mm, while the in-plane matrix size varied between 512 × 512, 768 × 768, and, in one instance, a higher resolution acquisition of over 2000 × 1900 pixels. Voxel resolutions demonstrated a wide spectrum, with the smallest sampling at 0.1 × 0.1 mm in-plane and slice thickness below 1 mm, and the largest spanning over 1 mm in-plane with slice thickness up to 1.5 mm. A description of the cases regarding scan regions and pathological findings is included in Table 3.
Imaging devices
Experiments have been carried out with data from a photon counting CT (NAEOTOM Alpha, Siemens Healthineers, Germany), a 3 T MRI (CIMA.X, Siemens Healthineers, Germany), an ultrasound (Epiq 5G, Philips Healthcare, the Netherlands), and a PET/CT (Philips Vereos, Philips Healthcare, the Netherlands).
Computational setup
The radiological desktop computer was a Hewlett-Peckard HP Z620 Workstation with an Intel Xeon CPU E5-2620 with 6 cores and 2.0 GHz, RAM of 16 GB, solely tasked with raw data anonymization and export. The workstation ran on a Microsoft Windows 10 Education operating system. Processing of priorly anonymized raw data was executed exclusively using an Apple MacBook Air with a M1 processor, 8 GB RAM and macOS 14.4.1. No GPU acceleration was used, despite the Apple M1 processor, because Fiji does not utilize GPUs in this build version. The task performed in 3D Slicer was also carried out without GPU acceleration.
Mobile device
For all displays of AR data an Apple iPhone 13 mini with 128 GB storage running on iOS version 17.4.1 and an iPad Pro with 1 TB storage running on iPadOS version 17.5.1 were used. We successfully tested the principal software compatibility on iPhone 7, iPhone 14 and iPhone 15 Pro.
Fiji (Fiji is just ImageJ)
Fiji Version 2.14.0/1.54f, an open-source platform built on ImageJ, was used for processing image data10. All mathematical operations were carried out within Fiji.
3D Slicer
3D Slicer (Version 5.0.3) is an open-source software platform for medical image informatics, image processing, and three-dimensional visualization. Herein, the ‘Create a DICOM Series’ was exploited to generate new DICOM series.
Medical Imaging XR
Medical Imaging XR (Version 2.6.8) is an open-source program, provided by ‘nooon WEB&IT GmbH’ and under the copyright of Medicalholodeck™. The software is capable to display DICOM stacks after cloud processing in AR. The user can adjust the LUT freely and crop the sample in real-time in relation of the mobile device to the projected sample.
VR-prep for single modality image stack generation
VR-prep uses Fiji to convert a DICOM data set (or other formats) into a.tiff file. The intensity values are verified, converted to a positive range, converted to isotropic 1 mm resolution, and finally, the series is converted to an unsigned 16-bit. The image series must initiate caudally and move cranially. Subsequent image stacks are a single .tiff file encompassing the entirety of the individual images. This.tiff file is imported to 3D Slicer to generate a DICOM series with the plugin “Create DICOM series”, that can be uploaded to Medical Imaging XR, for 3D object generation. Time need to upload the DICOM series to generate the QR code and to upload the AR model to the app on the tablet was measured with a stop watch.
VR-prep for combining different modality image stacks into a single VR-object
CT and PET images were first acquired independently and underwent the VR-prep operations within Fiji. The resulting two .tiff image stacks were subsequently aligned to ensure spatial correspondence between the two modalities. Following alignment, each voxel was initially assigned a CT-derived pseudo-Hounsfiled-Unit [pseudo-HU]. In particular, CT values were compressed into a 0–499 value range, scaling linearly in the range from − 1000 to + 1000 HU. Values below or in excess of this interval were assigned the pseudo-HU value 0 (for values <−1000 HU) or 499 (for values > + 1000 HU). PET values were then evaluated against a predefined threshold, and for voxels exceeding this threshold, a pseudo-HU was determined scaling linear to the individual set threshold (500 pseudo-HU) to a standard-uptake-value of 100 Bq/mL (defined as 999 pseudo-HU). For each coordinate, the CT-derived pseudo-HU value was initially set. If for a given coordinate the PET-derived signal exceeded the priorly mentioned threshold, the CT-derived value was overridden with the PET-derived value, mapped into the pseudo-HU space. This process produced a final integrated image stack in which each voxel contained a continuous pseudo-HU value, enabling visualization and quantitative analysis of both structural and functional information within a single dataset. Figure 6 illustrates the operations of this procedure schematically.
Quantification of performance
To evaluate the performance of VR-prep, we uploaded the original DICOM series as well as the VR-prep DICOM series to Medical Imaging XR and quantified the time. Following, we quantified the time to download the 3D object into an iPad Pro running on iPadOS version 17.5.1. For quality assessment of AR images, 3 radiologist with 4 (AMCB), 1 (MB) and 1 (LO) years of experience evaluated five parameters: adequacy of look up tables generated (labeled as LUT), sharpness, signal to noise ratio (SNR), possibility of correct identification of anatomical structures (labeled as Anatomy), and confidence in use of the generated AR image in diagnostic configurations (labeled as Diagnostic confidence). A 5-tier Likert scale was used with the following descriptors: 1 – extremely bad, 2 – bad, 3 -intermediate, 4 – good, and 5 – extremely good. All evaluators analyzed all images, blinded and at a random order. They were allowed as much time per image as they deemed necessary and were allowed to use all functions of the app, including change of intensity values, crop, and zoom.
To compare VR-prep performance with original DICOM series exported with different frame increments, we reformatted in PACS an image series to different slice thicknesses (3 and 5 mm) and used the original 1 mm to run VR-prep. The original image series was acquired at 1 mm increment, with a voxel size of 0.5566 × 0.5566 × 0.9990 mm3, the VR-prepped series had a voxel size of 1 × 1 × 1 mm3, and the reformatted DICOM series to 3 and 5 mm had a voxel size of 0.5594 × 0.5594 × 3.0057 mm3 and 0.5594 × 0.5594 × 4.9639 mm3, respectively. Duration of upload of DICOM series to MIXR and duration of download to the mobile device were quantified.
Integration of MRI and microscopy in VR-prep
An MRI DICOM series was procured on our PACS data base. The MRI DICOM series can be directly run in VR-prep.
For microscopy data set we downloaded a publicly available data set of a flower bud lobe, acquired with a light-sheet-fluorescence-microscope12. The downloaded images in .png format were imported to Fiji, converted into .tiff and processed further with VR-prep.
Integration of ultrasound in VR-prep
Ultrasound data sets are not DICOM series, and are typically a colored .jpeg. Images are acquired continuously with a probe, hence the time dimension is added at the expense of the stack-depth dimension. By following a stringent path at a stable speed using the probe, the operator can capture an anatomical region in pseudo-3D. We opened the .jpeg data set in Fiji, converted into.tiff and processed further with VR-prep.
Integration of PET-CT in VR-prep
PET and CT data set must be integrated in the same DICOM series to be transferred to MIXR. With this in mind, we adjusted in Fiji the intensity values of the CT series to a range of 0 to 100 and the PET series to a range of 105 to 255, in 8-bit. Series were added to each other and run in VR-prep.
Integration of time comparison 3D images
To combine image series of different time points side by side, we used Fiji command “combine stack”. Then, images were run in VR-prep.
Statistical analysis
GraphPad Prism (Version 10.4.0), developed by GraphPad Software, LLC, was used for the statistical evaluation. Significance was set to a p-value lower than 0.05 for a paired t-test. Results of the Likert scale are presented as median and Interquartile Range (IQR). The evaluation of the 5-tier Likert scale was calculated with GraphPad Prism (Version 10.4.0). Median and IQR were analyzed with Wilcoxon test and statistical significance was set to 0.05. ICC was calculated using SPSS, with a k = 3 and 95% CI.
Manuscript writing
We acknowledge the use of ChatGTP 3.5 for proofreading our manuscript.