This study was conducted under institutional review board approval. Medical trainees from Children’s National Hospital in Washington, DC, were recruited for the study (N=22). Of the participants, there were 1 first-year, 10 second-year, 4 third-year, and 1 fourth-year residents and 6 medical students. The participants were split into groups of 4 or 5 to maintain small group discussions. All participants gave informed consent prior to their participation. A minimum of 20 participants were recruited to achieve a power of 80% and a level of significance of 5% (2-sided) for detecting an effect size of 0.7 between pairs.

An experienced pediatric cardiologist from Children’s National Hospital acted as a moderator in the study. The moderator’s role and responsibilities were to give lectures on three chosen cases of CHD, provide instructions on how to interact with the display systems, and present answers to the diagnostic questions. The moderator’s interaction with the participants strictly followed a prewritten script.

The study evaluated three medical imaging display systems—conventional, nonimmersive VR, and full-immersive VR (Figure 2)—for group diagnostic discussions of the CHDs. The conventional display system visualized 2D medical images on a projector screen. The nonimmersive VR system projected 3D medical images visualized in Surface Pro tablet (Microsoft Corp) onto a shared screen. A mobile HMD, Gear VR (Samsung Electronics Co Ltd), was provided to each participant for the full-immersive VR system, where 3D medical images were visualized in a virtual world.

A group orientation was provided at the beginning of the study. After the participants gave informed consent, the demographic survey was distributed and the moderator introduced himself and requested that participants greet each other and introduce themselves. The study was organized into 3 cases of CHD, each with 3 separate phases: lecture, group discussion, and postdiscussion survey. To avoid order bias, the 3 cases of CHD were randomly coupled with 3 display systems and provided in random order (Table 2). These selections were accomplished by running a MATLAB (The MathWorks Inc) script that generated random discussion combinations and orders.

For the lecture phase, the moderator prepared PowerPoint slides with a brief summary of each CHD case. The moderator explained the deviations from the norm presented for each CHD case under discussion and its standard diagnostic approach. The group discussion phase was solely held by the participants, who were not permitted to ask the moderator any questions that could be a hint at the CHD diagnosis. Discussions were limited to 10 minutes but could adjourn early if consensus were made within a group. All discussions were audio recorded for measuring the time duration for each group to reach a consensus.

The postdiscussion survey was provided to be answered individually, based on the possibility of individual learning variance from group discussions [40]. The discussion that used FIV included an additional survey about the experience of wearing the HMD. A comparison survey was given to each participant at the end of the study.

Kruskal-Wallis and Dunn tests were performed to confirm the level of medical experience matched between the assigned groups (n=5). Additionally, an analysis of variance test was performed to test whether diagnostic accuracies varied between the groups. The Kruskal-Wallis test followed by the Dunn test were also used in comparing the changes in diagnostic accuracy between the display systems in each CHD variation. Friedman and Wilcoxon signed-rank tests were used in peer assessment and the comparison survey to determine whether responses differed between display systems. A Fisher exact test was performed to determine the influence of usability of HMD (ie, motion sickness and physical discomfort) on the postdiscussion survey and comparison survey responses. The Kruskal-Wallis test was performed to explore any influence of impression of VR before experiment on preference rating. All statistical analysis was performed on R 64-bit version 3.5.3 software (R Foundation for Statistical Computing) with significance being .05 or lower in P values.

Two participants claimed to have prior experience with using immersive VR from playing VR games and exploring real estate property in VR. Both participants had moderately positive impressions of VR (ie, 1 and 2 points on a scale between –5 and +5) compared with the group of 20 participants without prior experience with VR (2.6 [SD 1.96]). The Kruskal-Wallis test showed that the results on VR impression did not show any difference in the choice of display preference (CDS χ²=4.1, P=.53; NIV χ²=4.5, P=.49; FIV χ²=1.3, P=.93). Due to the small sample size of those with prior VR experience, we could not directly test if VR experience influenced the choice of the most preferred display.

Between 4 to 5 participants were assigned to each of 5 groups according to participant availability. The Kruskal-Wallis test (χ²₄=10.7, P=.03) was used to gauge differences in medical experience levels between the groups (Table 4). Further investigation with the Dunn test using the Benjamini-Hochberg method showed that group 3 had more medically experienced participants than group 5 (z=–2.19, P=.048). However, the diagnostic accuracy performance in CHD cases between the 5 groups did not show any statistical difference based on analysis of variance.

The discussion times of each group (n=5) and CHD variation are shown in Table 5. Despite being classified as the least complex CHD case in the study, the averaged discussion times for ASD (172 seconds) were slightly longer than for CoA (159 seconds). The trend was visible in all groups except for group 3. All groups spent the longest discussion time on the MAPCA case. The averaged discussion time for MAPCA was 382 seconds, more than twice the time for the ASD and CoA discussions.

Each participant received a diagnostic accuracy score ranging between 0 and 4 for each of the CHD cases. Two cardiologists individually graded the participants’ diagnostic performance according to the rubric (Table 3). The diagnostic accuracy scores were compared for each display system and CHD variation and broken down by cardiologist (Figure 5). No statistical differences were found between the grades of the two scorers (t=–1.01, P=.31).

The overall diagnostic accuracy difference between the display systems was statistically significant (χ²₂=9.0, P=.01) where FIV had the highest averaged accuracy (Table 6). Differences became even more prominent with increasing case complexity (χ²₂=14.1, P<.001; Table 6). For MAPCA, the average score percentage differences between the groups that used FIV to CDS and NIV were 54.49% and 146.82%, respectively. With the rise of CHD complexity, decreases in average scores of 35.59% and 82.86% were observed in CDS and NIV, respectively. Indeed, the Dunn test indicated that the averaged diagnostic accuracy of MAPCA for the FIV groups were significantly higher compared with the NIV groups (z=3.57, P=.001; Table 6).

No significant difference between the display systems for each peer assessment questionnaire was found using the Friedman test (Q1 χ²=0.4, P=.82; Q2 χ²=3.4, P=.18; Q3 χ²=0.4, P=.82; Q4 χ²=1.3, P=.53; Q5 χ²=1.8, P=.42; Q6 χ²=0.2, P=.934; Q7 χ²=3.4, P=.19).

Approximately two-thirds of the participants (15/22, 68%) ranked the FIV as the most preferred display system for performing group diagnostic discussions, and the rest of the participants (7/22, 33%) chose the NIV. The preference ranking ratings of the display systems were statistically significantly different based on the outcome of the Friedman test (χ²₂=31.6, P<.001). Further testing with Wilcoxon signed-rank tests indicated that the median ranking rating of the FIV was statistically significantly higher than the median ranking rating of the NIV (z=3.33, P<.001) and CDS (z=4.10, P<.001). The NIV (10/22, 46%) and the FIV (8/22, 36%) received a similar number of votes for the easiest display system to use in group discussions or roughly twice as many as the number of the votes for CDS (4/22, 18%). The Friedman test revealed that a statistically significant difference between the display systems existed on the display system ease ranking (χ²₂=20.6, P<.001). The median CDS ranking rating on ease of use were found to be statistically significantly lower than that of FIV (z=1.93, P=.047) and NIV (z=2.39, P=.01; Table 7) using the Wilcoxon signed-rank test.

A total of 27% (6/22) of participant reported motion sickness from wearing the HMD. These participants were more likely to provide lower scores on concentration (P=.009) and knowledge exchange (P=.046) of the discussion using the FIV. Almost half (10/22, 45%) of participants experienced some level of physical discomfort especially around their noses from the heaviness of the HMD. Of the participants wearing glasses, 40% (2/5) removed them while using the HMD due to physical discomfort. However, no statistical differences were found between the groups that did or did not report physical discomfort and wore or did not wear glasses on all surveys (Table 8 and 9) and diagnostic performance using FIV (Table 8).

The FIV involved wearing HMD, which caused some physical discomfort and motion sickness and required training. These experiences could result in negative emotions, which are shown to be negatively related to team performances [41]. However, the FIV was rated as the most preferred medical image display system. The survey responses revealed that the realistic and interactive visualization ability such as interior viewing, rotating, and zooming in/out of the 3D anatomical models were the reasons for the better rating. CDS required succinct and accurate verbal directions for adjusting the sagittal, coronal, and frontal planes to orient the cross-sectional viewing and then processing them into the volumetric anatomy. The higher demand of mental conceptualization made the CDS the most difficult display system to use.

FIV has been facing criticism for the absence of face-to-face communication, which is related to reduced group collaboration quality [42-44]. Some studies emphasized the use of avatars as a remedy to improve social presence and communication in VR [45,46]. Although face-to-face communication was not featured in our clinical viewing software, providing shared perspective of anatomies and a mechanism for concurrent feedback was sufficient for enhanced group diagnostic performance. FIV showed strong diagnostic accuracy regardless of the CHD complexity unlike the other display systems, which showed worsened accuracy with increasing complexity.

To investigate the group diagnostic performances, an active discussion environment with experienced physicians across multiple disciplines was desirable. However, the recruitment process was challenged by a limited pool of trained physicians and their busy schedules. The study, therefore, recruited medical trainees who varied in years of medical experiences. A larger number of participants would increase the power of statistical results, but there was limited availability of pediatric residents (40 residents per class, with competing clinical demands), making 22 participants a realistic recruitment achievement. Due to the small data size, high standard deviations were observed in CDS and NIV, which increases the range of true diagnostic accuracy value. However, since the probability of false positive is lower with a smaller data sample size [47], the probability of falsely rejecting the differences in MAPCA diagnostic accuracy between display system is small. Nevertheless, we plan to narrow the spectrum of CHD cases down to the STAT categories of 4 and 5 in a future study to further evaluate the impact of displays on complex CHD cases.

Gender difference was disregarded in the study since the ratio of women to men was 19 to 3. Groups were formed based on participant availability. We confirmed through the Dunn test using the Benjamini-Hochberg method that except for groups 3 and 5 (P=.048), years of medical experience (eg, residency standing) did not vary between the groups. Since there was no statistical diagnostic performance difference found between the groups, we conclude that the participants’ years of experience were not an influential factor for performing tasks provided in the study.

The exact reason behind FIV being more preferred than the NIV could not be identified through the comparison survey results. Since the features of FIV and NIV are identical except for the immersion aspect, we suspect the perceived novelty of the VR experience could have impacted the choice in preferred displays. There were 20 participants who had never experienced VR prior to the experiments. To them, everything about FIV was novel, therefore they could have experienced increased perceived reward and individual preference toward the VR [48], leading them to prefer the FIV for CHD diagnosis tasks. To test the hypothesis, a future study will include a survey on novelty and compare its result on the preference rating.

The Cardiac Review 3D currently uses DICOM images that are 3D reconstructed and segmented into stereolithography file-formatted data. Since 3D reconstruction is not a routinely performed task in medicine, stereolithography is not stored in DICOM or part of the electronic patient records. To be compatible with the existing medical workflow, the logistics of storing 3D reconstructed data in DICOM is being identified.

The Cardiac Review 3D is unique clinical viewing software with multiuser access and interaction. The software allows for visualization and manipulations of 3D anatomical models through zooming, rotating, panning, linear measurement, and adjustable clipping plane features. The text annotations, screenshots, and report features allow for taking notes in text and image forms for future access and archiving. User datagram protocol was implemented with virtual pointer for multiuser access and participation.

This study evaluated the group diagnostic discussion performances of the CDS, NIV, and FIV. Despite the lack of face-to-face communication and reduced concentration from motion sickness, the group discussions that used FIV demonstrated the best diagnostic accuracy overall and particularly for the most complex form of CHD. It also was the only display system that showed improving trend of diagnostic accuracy with increasing CHD complexity. The FIV relied on bulky hardware associated with physical discomfort, motion sickness, and increased learning process; however, it was still the most preferred display system for performing group diagnostic discussions.

The application of FIV has successfully supported improved diagnostic accuracy in CHD group discussions. FIV has the potential to bolster collaborative performance in discussion of other anatomies, medical education, and surgical planning. Expanding the significance of our findings, we believe that nonmedical fields such as computer-aided design, architecture, urban design, search and rescue, and military training that necessitate understanding of complex 3D structures may benefit from the use of FIV in collaborative discussions.

The Cardiac Review 3D provides features for medical doctors to visualize and interact with the patient anatomies in 3D in a group setting. Implementation of this technology could potentially bolster the diagnosis and preoperative planning of CHD, especially for complex cases (eg, MAPCA) by reducing the mental workload and capability of converting 2D cross-sectional images of anatomies into 3D and easily maneuvering around the anatomies.