CHAT-AF was a single-blinded parallel RCT, with a 4:1 allocation chosen to optimize process evaluation [15]. Trial registration with the Australian and New Zealand Clinical Trials Registry was submitted on November 25, 2020, but COVID-19 delays led to the trial only being registered on February 18, 2021 (85 d post initial submission; ACTRN12621000174886), and at this time, there were 22 participants enrolled.

Participants were recruited from inpatient and outpatient cardiology services at Westmead Hospital. English-speaking adult patients with documented AF who had a mobile phone were eligible. Pregnant women and those participating in another clinical trial focused on providing AF education were excluded. Original plans to conduct a multicenter trial did not proceed after news of the acquisition of the technology partner by another organization (details below).

Randomization was 4:1 (intervention: control), stratified by sex, in blocks of 5. The allocation sequence was incorporated into REDCap (Research Electronic Data Capture; Vanderbilt University) [16] with access groups enabled, where data analysts were blinded. Participants, care providers, and research assistants were not blinded. Participants were randomized to receive usual care or the conversational AI intervention.

All participants completed study assessments via electronic surveys at baseline (in hospital or sent a link via text messages or email), 3-, and 6-months (via a link sent by text message or email). Baseline demographics and medical history were collected by participant self-report or by electronic medical records review (Multimedia Appendix 1). Questionnaires to assess primary and secondary outcomes were also completed electronically, either in person (baseline—during initial hospital visit) or sent via text message or email (3- and 6-month follow-up).

The human and research ethics committee at Western Sydney Local Health District (2020/ETH02546) approved this study. Informed consent, either written or over-the-phone consent, was obtained from all study participants. Data were collected and stored on secure servers accessible to approved study personnel only. Minimal participant data were provided to the technology partner via a secure file transfer protocol to enable delivery of the intervention phone calls, text messages, and emails. All data were deidentified for data analysis and publication. No compensation was offered to participants.

The CHAT-AF intervention design and development have been previously described in detail elsewhere [15]. In summary, it consisted of 7 outreaches (“digital visits”) and a semipersonalized education website, which was available as part of the 6-month digital intervention (Multimedia Appendix 1). The technology in the intervention was provided by HMS (Health Management Systems, Inc).

The outreaches were delivered via fully automated conversational AI phone calls (with speech recognition and natural language processing capabilities) and supplemented with an online survey (a personalized link sent via text message or email) when participants could not be reached after 3 call attempts. Two main components underpinned the conversational AI in the automated calls [17]: (1) a speech recognition engine that recognized voice responses and translated them into text, and (2) natural language processing that identified the semantic and syntactic elements from user utterances, progressing the flow of the call depending on patient answers, in a decision tree format. Given the proprietary nature of the software and the acquisition of the technology partner by another company, we were unable to obtain details about the network architecture, speech recognition, natural language processing capabilities, and other aspects related to the AI models in this intervention. During the phone calls, patients received AF education (eg, lifestyle information on diet and physical activity, importance of general practitioner [GP] follow-up, medication adherence, alcohol intake, blood pressure control, stroke, sleep apnea, and AF procedures) and were required to verbally respond to risk assessment queries (eg, overall health status, GP follow-up, AF symptoms and impact on daily life, medication confidence, and adherence). Certain patient responses to risk assessment questions would trigger alerts leading to an escalation pathway with clinician support, where needed. For example, if patients reported poor overall health status or significant impact of AF symptoms on daily life, these alerts would be actioned within 24 to 48 hours through a phone call by the researcher and additional clinical follow-up if required.

The 7 digital outreaches were also accompanied by weekly text messages or emails (depending on preferences) to a personalized link to an educational website that was tailored based on baseline characteristics (smoking status, alcohol consumption, hypertension diagnosis, and anticoagulant or warfarin prescription). This website contained AF-related information in the form of videos, texts, and images, as well as external links to online resources from reputable sources (eg, Heart Foundation and National Prescribing Service).

At the end of July 2021, we stopped recruitment as the delivery of the intervention was interrupted due to unforeseeable circumstances involving the acquisition of the technology partner by another organization. The trial steering committee made a decision to continue the trial to ensure the full 6-month program was delivered to all enrolled participants, by enabling feasible delivery of intervention content through text messages and emails. The technology partner had notified the research team in advance, allowing for the opportunity to develop an alternative approach and pre-emptively notify participants of the change. Further, comprehensive reports on each participant’s intervention completion status were made available, which allowed for a more seamless transition period. At this point, 82 intervention participants were recruited, and of these, 20 had received all outreaches, with the remaining 62 being at differing stages of the intervention timeline. All participants had received at least 2 of 7 outreaches via the automated calls before premature study completion. The survey (delivered via REDCap) contained replicated content and questions as asked in the phone calls but required participants to click their responses. The hope had been to find another technology partner that could deliver the intervention according to our specifications, but we were unable to do this, and limited by the remaining budget, we stopped the study to report findings. These changes were planned, reviewed, and approved by the trial steering committee. This trial is reported according to the CONSERVE (CONSORT [Consolidated Standards of Reporting Trials] and SPIRIT [Standard Protocol Items: Recommendations for Interventional Trials] Extension for RCTs Revised in Extenuating Circumstances) statement [18].

The primary outcome was change in AF-QoL assessed as Atrial Fibrillation Effect on Quality-of-Life (AFEQT) [19] total score at 6 months. AFEQT total score (0‐100) is an average of subscales (symptom, daily activity, or treatment), with higher scores indicating better QoL.

Secondary outcomes included AFEQT subscales, AF knowledge [20], patient activation [21], patient assessment of care quality and self-management support [22], self-reported lifestyle behaviors, medication adherence (days of missed doses in prescription medications over the past week), health care service use (GP or cardiologist visits, emergency department presentation or hospitalization, and ablation or cardioversion procedure), and health outcomes (stroke or myocardial infarction). All outcomes were assessed at baseline and 6 months; AF-QoL (AFEQT) was also assessed at 3 months. A detailed list of outcomes, methods, and time of data collection is provided online (Multimedia Appendix 1).

Process evaluation outcomes for the intervention group included: outreach completion and perceived usefulness, and individual engagement (Multimedia Appendix 1). Outreach completion was calculated as the number of outreaches with at least half of the questions answered divided by the number of participants that received the outreach. Outreach perceived usefulness was defined as the number of individuals who answered, “yes” to the question “Did you find the information in this call/survey helpful?” and was divided by the number of participants who attempted the outreach (answered at least 1 question). Individual engagement was calculated as the number of outreaches completed by the participant and categorized, where 4 or more completed outreaches (of 7) were considered as “higher engagement.” Metrics for engagement with the educational website were also explored.

A sample size of 385 was required to detect a between-group difference of 7 in the total score of the AFEQT questionnaire with 80% power (α=.05; SD=19), accounting for a dropout rate of 10% [19,23]. The current study was limited in detecting differences in primary outcome as we were only able to recruit 27% (103 participants) of the intended sample size.

Analyses were prespecified in a statistical analysis plan (Multimedia Appendix 2 [15,19-22,24]) and were conducted according to intention-to-treat principles. Analyses were performed using R statistical software (version 4.1.2; R Project for Statistical Computing). Outcomes were analyzed using either ANCOVA or logistic regression, adjusting for baseline variables. All tests were 2-tailed, a P value of <.05 was considered significant, and odds ratios were reported with 95% CIs. Normally distributed continuous variables were expressed as mean and SD. Nonnormally distributed variables were expressed as the median and IQR.

A univariate logistic regression analysis to predict higher engagement was conducted with covariates of age, gender, ethnicity, education, type of AF, time since AF diagnosis, and CHA₂DS₂-VASC score (congestive heart failure, hypertension, age 75 y and older [2 points], diabetes, stroke [2 points], vascular disease, aged 65 to 74 years, and sex category [female]; calculated as a sum, where 1-point or 2-points [where indicated] is given when aforementioned characteristics are present) [24].

Between January and July 2021, we enrolled 103 participants (82 intervention and 21 control; Figure 1). We lost 16 intervention participants and 3 controls to follow-up (18.4%), and the primary outcome analysis included 66 intervention participants and 19 controls. The follow-up period was from July 2021 to April 2022. Mean age was 63.7 (SD 11.2) years and 69.9% were males (Table 1). The majority were nonuniversity graduates (75.5%), of non-Caucasian ethnicity (24.5%), and had paroxysmal AF (76.7%). Detailed characteristics are provided online (Multimedia Appendix 1).

No significant difference was observed between groups in the primary outcome of AFEQT total score (2.08, 95% CI −7.79 to 11.96; P=.46; Figure 2 and Table 2). There were 18.4% missing primary outcome data at 6 months; however, we found no evidence for differences in missingness based on age (grouped by 65 y or older), gender, or ethnicity (grouped by Caucasian or non-Caucasian). Prespecified sensitivity analyses were conducted; both imputation of 3-month AFEQT total score carried forward and baseline imputation revealed similar results to the primary analysis.

No difference was observed between groups in the change in AFEQT subscale scores (Figure 2, Table 2). Additional exploratory analyses revealed an improvement in AFEQT total score postintervention in the intervention group (baseline: 69.9, 95% CI 64.4 to 75.5; 6-months: 79.9, 95% CI 74.9 to 84.8; P=.01), with no improvements in the control group (Figure 2). Within-group differences revealed the intervention group improved in most AFEQT subscales postintervention (symptom, daily activity, treatment, P values<.05), with no improvements seen in the control group (Figure 2, Table 2). No significant differences were observed in AFEQT total and subscale scores from baseline to 3 months between groups (Multimedia Appendix 1).

No significant differences were observed in secondary outcomes of knowledge (AF knowledge), patient activation (Patient Activation Measure), patient assessment of care quality and self-management (Patient Assessment of Chronic Illness Care), or lifestyle behaviors (exercise, fruit and vegetable intake, alcohol intake, and smoking) at 6 months (Multimedia Appendix 1). No significant difference was observed between groups in the proportion that were adherent to medications, visited a GP or cardiologist, visited the emergency department, or were hospitalized and had an AF procedure (Multimedia Appendix 1). A total of 3% of intervention participants had a stroke or myocardial infarct in the previous 6 months compared to no controls (Multimedia Appendix 1).

A total of 338 of 550 outreaches were delivered via the conversational AI calls, and of these, 226 (66.9%) were completed (original delivery; Multimedia Appendix 1). The remaining 212 outreaches were delivered only by a survey tool, and of these, 112 (52.8%) were completed (modified delivery). The completion rate of the first outreach was 75.6% (62 calls and 0 surveys), and this dropped to 47.4% (8 calls and 29 surveys) by the final outreach, with an average completion rate across the 7 outreaches of 61.5% (Figure 3). On average, each participant completed 4.12 of 7 outreaches. A total of 51 participants (62.2%) had higher engagement (completed 4 or more outreaches), and there were no demographic variables influencing this outcome (age, gender, ethnicity, education, type of AF, time since AF diagnosis, and CHA₂DS₂-VASC). Most participants (56.1%) visited the educational website at least once, and the mean number of visits was 5.54 times. The most visited topic was general information about AF (138 visits), which included videos narrated by a local cardiologist. In terms of perceived usefulness, 88.4% of completed outreach was reported as useful (Figure 3) and this was similar in the original (89.1%) and the modified delivery (87.6%). Detailed process evaluation results are provided online (Multimedia Appendix 1).

The CHAT-AF study provides a proof-of-concept with initial data on the efficacy of a novel digital health follow-up strategy for patients with AF that leverages conversational AI. On average, participants completed 4 of 7 outreaches, and 88.4% of completed outreaches were reported as useful, suggesting this approach to delivering care is feasible and could enable chronic disease services to follow up with patients at scale with fewer in-person and staffed visits. Due to challenges with the technology partner acquisition, this study had only recruited 27% of its intended sample size and was subsequently underpowered for its primary outcome. The trial did not demonstrate a significant difference in the primary outcome between intervention and control; however, there was a benefit suggested by an exploratory prepost comparison that showed overall AF-QoL improved at 6 months in the intervention group from baseline, with no similar improvement observed in the control group.

Digital support programs show promise in improving QoL for patients with AF, but more robust studies are needed to determine effectiveness. Despite this study’s limited power to determine efficacy, other trials have reported improvements in AF-QoL, knowledge, medication adherence, and clinical outcomes, with similar digital health interventions (without conversational AI) that delivered a combination of health education, monitoring, and self-management support for patients with AF [4-6,25]. However, the paucity of literature in this space has been confined to mobile apps [4,6,25-27], text messaging [26,28], and web-based platforms [5,8,9], many of which have reported limited engagement. The current intervention was unique in its approach to interactively engage with patients through conversational technology. A comparable study by Guhl et al 2020 [4] involved an embodied conversational agent that consisted of a virtual avatar displaying verbal and nonverbal gestures to deliver education and heart rhythm monitoring support to patients with AF [4]. Participants interacted with the conversational agent 18 times over 30 days and had improved medication adherence and AF-QoL compared with usual care [4]. Notably, this intervention did not include an AI component and required users to respond to queries by clicking on prespecified options in the mobile app, rather than using speech recognition technology, which was used in CHAT-AF to facilitate more natural dialogue and better engagement.

AI-enabled conversational technologies allow for more human-like interactions and have shown promise in other populations. Existing trials of conversational AI have demonstrated improvements in insulin adherence and glycemic control in patients with type 2 diabetes [12], medication adherence in patients with hypertension and diabetes [11], and symptoms of depression and anxiety in college students [13,14]. These studies reported good engagement over a 2-week period, where college students exchanged 283 messages with a chatbot [14], and undertook 12 check-ins with another chatbot [13] regarding their mental health. Another study that used a voice-based conversational AI interface, resembling the phone calls in the current intervention, found patients with type 2 diabetes logged daily insulin use and fasting blood glucose levels almost every day for about 4 months [12]. Similar to the phone call–based delivery mode used in the current study, another study delivered cognitive behavioral therapy for pain management using voice-based conversational-AI phone calls and reported that 87.8% of calls were completed [29]. In our study, the transition from phone call to solely survey-based delivery may have resulted in completion rates (67%) lower than reported in the aforementioned study. Overall, evidence suggests conversational AI technologies can successfully engage and support patients with chronic disease management, but additional studies are needed to evaluate long-term engagement.

The deeper level of human-technology interaction allowed by conversational AI seems a key factor in achieving higher user engagement [12-14,30-32]. The unplanned change in intervention delivery in the current study offered an important opportunity to compare engagement between different communication modes. Interestingly, the shift from conversational AI phone calls to surveys resulted in a drop in overall outreach completion rates (from 66.9% to 52.8%). A hypothesized explanation may be the stronger appeal of the human-like interactions provided by these technologies and their ability to facilitate natural dialogues with users. This is in line with existing work indicating the possibility of a relationship between humans and nonhuman agents in the context of health [12-14]. As digital health interventions increasingly incorporate elements that make them interactive, adaptive, persuasive, and personalized, they become more able to reproduce elements of a therapeutic relationship and can better engage and support patients in their health journey [33].

The early stopping of the trial due to suspended delivery of the intervention by the technology partner posed challenges that are important to consider. For the trial steering committee, it was key that enrolled participants received the entirety of the 6-month program, to satisfy the duty of care for enrolled patients as well as to ensure fidelity in delivery of all educational content in the program. The decision to use REDCap [16] surveys when the automated phone calls had to be stopped offered a satisfactory solution to these concerns and also provided the opportunity to observe feasibility measures such as engagement and perceived usefulness rates between these 2 modes of delivery while the other elements were unchanged [30,31]. An attenuating factor for early stopping was that the trial steering committee was forewarned of when the company was going to withdraw services, and that allowed for adequate time to develop an alternative solution and streamline the transition for participants. Other articles have reported early trial stopping due to technology partner withdrawal of services [34]. Open communication and goal alignment have been proposed as key to achieving solutions that can benefit all parties involved and bridge the gap in “academia-industry” relationships [35,36]. Stronger partnership between sectors is increasingly needed as collaboration between technological, research, and clinical expertise is paramount to successfully implement digital health solutions.

The growing prevalence of cardiovascular diseases such as AF, which require ongoing management to prevent frequent hospitalization, puts a significant toll on health care resources. Interventions such as CHAT-AF have the potential to provide support at scale, with a risk management system that allows patients at risk of deteriorating to be identified, prioritized, and managed appropriately. Using phone calls to provide patient support has advantages over other mobile technologies (eg, apps and wearable devices) as it is not dependent on internet connectivity, phone models, or operating systems. Digital interventions such as CHAT-AF could be used to provide support to patients with AF in the community, including remote and rural areas where patients have traditionally had poorer access to health care services [37]. As demonstrated in this proof-of-concept study, the engagement, perceived usefulness, and initial suggestive findings of AF-QoL improvement argue that this technology should be further examined in larger RCTs.

This study has several limitations that need to be considered. This study was limited in not achieving its target sample size and multicenter reach due to premature study completion, and therefore, careful consideration needs to be made when interpreting the results as this study was underpowered. This also meant some participants on the intervention arm were unable to complete their conversational AI “visits” later into the planned follow-up period, which is likely to have contributed to the ability to evaluate the effectiveness of the intervention on outcomes. The authors are in the process of conducting a large multicenter RCT evaluation of an optimized version of the current intervention, to address the power, sample size, and interpretation limitations of this study (registered with the Australian and New Zealand Clinical Trials Registry, ACTRN12623000850673). Moreover, except for baseline characteristics (medical history), all outcome data were self-reported. There was a difference in attrition between the intervention (16/82) and control (3/21) groups, which may have impacted the primary analysis, and while there was no evidence of bias for missingness, caution needs to be taken in interpretation due to the limitations in lack of power and changes in intervention delivery. This study’s population was younger (mean age of 63.7 y) than the average patients with AF population in Australia (mean age of 75 y) [38], which may have occurred due to the higher likelihood of younger patients’ comfort and interest in participating in technology-based research [35]. Perceived usefulness was assessed as the final question of each outreach and therefore results are based on individuals that completed the particular outreach and its final question—this may not be reflective of overall intervention usefulness, as it does not reflect outreaches not completed by participants (344 of 550 outreaches were completed by participants and were rated in terms of their perceived usefulness, of these, 304 of the 344, 88.4%, were assessed positively based on usefulness). However, to capture the perspectives of a diverse group of the intervention cohort, we conducted qualitative interviews using purposive sampling techniques to ensure we captured the experiences of those with varying levels of engagement [38,39]. Furthermore, the study was delivered in English and included predominately Caucasian participants, highlighting the need for further work to validate this technology in larger trials with more diverse and representative patient populations.

This study found that a conversational AI follow-up program for patients with AF improved AF-QoL postintervention (but not compared with usual care). As the burden of AF continues to grow, novel technologies that can interact with patients and support them in their care journey will be needed, and digital health can provide this at a scalable level. However, larger-scale RCTs and implementation studies are needed to determine the effectiveness of conversational AI in improving AF outcomes.