Heart failure (HF) is a global concern associated with significant morbidity and mortality [1]. Recent findings from the ASIAN‐HF registry suggest a potential shift in the HF burden from North America, Western Europe, and Eastern Europe to the Asia-Pacific region [2].

According to the ASIAN‐HF registry, within Asia, Southeast Asian patients have the highest burden of risk factors and worse outcomes than Northeast and South Asian patients [2,3]. This burden pressures individuals, their families, and the health care systems through various costs, with the most prominent being repeated hospitalizations [1]. For example, as high as 10% of hospital admissions are related to HF. The total HF costs accounted for approximately 1.8% of total health expenditure [4].

Studies show that HF’s rehospitalization and mortality rates were influenced by patients’ medication nonadherence [5-7]. As poor self-motivation and inadequate medication knowledge are the typical reasons for medication nonadherence, doctors and health care workers should emphasize the importance of medication adherence by constantly providing appropriate encouragement and education to patients [8,9].

Research has shown that some of these factors leading to hospitalizations are preventable by close home monitoring supported by family or nurse practitioners [6]. Nonetheless, such programs are challenging to apply in our local setting due to the limited number of specialized HF nurses who can support the wider HF patient population.

Therefore, we explore related work that uses conversational agent (CA), a type of artificial intelligence (AI) application that can be leveraged to assist in the self-monitoring of patients with HF in the following section.

A CA is a computer program capable of understanding natural human language (in text, speech, or both forms) and responding autonomously using the same language [10]. They can be accessed through a variety of ways, such as social media platforms (eg, Facebook Messenger), websites, and smartphone apps, or deployed using stand-alone digital devices (eg, Alexa, Google Assistant, and Siri). The first CA, ELIZA, was created by Joseph Weizenbaum at the Massachusetts Institute of Technology in 1966 [11].

ELIZA was developed to converse with the users via text, imitating a psychotherapist, to fool them into believing that they were talking to a human being. Today, thanks to technological advancements in AI, CAs can handle much more complex tasks in a wide variety of fields, including finance, education, travel, and retail [12-15], and they are predicted to be used even more widely in the future [16].

Engaging in natural conversation with humans is the main characteristic of CA, and current methods refer to conversation theory (demonstrated in Figure 1 [17]), such as using advanced machine learning methods to extract users’ intents from their utterances (speech) [18].

For a CA to produce natural conversations in a narrative manner, the format of the content must be outlined through rule-based workflows, templates, or intent-driven approaches to create an output. Every CA that uses a natural language system relies on narrative design, also called conversation design, to produce that output.

Conversational design combines several disciplines, including copywriting, user experience design, interaction design, visual design, motion design, and, if relevant, voice and audio design. Conversation design not only requires using natural conversational language but also creates logically sound conversational flow and design specifications that capture the entire user experience. More recently, machine learning capabilities have been used in CA to provide the ability to learn from the data so that an adaptable context of responses can be provided to the users.

There are several ways to generate the responses. First, is the rule-based method in which the CA produces a response by selecting it from a pool of predetermined responses either following simple rules to match phrases or identifying specific keywords in the text [19].

The second type is the generative-based CAs, which use AI algorithms to develop a contextual response informed by the system’s previous and ongoing learning [20].

Rule-based CAs allow developers greater control over the conversation content and flow, which is a useful feature when developing CAs for health care. By contrast, AI algorithms, particularly neural networks, may develop decisions that are not explainable or understood by the end user, referred to as the black box [20]. In health care settings, the black box effect may lead to biased or erroneous decision-making and patient harm which is highly dependent on the type of algorithms used to learn and generate the responses.

Therefore, in our work, we choose to develop a rule-based CA, given that it will allow developers better control and transparency in the responses.

Researchers have effectively innovated the application of CA in the digital health (DH) area, covering functions such as scheduling doctor appointments, monitoring medication intake, checking symptoms, diagnosing, providing treatment plans, and helping patients with rehabilitation [21-24]. DH has a broad scope that includes categories such as mobile health, health information technology, wearable devices, telehealth and telemedicine, and personalized medicine [7].

There are existing applications developed for supporting patients with HF. CARDIAC is a human-centered conversational assistant that helps patients with HF monitor their health status through reminders, question answering, relevant data collection, and generating data tendencies and personal health records [25]. Another CA, DIL, improves the self-care and quality of life of patients with HF by motivating them to adhere to a healthy lifestyle, including a controlled diet, a continuous medication routine, and regular exercise [26]. As a medication advisor, CARMIE speaks in Portuguese and interacts with patients with HF in real time to provide quality answers to medication-related questions according to its knowledge representation model and patients’ prescriptions [27].

Based on our literature review [10-12,26,28-31], the existing CAs in the HF area concentrated on developing functional features’ effectiveness and accuracy. However, no study has specifically displayed a method for building agents’ natural language–based conversations to encourage and educate patients with HF about medication adherence, nor a standard for evaluating this type of CA design early in the development stage as a DH solution.

Therefore, our study aims to adopt established design methods and conceive them into a systematic method that uses a clinical observational study protocol. We use observational study protocol to produce new knowledge in improving conversational design, examine acceptability, and reduce uncertainties in the harmful effects of using CA in medication adherence. It will fill the gap of the existing studies in the DH domain in designing a CA (or chatbot) that encourages and educates patients about medication adherence.

The goal of our study is to conceive a design method for developing CA for patients’ use in medication adherence, using design thinking as the main process for gathering requirements, prototyping, and testing.

We apply our design method in the ongoing development of MARIA, a rule-based CA.

The end goal of the study is to identify improvements in the functionality and dialogue construction of MARIA. This could be applied to leverage existing technologies that use CA or chatbot, such as ChatGPT or AWS Lex, to adapt it within a health care setting.

In this paper, we report on the results of our observation study protocol applying our design method for CA development.

Our conceived WoZ in an observational study protocol was designed to simulate the interaction of MARIA with participants, aiming to validate (testing) and refine template responses (ie, CA’s workflow dialogue) while gathering user experience feedback.

Given that the aim of using WoZ was ultimately to improve the design of a rule-based CA, we did not control for participants’ beliefs about whether they were interacting with a real person or whether the study procedure (ie, the MARIA prototype) was successful. Instead, participants interacting with MARIA believed it was autonomous. Our researcher (CHY), acting as the wizard, operated MARIA from another room.

The number of participants varies from one work to the other with no consensus on the ideal number of participants when used in a WoZ method. For example, the work of Bonial et al [73] involved 10 participants in the study. On the other hand, Nielsen and Norman’s [74] recommendation for usability testing, which the WoZ also falls into, required 5 participants to test. By contrast, in requirements elicitation [75], there are no specific guidelines for the number of persons required; it can vary from 2 to 12 persons.

Given that there is no agreement on the number of sample sizes, we follow a qualitative study recommendation of 20 samples [76] as an initial sample size. Furthermore, because the protocol is designed as an interactive process, researchers may stop to recruit further sample size when analysis suggests that data are saturated (ie, not many differences in the responses at a certain point).

MARIA_PRO_VER_3_190122 is registered with the Malaysia Medical Ethics Committee. The Medical Research Ethics Committee, the Ministry of Health Malaysia, approved the study with the registration number NMRR-21-1388-60672 (IIR). Patients provided informed consent before their involvement in the study and consented to use their data for analysis. The patients were provided compensation after completing the WoZ study.

Participant names for this research have been deidentified and linked only with a study identification number. Therefore, the research did not identify the participant’s identity and instead used anonymized identification numbers on all the data sets. All data are stored in Monash University Malaysia REDCap secured cloud and kept for 3 years. Participants can write to the investigators to request access to study findings.

During the recruitment and study period, there were 2 researchers, researcher A and researcher B, each located in separate facilities. Researcher A was based in the cardiac clinic, whereas researcher B operated from the Clinical Research Center (CRC) office. Participants were assigned to the cardiac clinic with researcher A. Researcher B worked from the CRC office (refer to Figure 3).

The study protocol allowed only 1 participant at a time in each room, with each session being conducted sequentially, 1 participant following another. Textbox 2 provides an explanation of the roles and responsibilities of the researcher and participant. Part A details the roles of researcher A and the participant, while part B outlines the responsibilities of researcher B.

Study participants were recruited from the Hospital Queen Elizabeth II, Sabah in Malaysia. The participant recruitment process was from June 2022 to November 2022.

The recruitment process followed the Malaysian Good Clinical Practice guidelines. The participants for this study were identified by CHY (principal investigator) at the HF clinic. During the consultation, the investigator explained the study to the patients and provided the consent form. If the patient fulfilled the inclusion and exclusion criteria, they were given sufficient time to read, discuss the study, and ask any questions. All questions were answered by the investigator. After addressing the patient’s concerns, the patient signed the consent form.

The study population included patients with chronic HF who were currently being followed up at the Cardiology Department Outpatient Clinic in Hospital Queen Elizabeth II. The inclusion criteria were: (1) age above 18 years, (2) diagnosis of chronic HF for at least one year, (3) history of symptomatic HF, (4) ability to write and speak Malay and English, (5) ability to type and use mobile app messaging, and (6) ability to comply with the protocol.

The exclusion criteria were as follows: (1) the presence of a clinical condition that would interfere with participation in the interview and (2) mental or legal incapacitation preventing the patient from providing informed consent.

Typically, the sample size is small at the beginning, as the goal is to explore the system. With each improvement, the process continues until an acceptable usability score or set of requirements is achieved [73-75].

As stated in the “The WoZ Method for Observational Study Protocol” section, given the lack of agreement on sample size, we follow a qualitative study recommendation of 20 samples [76].

We use usability scoring as a quantitative standard to determine the acceptability of the system’s design before proceeding with implementation. Hence, for the initial sample size, we used a convenience sampling method, recruiting a minimum of 20 patients for the study.

The total time required for each participant to participate in the study was a maximum of 1 hour.

We developed a coding guideline for analyzing the utterances, as detailed in Textbox 3.

The researcher tested the coding guideline before providing it to the clinical researcher, who then used it to analyze the collected data from the study participants.

We used Hoffman et al’s [80] evaluation of user trust in AI systems. Our questionnaire includes Likert-scale questions rated from 1 to 5, where 1 represents “I disagree strongly” and 5 represents “I agree strongly.” Additionally, we included open-ended questions to understand the reasons behind the given ratings. The questionnaire focuses on evaluating our conversational template design from various aspects (Figure 6), including human-likeness.

For human-likeness, which encompasses MARIA’s natural human language use, personality consistency, and expressed emotions, we define the criteria used for usability scoring.

The evaluation outcomes indicate that our conversational template design generally met the needs of stakeholders, including end users, patients, and pharmacists. MARIA’s natural language interactions, along with its encouragement and education strategies, are expected to support medication adherence among patients with HF in the future. However, the study also highlighted concerns regarding system liability and raised discussions on the extent to which MARIA should provide educational content on medication interactions and side effects in response to patient inquiries.

The design thinking method provided an iterative process that actively engaged end users from the early stages of developing the MARIA prototype, a rule-based CA.

The involvement of a multidisciplinary group of stakeholders during the ideation phase facilitated the early conceptualization of the dialogue workflow, guided by psychoeducational theory—specifically, Adlerian therapy.

During the testing phase, the WoZ methodology and user satisfaction scoring were integrated into an observational study protocol. This approach enabled the collection of simulated real-world dialogues between patients and the MARIA prototype, operated by the wizard (pharmacists), allowing for iterative refinement and validation of the CA’s conversational design.

The dialogues generated between the wizard (pharmacists) and the patients were systematically analyzed using coding analysis. This approach enabled the categorization of utterances into dialogue workflow components, speech acts, and topics, facilitating a structured evaluation of MARIA’s conversational framework.

Speech acts—such as informing and expressing gratitude—were examined in relation to their associated topics and mapped to the dialogue workflow. This mapping validated the practical application of Adlerian theory, demonstrating its effectiveness in guiding the wizard to motivate patients toward medication adherence. Furthermore, the user satisfaction scores from patients confirmed the feasibility of applying Adlerian theory within the medication adherence dialogue workflow.

Additionally, the analysis identified instances where patient-initiated utterances—either new topics or responses—were not covered in the predefined dialogue workflow. These gaps highlighted areas for further refinement in MARIA’s conversational design.

Building on this, the coding analysis reinforced the critical role of the wizard—played by an appropriate expert, in this case, pharmacists—as a key stakeholder in shaping how MARIA’s dialogues should be modeled. For instance, it became evident that advising on medication interactions and side effects cannot be delegated to the CA, as these responses require human expertise to ensure patient safety. This insight guided the identification of various scenarios that must be accounted for from a patient safety perspective when designing MARIA’s dialogue framework.

Furthermore, the user satisfaction scoring on human-likeness and trust highlighted the necessity of ensuring that MARIA’s dialogues and use of emojis align with professional communication standards. Patients expressed a greater willingness to trust MARIA’s advice on medication adherence when interactions were conducted professionally. This finding underscores the importance of designing CA interactions that balance humanlike engagement with a level of professionalism that fosters trust and credibility.

Through further analysis of the WoZ chatting history, we identified specific areas in MARIA’s designed conversations that required optimization. These insights guided refinements to the current template design, ensuring a more effective and user-centered interaction experience. Based on these findings, we iterated on the conversational templates and provided the final version to the MARIA research team for future implementation.

In specific interactions, the MARIA medical team, drawing from their practical experience with patients with HF across various age groups, suggested that formal language use may be more suitable than casual language.

The use of words such as “cool” in MARIA’s responses may create a more relaxed conversational style, which could be effective for younger patients but may not align with the preferences of older patients. Replacing “cool” with “excellent” could be more universally accepted across all age groups.

Specific messages should be designed to emphasize patients’ responsibility in self-managing medication adherence. For example, MARIA should educate patients that they are not merely completing a task instructed by MARIA but actively working toward their own health goals. The messaging should reinforce that patients are empowered to take charge of their health, while MARIA serves as an assistant, supporting them in improving their health status.

Educating patients about medication in advance can help alleviate their concerns. MARIA should provide reference links to information on medication and HF for patients to review before following their medication plan. This approach can enhance patients’ understanding of proper medication use, improve their awareness of potential side effects, and reduce the risk of misunderstandings about treatment effectiveness. Additionally, it may help prevent severe emergencies.

The evaluation outcomes indicate that our conversational template design generally met stakeholders’ needs. MARIA’s natural language conversations, along with its encouragement and education strategies, are expected to support patients with HF in adhering to their medication. We identified several modifications that could enhance the applicability of the current conversational templates.

This section discusses the study’s limitations and directions for future research. In this study, we were constrained by the absence of a database containing basic medication knowledge and patient stories of successful adherence to HF medication at the prototype stage. Future development should focus on enriching MARIA’s knowledge database to better support the designed education and encouragement strategies. The database should include comprehensive medication information from reliable sources and feature shared experiences of patients with HF who have successfully adhered to their treatment. Additionally, MARIA should be trained to provide tailored encouragement for patients facing various challenges in medication adherence. While linking to existing reputable HF associations worldwide is essential, collecting and curating real-life encouragement stories at the local level could improve cultural relevance and applicability. Furthermore, the study’s participant pool was predominantly male, with limited female representation. This gender imbalance may affect the generalizability of the findings, and future research should ensure a more balanced representation to strengthen the applicability of the results.

Furthermore, as this is the initial stage of development, our focus was on covering a broad range of aspects rather than deeply exploring anthropomorphism. In future development stages, we plan to conduct a more detailed evaluation of anthropomorphism to enhance MARIA’s human-like interactions.

This study demonstrated that applying design thinking processes provides practical, interactive steps to engage users early in the design, prototyping, and testing of a CA for supporting patients in self-managing their medication. Furthermore, using the WoZ simulation method within an observational study protocol at the testing stage proved to be a valuable approach for refining the CA’s interaction model, validating its functionality, and assessing system reliability, user expectations, and potential constraints. Results from the WoZ simulation and user satisfaction scores indicated that MARIA is a feasible and acceptable medication assistant CA. Additionally, patients expressed a general willingness to integrate MARIA into their daily routines to enhance medication adherence at home.