Published on in Vol 6, No 1 (2022): Jan-Jun

Preprints (earlier versions) of this paper are available at, first published .
Factors Affecting Patient and Physician Engagement in Remote Health Care for Heart Failure: Systematic Review

Factors Affecting Patient and Physician Engagement in Remote Health Care for Heart Failure: Systematic Review

Factors Affecting Patient and Physician Engagement in Remote Health Care for Heart Failure: Systematic Review


Institute of Systems Molecular and Integrative Biology, University of Liverpool, Liverpool, United Kingdom

Corresponding Author:

Ahmed Al-Naher, BSc, MBBS, PhD

Institute of Systems Molecular and Integrative Biology

University of Liverpool

Block A: Waterhouse Buildings

1-5 Brownlow Street

Liverpool, L69 3GL

United Kingdom

Phone: 44 01517942000


Background: Adult chronic heart failure mainly affects an elderly population with multiple comorbidities that often require frequent medical visits to prevent poor health outcomes. However, the heart failure disease process reduces their independence by reducing mobility, exercise tolerance, and cognitive decline. Remote care technologies can bridge the gap in care for these patients by allowing them to be followed up within the comfort of their home and encourage their self-care. However, patients, carers, and health care professionals need to engage with the technology for it to be useful.

Objective: This systematic review explores qualitative primary studies of remote care technologies used in heart failure, to determine the factors that affect user engagement with the technology. This is explored from the perspective of patients, carers, and health care professionals.

Methods: Relevant studies published between January 1, 1990, and September 19, 2020, were identified from EMBASE, Ovid MEDLINE, PubMed, Cochrane Library, and Scopus. These studies were then synthesized using thematic analysis. Relevant user experiences with remote care were extracted using line-by-line coding. These codes were summarized into secondary codes and core concepts, which were further merged into overarching themes that encapsulate user experience with remote care.

Results: The review included 47 studies, which led to the generation of 5 overarching themes that affect engagement: (1) “Convenience” relates to time saved by the intervention; (2) “Clinical Care” relates to perceived quality of care and health outcomes; (3) “Communication” involves feedback and interaction between patients, staff, and carers; (4) “Education” concerns the tailored information provided; and (5) “Ease of Use” relates to accessibility and technical barriers to engagement. Each theme was applied to each user base of patient, carer, and health care professional in a different manner.

Conclusions: The 5 themes identified highlight aspects of remote care that facilitate engagement, and should be considered in both future design and trials evaluating these technologies.

JMIR Cardio 2022;6(1):e33366



The majority of people living with heart failure are elderly and have 3-5 severe comorbidities such as diabetes mellitus, chronic obstructive pulmonary disease, and ischemic heart disease [1]. The 1-year mortality from hospital admission is more than 30%, and 30%-50% of people living with heart failure are re-admitted to hospital each year [2]. Heart failure often leads to reduced mobility and shortness of breath on exertion. These manifestations can affect day-to-day living, reduce independence, increase the risk of falls, and impair access to health services [3].

Remote care technologies have the potential to benefit people living with heart failure by bridging the gap between clinical visits [4]. These technologies enable frequent observations in the home and give access to tools to aid self-care [5]. In the long term, personalized, community-based technology may reduce the burden of health care appointments and increase effective management of symptoms. Empowering self-management can enable early detection of health issues and promote interventions to prevent hospital admission [6]. However, only technologies which engage the users (including carers) can produce positive change. Engagement depends on the design and suitability of the technology to the cohort. Similarly, effective interventions must also engage health care professionals [7].

Non-engagement with health care technologies is widespread [7]. A meta-ethnography by Greenhalgh and colleagues [8] investigating technology-supported health programs generated the Non-adoption, Abandonment, Scale-up, Spread, Sustainability (NASSS) framework, which identifies 7 critical domains that impact whether a technology is adopted. These are the condition, the technology, the value proposition, the staff involved, the organization, the social/institutional context, and the interaction between domains. This approach gives valuable insights into the dynamic between users and medical technologies in general, but a more nuanced approach is needed to understand interventions in patients with heart failure having complex needs.

For instance, in the review by Simblett et al [7] regarding barriers to engagement of remote technology, a large proportion of dropouts from remote care studies were attributed to usability issues such as technical difficulties, over 5 times more than those that dropped out due to issues with their health status. In an unselected population this may be representative; however, in the cohort of patients with heart failure, patient barriers and priorities for engagement can be very different. Elderly populations are likely to have impairments in vision, dexterity, and hearing, which can impact usability in different ways. In addition, the higher incidence of comorbidities may mean more hospital admissions and variability in their health status. Therefore, a focused approach is needed to determine which factors play a greater role for engagement with this particular patient group so that future interventions can target these areas and incorporate the design elements that matter most. Otherwise, there is a risk of creating an intervention that is not well suited to the target population, which can lead to these large numbers of dropouts and disengagements.

The aim of this systematic review was to explore the use of remote distance technologies in heart failure. We focused on the effect that remote distance technologies have on engagement with care and investigated the perspectives of people living with heart failure, carers, and health care professionals interacting with the same technology. Our aim was to identify the unique contexts and issues found in this cohort, focusing on factors that influence adoption of, engagement with, and use of remote care interventions by people living with heart failure; and factors that affect the engagement of clinical staff with remote care intervention. We also considered which of the tensions identified in clinical trials became evident from examining patient and staff personal experiences.

Study Selection

Relevant studies published between January 1, 1990, and September 19, 2020, were identified from EMBASE, Ovid MEDLINE, PubMed, Cochrane Library, and Scopus (see Multimedia Appendix 1 for search history). Reference lists of all identified studies were manually searched for relevant publications.

Inclusion Criteria

  • Primary studies using qualitative methodologies to collect or analyze data—defined as studies where participants can enter free-text comments or answer at least one open interview question.
  • Studies that included patients diagnosed with adult chronic heart failure of all severities or their carers or the health care professionals involved in their care.
  • Studies describing remote care programs—defined as any intervention accessible from the patient’s home or local community, which provides the patient with education, assessment, investigation results, or otherwise replaces a service that would normally be offered within a formal clinical setting.

Exclusion Criteria

  • Nonprimary studies.
  • Studies with no qualitative element in the collection or analysis of data.
  • Studies that did not include patients with adult chronic heart failure and neither their carers nor health care professionals.
  • Non-English studies
  • Interventions already established for heart failure care within national/international guidelines.
  • Interventions involving implantable devices where user engagement is not a factor.

A random sample of 30% of abstracts were screened independently by 2 reviewers (AA and JD). The remaining abstracts were selected by a single reviewer (AA) using the double-screened sample as a foundation. Full texts were assessed for eligibility by 2 reviewers (AA and JD). Study characteristics were extracted using the Enhancing Transparency in Reporting the Synthesis of Qualitative Research (ENTREQ) statement [9], and with guidance from the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist [10]. We used the PROGRESS-Plus framework (ie, place of residence, race, occupation, gender, religion, education, socioeconomic status, social capital, plus personal characteristics/relationships/time-dependent variables) [11] to create a more comprehensive review guideline for identifying health inequalities and characterizing populations [12]. Quality assessment of these studies ensured that the findings extracted were reliable and that bias was minimized [13]. Methodology of included literature was assessed using The National Institute for Health and Care Excellence (NICE) Quality Appraisal Checklist for Qualitative Studies [14-16]. Studies were designated excellent (++), good (+), or poor (–) quality. Rather than excluding the poor-quality studies, we opted for a more inclusive study design to better encompass the range of user experiences and reach a saturation point for first-order codes.

Thematic Synthesis Methodology

We applied the thematic analysis model of qualitative evidence synthesis [17]. Textual data from the Results or Findings sections of publications were extracted verbatim (EPPI-Reviewer 4, version and line-by-line coding was used to generate first-order codes. First-order codes were grouped under second-order codes—umbrella terms that bridged commonalities between multiple first-order codes. Second-order codes were created independently by AA, JD, and a public advisor group, and the final list was determined by iterative consensus. Third-order, “core concept” codes were created based on patterns and inferences observed throughout the review. They involved a process of reflection and reiteration by the authors on all previously extracted data and codes within the context of the research question being asked, that is, what are the factors affecting engagement.

Core concepts and second-order codes were finally consolidated to create overarching themes, which were designed to be universal across patients, health care professionals, and carers, and to encapsulate all aspects of experiential user engagement. A patient participation group made up of several patients with heart failure was vital in establishing meaningful nomenclature for each of the themes and what they encompass. The title of each theme would then be intended to be interpreted in the context of its underlying description, based on its included secondary codes and core concepts, making it a unique thematic construction. At all stages of the review, the authors reflected on their own background and position and how it would affect the design, analysis, and interpretation of the research conducted [17].

Study Selection

Our initial search criteria found 5944 matching studies, of which 798 were duplicates and 4869 did not match the inclusion criteria based on their title and abstract. A further 230 studies failed to meet the inclusion criteria when full texts were screened. The remaining 47 studies were included in the full review, in addition to 5 studies referenced by these papers (Figure 1). The PRISMA checklist (Multimedia Appendix 2) was used to analyze the included studies. Fifty-two studies were thus included in the final review (see Multimedia Appendix 3 for characteristics of studies table). The rationale for quality assessment scores is described in Multimedia Appendix 4.

Figure 1. PRISMA flow diagram for included studies. PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
View this figure

A total of 33 studies covered remote monitoring systems used to review a patient’s status from their home [18-50]. Clinical decision tools, used to complement clinician management plans, were trialed in 4 studies [23,51-53]. As many as 4 studies involved patient health information platforms [23,54-56], 5 online patient self-management tools via an online portal, [23,56-59], and 4 educational tools delivering information for self-care through various means [23,58,60,61]. Community remote care, involving occasional home visits by nurses, comprised 2 studies [62,63], as did telephone consultations [48,64]. There was 1 instance each of a peer-support system [65] and a pharmacy-based consultation [66]; 2 studies delivered a concept of a remote care intervention and gathered opinions based on a theoretical design [46,67].

People living with heart failure, carers, and health care professionals are not equally represented in these sources: 9 studies included perspectives from health care professionals alone [31,37,39,44,45,48,51-53]; 29 included patients alone [18,19,21-24,28-30,33-35,38,40,41,46,47,54,56,57,59,60,62-65,67-69]; 2 included patients and their carers [20,58]; 9 included patients and health care professionals [26,32,36,42,43,49,55,61,66]; and 3 included health care professionals, patients, and carers [25,27,50].



We extracted 110 separate primary descriptive codes from the Results sections of the included studies (Multimedia Appendix 5). We used a combination of grouping and reflective review to categorize these themes, which resulted in 30 secondary codes (Multimedia Appendix 6) and 13 core concepts. Secondary codes and core concepts were synthesized to give 5 themes that seek to encapsulate all aspects of user engagement: Convenience, Ease of Use, Education, Clinical Care, and Communication (Figure 2). Although themes are applicable across patients, carers, and health care professionals, they were associated with different secondary codes and core concepts.

Figure 2. Final thematic map representing the key points of each engagement theme and what they mean to each user group.
View this figure

Convenience is any aspect of the intervention that makes the user’s life easier than the preintervention stage, by either time-saving or giving them the freedom to do more, or enhances their comfort in the current environment.

Positive patient experiences included those where the intervention had saved the patient travel time, or opened up new options for the patient to manage their care at home, and be comfortable in their home environment [20]. Negative experiences were associated with extra work, for example, in a complex self-monitoring system that takes time away from daily activities [35]. Loss of control was also commonly reported as a negative experience [35]:

I think you feel like you're not in control of your life...I just felt that, well, it certainly wasn't for me, and to, from how he explained it, um, you tended to have to do your blood test every single day...I try to be a bit more relaxed and...I just felt it, it did put a bit more pressure on know, holidays or if I had to stay at my mum's, oh God, I've got to come home and do the machine.

For carers, positive aspects related to increasing options for managing care and increased freedom. Most carer experiences fell into negative codes regarding losing control of their tasks and increasing their workload and hence their stress. Throughout this theme, the concepts of workload, responsibility, and stress correlated with each other with regard to trialing a new intervention.

Positive experiences for health care professionals related to time-saving through faster and more efficient decision making. An example from Taylor and colleagues [37] showed how remote care technologies can help in understaffed situations:

We are being asked to see more patients with no additional resources.... How can we release a little bit of our capacity? Because our capacity is at absolute maximum all the time...I think telehealth helps from that point of view
[District Nurse 4, Site A]

Negative experiences related to the workload associated with implementing the intervention. After using the intervention, some health care professionals also reported that patients had become too dependent on the staff via the increased monitoring they provided [25]. Many clinical professionals were also reluctant to change as new methods were seen to restrict their preferred working pattern [39].

Ease of Use

This refers to the technical design elements of the intervention, and how user-friendly and accessible it was.

Positive patient experiences were associated with an easy-to-use intervention, with simple instructions and very few technical difficulties. Patient negative experiences involved devices that were intrusive, difficult to use, and had many technical difficulties. High costs and unreliability of the intervention made some technologies “less accessible.” For example, Agrell and colleagues [18] described a remote care home hub with integrated stethoscope and blood pressure cuff, which proved difficult for sick patients to use.

Positive carer experiences centered around technology that was easy to use (eg, had a clear user interface) and reliable. Much of the negative feedback from carers was related to the stress caused by technical malfunctions of the device [54].

Health care professionals were positive about interventions that were easy to administrate and explain to patients and staff, or provided information in an intuitive manner. They favored interventions that took less time and effort to install and establish into current practice, such as a system that integrates remote monitoring into existing electronic patient records [52]. Negative feedback from health care professionals was associated with high cost, lack of security, poor user interface, lack of training provision, and unreliable results. An added issue for health care professionals was the lack of support and technical expertise to deal with problems [44].


This relates to the ability of the intervention to provide appropriate, user-tailored information.

Positive patient experiences were reported in interventions that delivered self-care information in a simple, structured format, which was easy for patients to make use of. This helped increase their confidence in managing their condition, which in turn increased their engagement [55,64]. Negative patient experiences occurred where either useless or irrelevant information was given. This included remote care devices that give no immediate feedback, and websites or electronic health records that used medical jargon [55].

For carers, it was important to learn about the condition of the patient and understand disease progression. It was also important to access up-to-date information on the status of the patient. Remote interventions that made this easy for them generated positive feedback. When this information was difficult to access or mired in inaccessible jargon, it led to disengagement and discontentment [54].

For health care professionals, education centered on informative benefit that an intervention added to current practice. Positive experience involved interventions that provided guidelines or clinical suggestions to aid in patient management. Clinicians also identified the educational benefit on self-care habits of patients [27]. Education provided through a remote care intervention raises the standard of clinical treatment and involves the patient in their own management. This continuous benefit is valued by health care professionals. By contrast, the negative experiences of education involved insufficient, irrelevant, or poor-quality clinical information [22]. An intervention that did not provide feedback or allow patients to be updated on their health status provides little motivation for using the device or improving self-care, and this had a knock-on effect on health care professionals’ perceptions of the usefulness of the device.

Clinical Care

This was a clearly defined theme that involved either improving or hindering the current clinical care provided for heart failure management. In most cases, the effects of this theme could be measured and observed in terms of patient outcome over the long term.

Patients valued interventions that supported better symptom control and improved confidence in their treatment management plan. They often commented that certain remote care interventions made them feel more “looked after” once they were aware that their measurements were being monitored from afar [22]. Negative experiences involved difficulty integrating the intervention into their daily lifestyle. Some patients felt that the intervention had no effect and that the technology was ultimately not needed [28].

Positive aspects for carers focused on them feeling more supported while using the intervention. The reassurance of extra clinical support helped them perform their task so that they felt they need not struggle alone [20,54]. Negative aspects of clinical care from carers were often found where patients already had sufficient care to meet their needs. Additionally, there were cases where the patients were too sick to be under an automated monitoring system and required personal supervision. Where carers mistrusted technology, its introduction was an extra source of stress [35].

For health care professionals, positive aspects involved being able to administer guideline-recommended management reliably and with less error. The ability to provide proactive treatment was also considered to be of great benefit. Often, increased monitoring frequency allowed the staff to identify deterioration more quickly and intervene earlier to prevent worsening outcomes. This improved the safety and quality of management and also fulfilled a “safety-netting” criteria that were found to be very valuable [22]. Remote care bridges the gap between primary and community care, where patients with heart failure are seen infrequently and may have periods of long stability, but may also be at risk of sudden deterioration. Negative aspects of clinical care for staff were equally impactful. They involved interventions that disrupted management, often by giving unreliable, incomplete, or false information, or simply made no difference to management decisions, or patient outcome. In the example by Sharma and colleagues [44], the remote care offered features such as home blood pressure monitoring, but did not provide enough information to assess for an infection. Health care providers perceived that some technology may hinder usual care in this way, and disrupt the current “face-to-face” care.


This theme encompasses the quality and frequency of interpersonal contact involved with the intervention.

When discussing communication, patients often referred to the frequency of contact with their nurse or doctor. In general, patients favored human contact to being monitored by a noninteractive device. Therefore, interventions that facilitated human contact appealed to patients [18,29]. Patients commented that greater human contact helped with their feelings of isolation and encouraged them to self-care. Remote care interventions that facilitated interaction with health care professionals were seen as valuable and worth the investment. Patients also liked interventions that could connect with their family and friends for further support. Some social interventions connected patients with each other and these showed widespread approval within this cohort of patients with heart failure. A chronic illness that reduces mobility often impacts the social activities of the patient and it is not surprising that this isolation can often be overlooked when considering these interventions [65].

Conversely, many of the remote care interventions led to reduced contact with physicians. Patients often missed the human contact element of regular clinic visits. Rather than having a device to provide information for them, they preferred being able to ask questions from staff members directly [18]. Without a human contact element, some felt more distant and perceived that something was lacking from the consultation. While remote care may obtain information efficiently, sometimes human contact is more reassuring and has a bigger impact on patient engagement.

For carers, “isolation” was predominantly related to the fear of leaving the patient on their own. The ability of remote care to mediate communication between both the patient and carer with health care professionals was seen as valuable and reassuring [26].

Health care professionals are often concerned with patient contact and with connection to specialists. Positive communication with other staff members is found in interventions that seamlessly connect multidisciplinary teams [37]. This in turn encouraged teamwork. Regarding staff to patient communication, health care professionals felt that frequent contact led to better awareness of the patient’s tendencies and hence earlier and better decisions on their care [27,31]. The increased availability of communication with patients via remote care was felt to increase trust between the staff and their patients and staff felt their patients were more open to them. This fosters a good clinical environment, especially in the management of chronic disease where familiarity with the patient is vital to detecting early deterioration. Increased communication opportunities also motivate both parties to continue to use the device, for reassurance and safety-netting. Negative staff experiences with communication included systems that did not connect with other team members or caused a disruption in teamwork. An example was a remote monitoring system that provided the general practitioner with extra information but was unable to send this information on to heart failure hospital consultants, and thus made referrals harder [22]. Staff also found that interventions that reduced interaction with patients received much less support from both parties. Reduced face-to-face time due to remote intervention created a sense of distance with the patient, which was concerning to some staff [22].

Health Inequalities


In our review of interventions and the populations they were used on, various health inequality issues became apparent: usability, patient selection criteria, demographic distribution, and the potential to increase inequality gap through implementation of remote care.


In a heart failure cohort, interventions need to be usable by patients who have visual impairments or problems with dexterity [43]. If the intervention is not designed to take this into account, it can exclude the patients that might have benefitted the most. Riley and colleagues [34] recount an example of one such patient who struggled to use a remote care device, leading to undue stress:

I keep losing the finger contact and because of my sight I have to search hard to find it and that unfortunately sends my blood pressure up and then I have to redo the test.

In addition to co-ordination and visual problems, some patients were unable to use a weighing device because it required them to remain standing for several minutes [26].

Patient Selection

In any interventional trial, inequality may arise between patients who do and do not have access to remote interventions. Clinical decisions on which patients are most suited to the intervention can be subjective. The weight monitoring intervention in the study by Johnston and Weatherburn [26] demonstrated this selective service based on clinician opinion, creating a defined cohort of highly monitored patients above those of current care. With limited resources, clinicians may set up their own criteria for prioritizing access to remote care based on need, as they do with other treatments [22]. Over a short intervention trial period, some clinicians felt that monitoring was not useful in patients they deemed stable. However, this may lead to situations where patients are only provided with remote monitoring once deterioration has occurred.

Demographic Representation

In intervention trials, the criteria for proving the new remote care intervention are often decided a priori by the research group. Studies considered here show a distinct preponderance to White ethnicity and male gender. In addition, there is generally poor reporting on religion, socioeconomic status, social capital, disability, and vulnerable groups. It is therefore difficult to assess the impact of remote care integration on more disparate groups within populations.

Widening the Inequality Gap

Some clinicians showed anxiety that technology could widen the health inequality gap. The study by Earnest and colleagues [55] highlighted concerns around how remote care would translate in areas with fewer resources. Many remote care interventions take the form of highly technologically advanced, expensive devices, which, while rich with clinical data, are prohibitive for widespread use in terms of costs and resource drain. It is worth noting the impact of these interventions on the rest of the clinical care of the population, whether it creates a wider benefit to the whole or simply allows greater monitoring for those who have the most care in the first place.

Principal Findings

Throughout the endeavor to define engagement with heart failure remote care by patients, carers, and staff, this review has encompassed a wide range of possible interventions and environments, exploring the qualitative experiences from a vast array of users until the point of saturation. The experiential discoveries have been organized into 5 overarching themes that can be applied to each user base: Convenience, Ease of Use, Education, Clinical Care, and Communication. These themes go some way to giving insight to designers of such technology on how to tailor their intervention and improve uptake to create a greater benefit for their intended users.

Recently, the advent of COVID-19 has led to the prioritization of remote care as a way of managing vulnerable populations such as these while reducing the risk of in-person exposure [70]. While technology usability models for remote care have been explored [7,8], there are yet many clinical studies specific to heart failure remote care that emphasize more research into engagement for this patient group [71,72]. Hence the themes generated in this review aid to bridge the gap between generic usability models and the heart failure population, helping to apply existing knowledge in a more personalized way for more engaging interventions [73].

Implementing Engagement Themes Into Technology Design

Each theme has its own technological implications in the design of an intervention depending on the user group. Adding “Convenience” to a remote care intervention involves improving comfort or saving time in a way that has a significant impact on their daily life. This means that the user should be able to carry out typical tasks, such as patient self-care or a clinician’s daily reviews, but in a more efficient manner or at a location more convenient for both parties.

Adding “Ease of Use” involves tailoring the experience with the user in mind. For elderly patients with heart failure, this means a simple and intuitive interface, requiring little or no technical knowledge. The closer the intervention reflects normal daily activity, the more seamless the transition toward its use. For clinicians, interventions should provide the necessary information for clinical decision making without unnecessary complexity. Technical difficulties are some of the most important barriers to effective implementations of remote care. Technologies that are reliable, easy to install, and integrate into current systems will earn the trust of staff.

Improving the “Education” of an intervention involves allowing it to provide information in a relatable way that is specific to the user’s situation, for example, to aid patient self-care and to increase their sense of control and self-reliance [27]. For health care professionals, the intervention should help update staff on current guidelines and recommended practice. Information should be understandable and usable by all health care professionals from community nurses to heart failure specialists.

Adding the “Clinical Care” component for patients means improving the perception of greater care. These perceptions are enhanced by frequent feedback, contact with clinicians, and impactful changes to their lifestyle, which foster a sense of being “well looked after” [65]. For health care staff, clinical care relates to improving health outcomes. A robust series of clinical trials that demonstrate the clinical value of an intervention is vital to inform and justify implementation. Beyond this, clinicians will prioritize the interventions that address an unmet need in service provision. Overall, clinical care is a vital part of any remote care assessment, and must be assessed in the context of patients’ needs, their current care, and the resources available to the staff.

Improving “Communication” is achieved by allowing further communication to take place, both between patients and staff and between peers, fostering a sense of more involved care [25]. Additionally, many patients value the option of having remote care to connect to their support network. Health care professionals highly value mechanisms in remote care that allow multiple specialties and services to integrate together and communicate. Many people living with heart failure are elderly and have multiple comorbidities necessitating a complex regimen of medication and outpatient health services to maintain their well-being. Too often, the flow of patient information between these services becomes lost or confused. This is an obstacle to communication that well-designed remote care can help overcome.

Implementation of Remote Care Within Clinical Trial Design

Based on our observations, we noted certain practices that need to be carefully considered during the design of remote care evaluation studies. Recorded experiences from users contrast and contradict each other even within the same study, and thus a sufficiently large sample size is important. The intervention must be distributed across a diverse ethnographical population, including those with poor literacy, and those with disabilities as well as diversity in gender and socioeconomic backgrounds. Selective inclusion or exclusion of patients with extremes of the condition may lead to bias. In our view, it is important to consider both the patient and health care professionals’ viewpoint in the same study. This gives a good estimate as to how readily staff were able to use the intervention, and the impact that it has on their current care. It is also important to know where in the established care pathway the new technological intervention will sit; this affects who administers the device, to which cohort, and through which electronic systems. Interventions that do not consider the gaps in current care may cause disruption for staff and create additional steps that do not complement nor aid their current care environment [35]. Finally, in a few cases, patients and staff had negative experiences with the third-party support provided around the device’s use. This could be improved with effective trial planning.


Limitations in Methodology

We have attempted to mitigate bias by double-coding, cross checking, and using grounded theory processes to construct a “blank-slate” perspective. However, in line-by-line coding some lines may have a double-meaning, fitting into multiple codes, and decisions on coding are subjective.

Limitations in Studies

The studies included were of a wide range of different methodological and analytical qualities. First-order codes from poor-quality studies may be subject to greater variability than high-quality studies. However, we included studies of all quality to avoid missing useful information and to achieve saturation. The wide range of study sizes may skew the experiences toward those that are mentioned in small sample sizes, as it increases the frequency of appearance of the related codes. We have listed the sample size of each study in the characteristics of studies table (Multimedia Appendix 1).

Limitations in Phenomena Studied

The researcher’s reflexivity plays a role in the overall generation of the themes and can lead to first-degree bias. This process happens both inductively and intuitively and it cannot truly be distanced from preconceptions and beliefs of the reviewers themselves.


Overall, the 5 themes generated by this study overlap in many ways to create an engaging technology. Successful remote care interventions interact meaningfully with the user and thus instigate a change in self-care or in the working practice of health care professionals. Increased communication due to a remote care intervention leads to a perception of greater care from the patient. This in turn leads to improved feedback to the clinician and an improved perception of the devices’ educational and clinical benefits. Likewise, convenience can be an important component contributing to ease of use. Successful and engaging interventions should combine these 5 elements into their design to increase the engagement of their users and lead to a greater benefit in this elderly comorbid population that needs this support the most.


This research was funded by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North West Coast. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health. We acknowledge the invaluable input from the heart failure patient group at Liverpool Heart and Chest Hospital in naming the secondary codes and themes.

Authors' Contributions

AAN proposed the study concept; designed the methodology; conducted literature searches; and contributed to the data entry, interpretation of results, and drafting the manuscript. JD contributed to data entry, interpretation and analysis of results, and critical appraisal of the manuscript. KAS contributed to the layout of the draft and development of multimedia appendices as well as editing the manuscript for publication. All authors critically assessed and approved the final manuscript.

Conflicts of Interest

MP has received partnership funding for the following: MRC Clinical Pharmacology Training Scheme (co-funded by MRC and Roche, UCB, Eli Lilly, and Novartis); Joint PhD funding from EPSRC and Astra-Zeneca; and grant funding from VistaGen Therapeutics. He also has unrestricted educational grant support for the UK Pharmacogenetics and Stratified Medicine Network from Bristol-Myers Squibb. He has developed an HLA genotyping panel with MC Diagnostics, but does not benefit financially from this. MP is also part of the IMI Consortium ARDAT. None of these funding sources were used for the work presented in this paper.

Multimedia Appendix 1

Search history for systematic review.

DOC File , 106 KB

Multimedia Appendix 2

PRISMA 2020 checklist.

DOC File , 77 KB

Multimedia Appendix 3

Characteristics of studies included with the first order codes generated from each study.

DOCX File , 101 KB

Multimedia Appendix 4

NICE quality appraisal.

DOC File , 238 KB

Multimedia Appendix 5

First-order codes identified in studies included in systematic review, with associated descriptions.

DOC File , 201 KB

Multimedia Appendix 6

Synthesis of second-order codes.

DOC File , 241 KB

  1. Cardiovascular disease statistics, 2015. British Heart Foundation.   URL: [accessed 2018-05-01]
  2. National Heart Failure Audit April 2013 - March 2014. National Institute for Cardiovascular Outcomes Research.   URL: [accessed 2018-05-01]
  3. Dhalwani NN, Fahami R, Sathanapally H, Seidu S, Davies MJ, Khunti K. Association between polypharmacy and falls in older adults: a longitudinal study from England. BMJ Open 2017 Oct 16;7(10):e016358 [FREE Full text] [CrossRef] [Medline]
  4. Gustafsson F, Arnold JMO. Heart failure clinics and outpatient management: review of the evidence and call for quality assurance. Eur Heart J 2004 Sep 01;25(18):1596-1604. [CrossRef] [Medline]
  5. Masterson Creber RM, Hickey KT, Maurer MS. Gerontechnologies for Older Patients with Heart Failure: What is the Role of Smartphones, Tablets, and Remote Monitoring Devices in Improving Symptom Monitoring and Self-Care Management? Curr Cardiovasc Risk Rep 2016 Oct 15;10(10):30 [FREE Full text] [CrossRef] [Medline]
  6. Evangelista LS, Lee J, Moore AA, Motie M, Ghasemzadeh H, Sarrafzadeh M, et al. Examining the effects of remote monitoring systems on activation, self-care, and quality of life in older patients with chronic heart failure. J Cardiovasc Nurs 2015;30(1):51-57 [FREE Full text] [CrossRef] [Medline]
  7. Simblett S, Greer B, Matcham F, Curtis H, Polhemus A, Ferrão J, et al. Barriers to and Facilitators of Engagement With Remote Measurement Technology for Managing Health: Systematic Review and Content Analysis of Findings. J Med Internet Res 2018 Jul 12;20(7):e10480 [FREE Full text] [CrossRef] [Medline]
  8. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies. J Med Internet Res 2017 Nov 01;19(11):e367 [FREE Full text] [CrossRef] [Medline]
  9. Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol 2012 Nov 27;12(1):181 [FREE Full text] [CrossRef] [Medline]
  10. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007 Dec;19(6):349-357. [CrossRef] [Medline]
  11. O'Neill J, Tabish H, Welch V, Petticrew M, Pottie K, Clarke M, et al. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. J Clin Epidemiol 2014 Jan 01;67(1):56-64. [CrossRef] [Medline]
  12. Oliver S, Kavanagh J, Caird J, Lorenc T, Oliver K, Harden A. Health promotion, inequalities and young people's health: a systematic review of research. PPI-Centre, Social Science Research Unit, Institute of Education, University of London. 2008 Oct 01.   URL: [accessed 2022-03-14]
  13. Carroll C, Booth A, Lloyd-Jones M. Should we exclude inadequately reported studies from qualitative systematic reviews? An evaluation of sensitivity analyses in two case study reviews. Qual Health Res 2012 Oct 03;22(10):1425-1434. [CrossRef] [Medline]
  14. National Institute for Health and Care Excellence. Methods for the development of NICE public health guidance (third edition): Process and methods. NICE.   URL: https:/​/www.​​article/​pmg4/​chapter/​appendix-h-quality-appraisal-checklist-qualitative-studies [accessed 2018-05-01]
  15. Critical Appraisal Skills Programme. CASP Checklist: 10 questions to help you make sense of a Qualitative research. CASP. 2019.   URL: [accessed 2022-03-14]
  16. Brunton L, Bower P, Sanders C. The Contradictions of Telehealth User Experience in Chronic Obstructive Pulmonary Disease (COPD): A Qualitative Meta-Synthesis. PLoS One 2015 Oct 14;10(10):e0139561 [FREE Full text] [CrossRef] [Medline]
  17. Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 2008 Jul 10;8:45 [FREE Full text] [CrossRef] [Medline]
  18. Agrell H, Dahlberg S, Jerant AF. Patients' perceptions regarding home telecare. Telemed J E Health 2000 Dec;6(4):409-415. [CrossRef] [Medline]
  19. Bartlett YK, Haywood A, Bentley CL, Parker J, Hawley MS, Mountain GA, et al. The SMART personalised self-management system for congestive heart failure: results of a realist evaluation. BMC Med Inform Decis Mak 2014 Nov 25;14(1):109 [FREE Full text] [CrossRef] [Medline]
  20. Dinesen B, Nøhr C, Andersen SK, Sejersen H, Toft E. Under surveillance, yet looked after: telehomecare as viewed by patients and their spouse/partners. Eur J Cardiovasc Nurs 2008 Sep 01;7(3):239-246. [CrossRef] [Medline]
  21. Dubois A, Santos-Eggimann B. Evaluation of patients' satisfaction with hospital-at-home care. Eval Health Prof 2001 Mar 01;24(1):84-98. [CrossRef] [Medline]
  22. Fairbrother P, Ure J, Hanley J, McCloughan L, Denvir M, Sheikh A, Telescot programme team. Telemonitoring for chronic heart failure: the views of patients and healthcare professionals - a qualitative study. J Clin Nurs 2014 Jan 04;23(1-2):132-144. [CrossRef] [Medline]
  23. Finkelstein J, Wood J. Implementing home telemanagement of congestive heart failure using Xbox gaming platform. 2011 Aug 30 Presented at: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 30 August to 3 September, 2011; Boston, MA.
  24. Gund A, Ekman I, Lindecrantz K, Sjoqvist B, Staaf E, Thorneskold N. Design evaluation of a home-based telecare system for chronic heart failure patients. 2008 Presented at: 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; August 20-24, 2008; Vancouver, BC.
  25. Hunting G, Shahid N, Sahakyan Y, Fan I, Moneypenny CR, Stanimirovic A, et al. A multi-level qualitative analysis of Telehomecare in Ontario: challenges and opportunities. BMC Health Serv Res 2015 Dec 09;15:544 [FREE Full text] [CrossRef] [Medline]
  26. Johnston G, Weatherburn G. Automated weight monitoring in chronic heart failure: the excluded majority. J Telemed Telecare 2010 May 28;16(4):190-192. [CrossRef] [Medline]
  27. Kenealy TW, Parsons MJG, Rouse APB, Doughty RN, Sheridan NF, Hindmarsh JKH, et al. Telecare for diabetes, CHF or COPD: effect on quality of life, hospital use and costs. A randomised controlled trial and qualitative evaluation. PLoS One 2015 Mar 13;10(3):e0116188 [FREE Full text] [CrossRef] [Medline]
  28. Lind L, Karlsson D. Telehealth for "the digital illiterate"--elderly heart failure patients experiences. Stud Health Technol Inform 2014;205:353-357. [Medline]
  29. Lyngå P, Fridlund B, Langius-Eklöf A, Bohm K. Perceptions of transmission of body weight and telemonitoring in patients with heart failure? Int J Qual Stud Health Well-being 2013 Dec 16;8:21524 [FREE Full text] [CrossRef] [Medline]
  30. Nanevicz T, Piette J, Zipkin D, Serlin M, Ennis S, De Marco T, et al. The feasibility of a telecommunications service in support of outpatient congestive heart failure care in a diverse patient population. Congest Heart Fail 2000 May;6(3):140-145 [FREE Full text] [CrossRef] [Medline]
  31. Odeh B, Kayyali R, Nabhani-Gebara S, Philip N. Implementing a telehealth service: nurses' perceptions and experiences. Br J Nurs 2014 Nov 26;23(21):1133-1137. [CrossRef] [Medline]
  32. Paget T, Jones C, Davies M, Evered C, Lewis C. Using home telehealth to empower patients to monitor and manage long term conditions. Nurs Times 2010 Nov 16-22;106(45):17-19. [Medline]
  33. Rahimpour M, Lovell NH, Celler BG, McCormick J. Patients' perceptions of a home telecare system. Int J Med Inform 2008 Jul;77(7):486-498. [CrossRef] [Medline]
  34. Riley JP, Gabe JP, Cowie MR. Does telemonitoring in heart failure empower patients for self-care? A qualitative study. J Clin Nurs 2013 Sep;22(17-18):2444-2455. [CrossRef] [Medline]
  35. Sanders C, Rogers A, Bowen R, Bower P, Hirani S, Cartwright M, et al. Exploring barriers to participation and adoption of telehealth and telecare within the Whole System Demonstrator trial: a qualitative study. BMC Health Serv Res 2012 Jul 26;12:220 [FREE Full text] [CrossRef] [Medline]
  36. Svagård I, Austad HO, Seeberg T, Vedum J, Liverud A, Mathiesen BM, et al. A usability study of a mobile monitoring system for congestive heart failure patients. Stud Health Technol Inform 2014;205:528-532. [Medline]
  37. Taylor J, Coates E, Brewster L, Mountain G, Wessels B, Hawley MS. Examining the use of telehealth in community nursing: identifying the factors affecting frontline staff acceptance and telehealth adoption. J Adv Nurs 2015 Feb;71(2):326-337. [CrossRef] [Medline]
  38. Whitten P, Bergman A, Meese MA, Bridwell K, Jule K. St. Vincent's Home telehealth for congestive heart failure patients. Telemed J E Health 2009 Mar 17;15(2):148-153. [CrossRef] [Medline]
  39. Whitten P, Collins B. Nurse reactions to a prototype home telemedicine system. J Telemed Telecare 1998 Dec 02;4 Suppl 1(1_suppl):50-52. [CrossRef] [Medline]
  40. Woodend AK, Sherrard H, Fraser M, Stuewe L, Cheung T, Struthers C. Telehome monitoring in patients with cardiac disease who are at high risk of readmission. Heart Lung 2008 Jan 01;37(1):36-45. [CrossRef] [Medline]
  41. Hall AK, Dodd V, Harris A, McArthur K, Dacso C, Colton LM. Heart failure patients' perceptions and use of technology to manage disease symptoms. Telemed J E Health 2014 Apr;20(4):324-331 [FREE Full text] [CrossRef] [Medline]
  42. Seto E, Leonard KJ, Cafazzo JA, Barnsley J, Masino C, Ross HJ. Perceptions and experiences of heart failure patients and clinicians on the use of mobile phone-based telemonitoring. J Med Internet Res 2012 Feb 10;14(1):e25 [FREE Full text] [CrossRef] [Medline]
  43. Seto E, Leonard KJ, Masino C, Cafazzo JA, Barnsley J, Ross HJ. Attitudes of heart failure patients and health care providers towards mobile phone-based remote monitoring. J Med Internet Res 2010 Nov 29;12(4):e55 [FREE Full text] [CrossRef] [Medline]
  44. Sharma U, Barnett J, Clarke M. Clinical users' perspective on telemonitoring of patients with long term conditions: understood through concepts of Giddens's structuration theory & consequence of modernity. Stud Health Technol Inform 2010;160(Pt 1):545-549. [Medline]
  45. Sharma U, Clarke M. Nurses' and community support workers' experience of telehealth: a longitudinal case study. BMC Health Serv Res 2014 Apr 10;14:164 [FREE Full text] [CrossRef] [Medline]
  46. Cajita MI, Hodgson NA, Lam KW, Yoo S, Han H. Facilitators of and Barriers to mHealth Adoption in Older Adults With Heart Failure. Comput Inform Nurs 2018 Aug;36(8):376-382 [FREE Full text] [CrossRef] [Medline]
  47. Grace SL, Taherzadeh G, Jae Chang IS, Boger J, Arcelus A, Mak S, et al. Perceptions of seniors with heart failure regarding autonomous zero-effort monitoring of physiological parameters in the smart-home environment. Heart Lung 2017 Jul 01;46(4):313-319. [CrossRef] [Medline]
  48. Heckemann B, Wolf A, Ali L, Sonntag SM, Ekman I. Discovering untapped relationship potential with patients in telehealth: a qualitative interview study. BMJ Open 2016 Mar 02;6(3):e009750 [FREE Full text] [CrossRef] [Medline]
  49. Seto E, Morita PP, Tomkun J, Lee TM, Ross H, Reid-Haughian C, et al. Implementation of a Heart Failure Telemonitoring System in Home Care Nursing: Feasibility Study. JMIR Med Inform 2019 Jul 26;7(3):e11722 [FREE Full text] [CrossRef] [Medline]
  50. Seuren LM, Wherton J, Greenhalgh T, Cameron D, A'Court C, Shaw SE. Physical Examinations via Video for Patients With Heart Failure: Qualitative Study Using Conversation Analysis. J Med Internet Res 2020 Feb 20;22(2):e16694 [FREE Full text] [CrossRef] [Medline]
  51. Abidi SR, Stewart S, Shepherd M, Abidi SSR. Usability evaluation of family physicians' interaction with the Comorbidity Ontological Modeling and ExecuTion System (COMET). Stud Health Technol Inform 2013;192:447-451. [Medline]
  52. Green CJ, Fortin P, Maclure M, Macgregor A, Robinson S. Information system support as a critical success factor for chronic disease management: Necessary but not sufficient. Int J Med Inform 2006 Dec;75(12):818-828. [CrossRef] [Medline]
  53. Leslie SJ, Hartswood M, Meurig C, McKee SP, Slack R, Procter R, et al. Clinical decision support software for management of chronic heart failure: development and evaluation. Comput Biol Med 2006 May;36(5):495-506. [CrossRef] [Medline]
  54. Barron J, Finkelstein J. Feasibility of providing personalized health information to older adults and their caregivers. 2013 Presented at: 2013 IEEE Point-of-Care Healthcare Technologies (PHT); January 16-18, 2013; Bangalore, Karnataka, India p. 232-235.
  55. Earnest MA, Ross SE, Wittevrongel L, Moore LA, Lin C. Use of a Patient-Accessible Electronic Medical Record in a Practice for Congestive Heart Failure: Patient and Physician Experiences. J Am Med Inform Assoc 2004 Sep 01;11(5):410-417. [CrossRef]
  56. Lundgren J, Andersson G, Dahlström Ö, Jaarsma T, Köhler AK, Johansson P. Internet-based cognitive behavior therapy for patients with heart failure and depressive symptoms: A proof of concept study. Patient Educ Couns 2015 Aug;98(8):935-942. [CrossRef] [Medline]
  57. Payne AY, Surikova J, Liu S, Ross H, Mechetiuc T, Nolan RP. Usability Testing of an Internet-Based e-Counseling Platform for Adults With Chronic Heart Failure. JMIR Hum Factors 2015 May 08;2(1):e7 [FREE Full text] [CrossRef] [Medline]
  58. Akyar I, Dionne-Odom JN, Bakitas MA. Using Patients and Their Caregivers Feedback to Develop ENABLE CHF-PC: An Early Palliative Care Intervention for Advanced Heart Failure. J Palliat Care 2019 Apr 28;34(2):103-110. [CrossRef] [Medline]
  59. Lundgren J, Johansson P, Jaarsma T, Andersson G, Kärner Köhler A. Patient Experiences of Web-Based Cognitive Behavioral Therapy for Heart Failure and Depression: Qualitative Study. J Med Internet Res 2018 Sep 05;20(9):e10302 [FREE Full text] [CrossRef] [Medline]
  60. Selman L, McDermott K, Donesky D, Citron T, Howie-Esquivel J. Appropriateness and acceptability of a Tele-Yoga intervention for people with heart failure and chronic obstructive pulmonary disease: qualitative findings from a controlled pilot study. BMC Complement Altern Med 2015 Feb 07;15:21 [FREE Full text] [CrossRef] [Medline]
  61. Strömberg A, Ahlén H, Fridlund B, Dahlström U. Interactive education on CD-ROM—a new tool in the education of heart failure patients. Patient Education and Counseling 2002 Jan;46(1):75-81. [CrossRef]
  62. Näsström LM, Idvall EA, Strömberg AE. Heart failure patients' descriptions of participation in structured home care. Health Expect 2015 Oct 21;18(5):1384-1396 [FREE Full text] [CrossRef] [Medline]
  63. Young B, Purden M, Sauvé N, Dufour L, Common C. A "basket of care" for heart failure patients managing at home: evaluating a community-based nursing intervention from a patient's perspective. Can J Cardiovasc Nurs 2008;18(4):10-19. [Medline]
  64. Bekelman DB, Hooker S, Nowels CT, Main DS, Meek P, McBryde C, et al. Feasibility and acceptability of a collaborative care intervention to improve symptoms and quality of life in chronic heart failure: mixed methods pilot trial. J Palliat Med 2014 Feb;17(2):145-151 [FREE Full text] [CrossRef] [Medline]
  65. Heisler M, Halasyamani L, Resnicow K, Neaton M, Shanahan J, Brown S, et al. "I am not alone": the feasibility and acceptability of interactive voice response-facilitated telephone peer support among older adults with heart failure. Congest Heart Fail 2007 May;13(3):149-157 [FREE Full text] [CrossRef] [Medline]
  66. Lowrie R, Johansson L, Forsyth P, Bryce SL, McKellar S, Fitzgerald N. Experiences of a community pharmacy service to support adherence and self-management in chronic heart failure. Int J Clin Pharm 2014 Feb 30;36(1):154-162. [CrossRef] [Medline]
  67. Zulman DM, Jenchura EC, Cohen DM, Lewis ET, Houston TK, Asch SM. How Can eHealth Technology Address Challenges Related to Multimorbidity? Perspectives from Patients with Multiple Chronic Conditions. J Gen Intern Med 2015 Aug;30(8):1063-1070 [FREE Full text] [CrossRef] [Medline]
  68. Chantler T, Paton C, Velardo C, Triantafyllidis A, Shah SA, Stoppani E, et al. Creating connections - the development of a mobile-health monitoring system for heart failure: Qualitative findings from a usability cohort study. Digit Health 2016 Oct 10;2:2055207616671461 [FREE Full text] [CrossRef] [Medline]
  69. Dang S, Karanam C, Gómez-Orozco C, Gómez-Marín O. Mobile Phone Intervention for Heart Failure in a Minority Urban County Hospital Population: Usability and Patient Perspectives. Telemed J E Health 2017 Jul;23(7):544-554. [CrossRef] [Medline]
  70. Charman SJ, Velicki L, Okwose NC, Harwood A, McGregor G, Ristic A, et al. Insights into heart failure hospitalizations, management, and services during and beyond COVID-19. ESC Heart Fail 2021 Feb 24;8(1):175-182 [FREE Full text] [CrossRef] [Medline]
  71. Dorsch MP, Farris KB, Rowell BE, Hummel SL, Koelling TM. The Effects of the ManageHF4Life Mobile App on Patients With Chronic Heart Failure: Randomized Controlled Trial. JMIR Mhealth Uhealth 2021 Dec 07;9(12):e26185 [FREE Full text] [CrossRef] [Medline]
  72. Athilingam P, Jenkins BA, Zumpano H, Labrador MA. "Mobile technology to improve heart failure outcomes: A proof of concept paper". Appl Nurs Res 2018 Feb;39:26-33. [CrossRef] [Medline]
  73. MacKinnon GE, Brittain EL. Mobile Health Technologies in Cardiopulmonary Disease. Chest 2020 Mar 01;157(3):654-664 [FREE Full text] [CrossRef] [Medline]

COREQ: Consolidated Criteria for Reporting Qualitative Research
ENTREQ statement: Enhancing Transparency in Reporting the Synthesis of Qualitative Research statement
NASSS framework: Non-adoption, Abandonment, Scale-up, Spread, Sustainability framework
NICE: The National Institute for Health and Care Excellence
PROGRESS-Plus framework: place of residence, race, occupation, gender, religion, education, socioeconomic status, social capital, plus personal characteristics/relationships/time-dependent variables

Edited by A Mavragani; submitted 05.09.21; peer-reviewed by L Ali, T Murphy; comments to author 04.12.21; revised version received 18.12.21; accepted 08.01.22; published 06.04.22


©Ahmed Al-Naher, Jennifer Downing, Kathryn A Scott, Munir Pirmohamed. Originally published in JMIR Cardio (, 06.04.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cardio, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.