RESEARCH ARTICLE
Patients at the Centre: Methodological Considerations for Evaluating Evidence from Health Interventions Involving Patients Use of Web-Based Information Systems
Elizabeth Cummings*, Paul Turner
Article Information
Identifiers and Pagination:
Year: 2010Volume: 4
First Page: 188
Last Page: 194
Publisher Id: TOMINFOJ-4-188
DOI: 10.2174/1874431101004010188
Article History:
Received Date: 3/8/2009Revision Received Date: 8/1/2010
Acceptance Date: 12/1/2010
Electronic publication date: 15/9/2010
Collection year: 2010
open-access license: This is an open access article licensed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted, non-commercial use, distribution and reproduction in any medium, provided the work is properly cited.
Abstract
Building an evidence base for healthcare interventions has long been advocated as both professionally and ethically desirable. By supporting meaningful comparison amongst different approaches, a good evidence base has been viewed as an important element in optimising clinical decision-making and the safety and quality of care. Unsurprisingly, medical research has put considerable effort into supporting the development of this evidence base, and the randomised controlled trial has become the dominant methodology. Recently however, a body of research has begun to question, not just this methodology per se, but also the extent to which the evidence it produces may marginalise individual patient experiences, priorities and perceptions.
Simultaneously, the widespread adoption and utilisation of information systems (IS) in health care has also prompted initiatives to develop a stronger base of evidence about their impacts. These calls have been stimulated both by numerous system failures and research expressing concerns about the limitations of information systems methodologies in health care environments. Alongside the potential of information systems to produce positive, negative and unintended consequences, many measures of success, impact or benefit appear to have little to do with improvements in care, health outcomes or individual patient experiences.
Combined these methodological concerns suggest the need for more detailed examination. This is particularly the case, given the prevalence within contemporary clinical and IS discourses on health interventions advocating the need to put the ‘patient at the centre’ by engaging them in their own care and/or ‘empowering’ them through the use of information systems.
This paper aims to contribute to these on-going debates by focusing on the socio-technical processes by which patients’ interests and outcomes are measured, defined and evaluated within health interventions that involve them using web-based information systems. The paper outlines an integrated approach that aims to generate evidence about the impact of these types of health interventions that are meaningful at both individual patient and patient cohort levels.
INTRODUCTION
Health care systems in Australia and across the world are facing major challenges in how to maintain the provision of quality care for all, in a manner that is economically sustainable. Ageing populations and the related growth in the incidence of chronic disease are contributing significantly to spiralling health costs. As a consequence system affordability has now become a major driver underpinning initiatives aimed at health care reform. Currently, no health care system has developed a comprehensive response to meet these challenges but there is agreement and support for new models of service delivery that can enhance the efficiency and effectiveness of the system.
Considering the complexity of challenges faced, it is perhaps not surprising that there are different perspectives on what needs to be done and by whom, how, where and when. In this context, debates on health service delivery and quality of care have become increasingly prominent. Unfortunately, these debates have tended to be fragmented with a variety of approaches advocated each based on different measures and types of evidence to support their claims of value. It can however be argued that developing comprehensive solutions to the challenges faced requires a more integrated approach.
“Most quality improvement activities in the world are still largely a reflection of the specific beliefs of specific parties about the best way to improve patient care. Crossing borders among professional pride and self-regulation, external accountability, payer profit, organisational development, and pleasing and involving patients can help us overcome the obstacles to optimal medical care” [1].
In parallel with these developments, the widespread adoption and utilisation of information systems in health care has also been advocated as a major driver for stimulating the changes necessary to respond to the challenges faced.
“Both Australian and international studies support investment in health information management and information & communication technologies (IM&ICT) as a fundamental component of system reform… to improve health outcomes while helping to offset the health cost increases expected as a result of advances in medical technologies and pharmacology, and the inevitable aging of the population. In addition, improving the collection, management and dissemination of health information will boost administrators’ and policymakers’ ability to undertake informed, system-wide planning, resulting in more effective and efficient deployment of health resources” [2].
By opening up opportunities for increased information access, improved information delivery, update and evaluation, the economic case for more information systems is strong. However, this deployment also raises a number of socio-technical, clinical and legal challenges that have become increasingly evident as the implementation of more sophisticated solutions have met with mixed success or failed to generate their forecast benefits [3]. This has led to calls for the utilisation of more sophisticated socio-technical approaches to the design, implementation and evaluation of health projects involving information systems [4, 5].
While debates on the role and impacts of different approaches are likely to continue, from a methodological perspective finding an integrated way to meaningfully evaluate the different types of evidence they use, may contribute to supporting more effective solutions to be developed. To explore these methodological issues, this paper considers how contemporary clinical and IS evaluation discourses on health interventions approach patients. More specifically, the paper focuses on the socio-technical processes by which patients’ interests and outcomes are measured, defined and evaluated within health interventions that involve them using web-based information systems. The paper commences with a critical appraisal of the limitations of current clinical and IS evidence and evaluation processes of patients involved in these types of health interventions. The paper proceeds to outline an integrated approach that aims to generate evidence about the impact of these types of health interventions that are meaningful at both individual patient and patient cohort levels.
CLINICAL EVIDENCE AND EVALUATION
With the increasing incidence of chronic illness, health interventions encouraging people with a chronic disease to take a more active role in managing their own condition have become popular [6,7]. To date the evaluation of these interventions has focussed either on the impact of self-management upon health outcomes or on health resource usage and has been based on evidence produced through randomised controlled trials (RCTs). The evidence produced suggests that these health interventions can be as effective as the introduction of new medications [8]. However, there have been concerns raised about limitations of only evaluating the impact of interventions involving patients through the quantitative techniques utilised in RCTs [9, 10].
The notion that all interventions within the medical field are comparable to a drug intervention and should therefore be evaluated in the same way, uncritically privileges evidence generated through randomised controlled trials over evidence generated in other ways [9, 10]. The persistent claims that RCTs are the ‘gold standard’ of health intervention evaluation results primarily because of the dominance of the scientific research paradigm within which healthcare exists [11]. RCTs merely reflect good experimental design, however there are many potential shortcomings in the design and implementation of RCTs [9, 10]. Indeed the dominance of RCTs as ‘the method’ for evaluating health interventions has begun to attract some significant criticism [12] and the utility and validity of other approaches for use in evaluating health interventions have begun to emerge [13].
Major proponents of the RCT as the ‘gold standard’ for the evaluation of health interventions are advocates of evidence based medicine (EBM). EBM has the laudable goal of promoting the improvement of patient care through the use of up-to-date high quality, research evidence to make healthcare decisions. However, implicit within the EBM approach is a hierarchy of evidence that places evidence from RCTs and other population level research above other types of research evidence [10]. For health interventions, particularly those that engage the patient as an active participant, this hierarchy is of concern, as it prioritises population level evidence and reduces emphasis upon individuals. Related to this is the reality that health professional member organisations frequently base their development of clinical practice guidelines on the evidence from RCTs and systematic reviews. While not mandatory, these guidelines often place significant pressure upon individual health professionals to adhere to them, particularly for medico-legal reasons [14]. RCTs frequently concentrate on only a small number of measurable outcomes and may ignore patients’ experiences and/or any unexpected effects from interventions that are more difficult to quantify or measure easily. During extended RCTs it is also difficult to isolate and evaluate any single intervention to avoid outside factors adversely affect the stability of the intervention [15].
More broadly, some researchers have raised questions about the dependence of RCTs upon their funding sources to produce outcome bias. This research was based upon a survey of published RCTs between 1990 and 2000 [16]. It revealed RCTs funded by for-profit organisations more frequently reported positive outcomes than those funded by not-for-profit organisations. These results raise questions around the validity of trial design and ethics. Perhaps most importantly they raise concerns in relation to the quality of the evidence that may now be used to treat patients.
However, aligned to this point is the reality check that even where evidence is produced through RCTs, gaps continue to exist between this evidence and its actual use in clinical practice [17]. Others research have argued that where RCTs evidence is used there is a need for more effective mechanisms to tailor this evidence to individual patients [18]. Perhaps of more relevance here however, have been critiques of how RCTs tend to marginalise individual patient experiences, priorities and perceptions [19, 20]. Most importantly however, the point needs to be made that whilst the goals of RCTs are laudable the practice, particularly where the interventions involve the patients themselves, continue to leave much to be desired. As has been identified:
“Although evidence-based medicine advocates laudably took improving patient care as their goal, the methods they use to define and measure outcomes remained firmly in specialist hands. The collaborations set up by Cochrane and other similar groups were partnerships between researchers and clinicians, not doctors and patients…the most easily measured aspects of physiological change and therapeutic efficacy did not necessarily capture the markers of recovery that patients themselves deemed most important” [21].
It should be emphasized that the intention of this critical appraisal is not to present a unilateral argument for the discarding of RCTs but rather to highlight that there are some limitations that need to be considered. This is particularly the case where health interventions aim to actively involve patients and aim to generate evidence that meaningfully captures outcomes that reflect individual and cohort impacts [22]. The dominance of RCTs as a means of evaluating health care interventions continues to create a conflict between statistical evidence and the individual patient experience that needs to be more actively addressed if we are not to make the term patient-centred increasingly meaningless.
Approaches to the Evaluation of Health Information Systems
The diffusion of information and communication systems within the health system clearly has the potential to support a ‘better connected health system’ that is more efficient, safer, knowledge-based, and patient-centred. However, there is a need for health information systems researchers to avoid downplaying the complexity and uncertainty that exists in the delivery of health care services or to make problematic assumptions about how information systems will benefit patients, health professionals and the health care system as a whole [23, 24].
Based on the history of health information systems deployments that have traditionally focused on health professionals and hospitals, it is not surprising that in developing evidence of their impact statistical methods including RCTs were until recently considered the methods of choice [23]. These evaluative approaches tried to compare the introduction or testing of an information system in a process analogous to the testing of a drug or other discrete intervention. Indeed, some health information systems researchers continue to be proponents of these methods [25]. However, the increasing penetration of the Internet and access and use of on-line health information by patients and citizens has made the use of other methods for evaluating the impact of information systems become more widely accepted [23, 26-29]. Similarly as the health information systems themselves have evolved the focus of evaluation has moved from hardware and systems architectures towards the end-user where new techniques and evaluation criteria focused on usability, interfaces and content have been utilised [30-33]. This stated, there continues to be a need to improve the quality of reporting of evaluation studies [34].
More generally, there is a need to also recognise that reported benefits from the introduction of health information systems may be questionable or use measures of success that have little to do with improvements in care or patient outcomes [35, 36]. There is also a need for health information systems researchers to reflect on their underlying assumptions about the role, impact and importance of information. Particularly given that it is only one factor among many that influences health attitudes, perceptions, actions and outcomes.
As health interventions involving patients’ use of web-based information systems become more common, there are dangers for health information systems researchers of ending up treating patients as merely ‘end-users’ of their systems. When incorporating patients into design processes there is a need to continue to prioritise individual patient experiences, priorities and perceptions [24, 37, 38]. Indeed, supporting patients’ active participation in their own care makes it imperative that we move beyond assessments of benefit in terms of the cohort so that systems engage with individual patient’s and move us closer to genuine ‘individualised care’ and ‘patient empowerment’ [39].
In reviewing conventional approaches that have been used in the evaluation of health information systems it is clear that many have simply been adopted directly from techniques previously used in other industry sectors. For example, evaluations of systems development lifecycles have tended to be quantitative in nature and have almost exclusively focused on looking for intended benefits in terms of the technologies themselves, overall system performance or economic outcomes [26, 27]. There have also been large numbers of evaluation studies into interface design [40, 41], systems success factors [42, 43] and user satisfaction [44].
More recently, as the use of interpretivist research philosophies in information systems have become more widely accepted, user centred approaches to the evaluation of health information systems have become more common. These user-centred approaches have tended to focus on the continuous improvement of the system, with evaluation and review undertaken throughout the development lifecycle and with the input from end-users integral to the process [28]. Usability is a key concept within these approaches and refers to the degree to which an information systems is useful, efficient, effective and enjoyable to use [29]. However, within the health care domain these approaches acknowledge that information needs and flows can be difficult to determine precisely [28, 45].
There are now a vast array of methods employed within user-centred approaches including user-centred design (UCD), Human Computer Interaction (HCI) and Participatory Design (PD) [46]. Significantly, while this trend towards the utilisation of a variety of evaluation approaches is promising it brings with it a range of problems. It is now even more challenging to compare results generated from these approaches when considering particular health information systems or applications [34]. Clearly the evaluation of different health information systems requires different methodologies. For example, an information system designed for a healthcare organisation and a collaborative work environment such as a hospital requires different methods in comparison to a system designed for an individual care provider or a healthcare consumer. This is even more important when considering how the socio-technical network of a system, its users, and the target organisation develops following its implementation, and this itself requires further comprehensive study of the system’s behaviour in context. Furthermore, both types of evaluation approaches, those used to investigate information systems within real implementation environments and those used to evaluate information systems impacts at a single user level under controlled conditions, have limitations. For example, it can be argued that approaches that evaluate information systems under controlled conditions, such as simulation studies, may produce little insight into the impacts of these systems in a real-world organisation [47, 48].
Significantly, this brief review of approaches to the evaluation of health information systems suggests a division between health interventions that are primarily driven by technologists focused on improving ‘information systems’ and those that are primarily driven by health professionals focused on improving ‘cohort health outcomes’. For both there is a need to genuinely consider the individual patient as participant in these interventions and what is implied when we talk about the ‘patient at the centre’, patient empowerment and patient centred design [49].
KEEPING PATIENTS AT THE CENTRE
Schneider and Lane [50] have argued:
“Too often, a patient's treatments are performed "by the book" rather than being tailored to the patient's specific needs. To put it more bluntly, much of the care that today's patients receive is "medicine by numbers," which is best suited for that mythical abstraction, the "statistically average patient"”.
Health interventions involving patients using information systems present challenges to conventional clinical and information system approaches to the generation and evaluation of evidence on impacts and outcomes.
From a clinical perspective the analysis of discrete sets of independent variables within RCTs is purposeful but tends to provide evidence of changes only at the cohort level of those involved in the health intervention. This is despite knowledge of the fact that the interplay of psychological and social factors directly impact upon an individual’s self-perceived health status during their participation in clinical interventions [9-11, 15, 18, 19, 21].
From an information systems perspective user-centred approaches do involve the ‘user’ but may miss the patient! Focusing on technological acceptance or usability is sensible but it may often fail to examine in any detail the interplay of personal factors that make up an individual patients technology experiences, attitudes and responses. Further, in the realm of patients’ as ‘users’ the implicit assumption of mandatory use must be discarded as individual patients adopt and use systems in ways that suit themselves and their lifestyles and not necessarily as anticipated by clinicians or information systems professionals [23, 24, 26, 29, 31, 33, 45, 49].
From a methodological perspective the tendency amongst these researchers to treat epistemology and method as being inseparable is evident [51, 52]. This is despite an awareness of the utility of employing a greater level of pragmatism when engaged in research with human participants [53]. Olson [54] argues that the distinction between qualitative and quantitative data is purely a distinction of data types and does not predict how these data should be treated and that research philosophy is of more importance in determining how data is analysed than the data type per se. Therefore in generating and evaluating evidence from health interventions involving patients use of web-based information systems, a case can be argued for analysing numeric and non-numeric data in an effort to ‘keep the patient at the centre’ [55].
Of course, working through the implications of these points is much more difficult in practice than in theory. From a patient’s perspective the huge array of web-based health resources poses its own problems including differentials amongst individual patients to be able to categorise or recognise the validity and reliability of the content provided [33]. Although these resources may improve patient knowledge, their effect on objective clinical outcomes is inconsistent [56, 57]. It is also important to be realistic about the capacity of patients’ to embrace web-based information systems, as financial and skill-based barriers may inhibit access to on-line information. These issues relate to broader concerns about how to maintain a patient focus in health interventions in order to avoid the creation of an ‘eHealth divide’ [49]. In the era of rapidly ageing populations and growing chronic illness it is also sobering to consider that it may continue to be those individuals with the most limited access to resources (financial, educational etc. and including Internet access) who are further marginalised by these types of health interventions.
For clinical and information systems researchers these types of health interventions may also challenge some deeply held beliefs about ‘what we think we are doing when we intervene’ versus how this is perceived by our patients (participants). If researchers are really serious about ‘keeping the patient at the centre’ then the reality of discontinued use of these types of health interventions must be faced. Otherwise there is the risk of ending up trying to replace a dependency on health professionals for a dependency on information systems and the professionals who implement them [58]. There is a need to maintain an awareness of the fact that patients may be unwilling or unable to take on the new responsibilities of ‘empowerment’ and that even if they do, that this will axiomatically lead to improved quality of care whether evaluated at the individual or cohort level [49]. To date, evidence on the impact of attempts to ‘empower’ patients, and the actual impact this has on care practices and outcomes remains limited [59]. Indeed, despite the rhetoric, it appears evident that the impact of the availability of web-based clinical and other health information on patient-doctor relationships is mixed and dependent on a range of other factors that are not related to information or the systems that make it accessible [21, 60].
METHODOLOGY
Developing an Integrated Approach to Evaluating Evidence
The issues and challenges for clinical and information systems researchers arising from health interventions involving patients using web-based information systems will continue to generate a range of approaches. From a methodological perspective, and with the aim of ‘keeping the patient at the centre’, this section of the paper outlines an integrated way to meaningfully evaluate different types of evidence on the impacts of these types of interventions. This approach recognises that a range of socio-technical processes impact directly on how patients’ interests and outcomes are measured, defined and evaluated within health interventions. Critically, this approach argues that within the confines of a RCT involving patients use of web-based information systems, it is important to also examine individual participants. More specifically this involves exploring how individual’s interpret and discuss their various experiences and to gain insight into individual experiences of the health intervention. It is argued that exploring and evaluating the interplay of factors that impact on individual patients’ choices to adopt and subsequently use web-based information systems, and the effect these decisions and uses ultimately have on personal outcomes, perceptions and priorities is very important.
Study Context
Building on RCTs evaluation of intervention cohorts to investigate the experiences of individual patients is difficult and time-consuming but generates an increased depth of knowledge at both cohort and individual levels on the impact of a health intervention. The integrated approach briefly outlined here has already been employed within two controlled trials that aimed to assist patients develop self-efficacy for self-management of their chronic conditions and used web-based information systems [24, 37, 49, 59, 61].
In each controlled trial participants were introduced to a web-based information system as a non-mandatory part of the clinical intervention. Through the necessity to conform to the RCT protocols the information system was limited in scope. A subset from each RCT intervention group was selected. This subset consisted of two subgroups; patients who used the web-based system and patients who chose not to opting for a paper-based alternative. A three phase evaluation was undertaken to explore the differences in outcomes and experiences of the participants within each group.
The first phase of the evaluation examined the project outcomes using the standardised outcome measures collected in the RCT in relation to the two sub-groups. These data were collected at baseline then quarterly for twelve months. Using these traditional outcome measures and comparative principles it could be assumed that any statistically significant differences between the two groups were the result of the influence of the web-based information system. The analysis undertaken used the mathematically derived standard measure, effect size, enabling comparison of change between or within groups [62, 63].
The second phase of the investigation used audio-recorded semi-structured interviews to explore the experiences of the participants from these sub-groups to examine how the use of the information system had influenced their experiences within the RCT. Interviews were transcribed and then thematic coding was used to analyse the data [64]. The coded interview data was compared across and within the two sub-groups to establish similarities and differences between and within the groups.
The third phase involved the combination of data from phases one and two for each of the participants, and explored in detail the inter-relationships amongst factors influencing participant’s health outcomes and experiences over the duration of their participation in the trial. Phase three provided access to a rich and highly detailed picture of the individual participant’s health outcomes and experiences and allowed the research to expand the depth of their investigation and understanding. This process can be seen to reconstruct the individual and their experiences and enabled the research to develop individual case studies that provided a range of evidence to expand upon the data analysis from the previous two phases [61].
SUMMARY
Using this methodology in both of the health interventions led to the generation of some very interesting results. In both instances phase one analysis based on conventional statistical analysis of the cohort clearly and unequivocally revealed that the web-based information system did not have a positive effect and there was the possibility of some minor negative effect. This evidence was common across all statistical measures deployed. However, analyses generated from phases two and three revealed a more complex account of the intervention and the impacts on user experiences, capacities for behavioural change, and the influence of individual disease knowledge and stage.
More specifically, it was highlighted that at the individual level there were a highly varied set of reactions to the introduction of the web-based information systems that were not revealed in phase one analysis. These reactions result from the inter-relationships between many individual factors including the severity of disease, self-image, family context, confidence, social support and disease understanding.
The results demonstrated many different experiences and unintended consequences including a participant with severe disease, who lived alone, who reported a transformation of their life through the use of solitaire, email and changed family status. For this individual patient, this did lead to an upward trend in symptoms for some time even though this trend was not statistically significant. This participant also continued to use the web-based information system for the duration of the project and found some benefit from this in terms of managing their disease and also considered that it made a difference to them in terms of quality of life. Another participant, who initially had elected to adopt the web-based information system, did not manage to use the system or the computer at all during the intervention. This participant had a moderately severe chronic condition and was primary carer for her husband who had a similar but severe chronic condition. Her husband was extremely unsupportive of her participation in the health intervention and refused to allow her access to further training in the use of the web-based information system. Despite these obstacles this participant persevered and acquired a good understanding of the aims of the health intervention and improved her self-efficacy for self-management through her participation [61].
CONCLUSIONS
This paper has considered how contemporary clinical and IS evaluation discourses on health interventions approach patients. More specifically, the paper has focused on the socio-technical processes by which patients’ interests and outcomes are measured, defined and evaluated within health interventions that involve them using web-based information systems.
The paper has presented a critical appraisal of the limitations of current clinical and IS evidence and evaluation processes in relation to patients involved in these types of health interventions. It has also outlined an integrated approach to evaluation that has been used to generate evidence about the impact of these types of health interventions. This approach has produced evidence about the impact of the intervention and the web-based information systems that are meaningful at both individual patient and patient cohort levels. The integrated approach outlined encourages the utilisation of both quantitative and qualitative data across cohorts, groups and individuals to build a detailed view of participants’ experiences and outcomes within this type of health intervention. By combining the analysis of RCT data with interview data this approach allows for evidence to be generated at multiple levels of abstraction that more readily supports ‘keeping the patient at the centre’ in evaluation.
As the number of health interventions directly involving patients and their use of information systems increase it is anticipated that the approach outlined will contribute to more comprehensive evaluations of the impact of interventions on patients and to the accommodation of perspectives that patients themselves recognise, acknowledge and deem relevant.
ACKNOWLEDGEMENTS
The authors would like to acknowledge the members of the research teams conducting the controlled trials with whom they have collaborated during the development of this methodology: A Robinson, H Courtney-Pratt, H Cameron-Tucker, DW Reid, R Wood-Baker, EH Walters, C Wainwright, J Hauser and M Jessup.
REFERENCES
| [1] | Grol R. Improving the quality of medical care: Building bridges among professional pride, payer profit, and patient satisfaction JAMA 2001; 286: 2578-86. |
| [2] | Boston Consulting Group. Australian National Health Information Management Information & Communications Technology Strategy Final Report B C Group, National Health Information Group (NHIG) and Australian Health Information Council (AHIC) on behalf of the Commonwealth Government of Australia 2004. |
| [3] | Harrison M, Koppel R, Bar-Lev S. Unintended Consequences of Information Technologies in Health Care – An Interactice Sociotechnical Analysis J Am Med Inform Assoc 2007; 14(5): 542-9. |
| [4] | Coiera E. Four rules for the reinvention of health care BMJ 2004; 328: 1197-99. |
| [5] | Jadad AR, Delamothe T. What next for electronic communication and health care? BMJ 2004; 328: 1143-4. |
| [6] | Coleman MT, Newton KS. Supporting self-management in patients with chronic illness Am Fam Physician 2005; 72: 1503-0. |
| [7] | Downs JA, Roberts CM, Blackmorel AM, Le Souef PN, Jenkins SC. Benefits of an education programme on the self-management of aerosol and airway clearance treatments for children with cystic fibrosis Chron Respir Dis 2006; 3(1): 19-27. |
| [8] | Lorig K, Ritter P, Stewart A, et al. Chronic disease self-management program: 2-year health status and health care utilization outcomes Med Care 2001; 39(11): 1217-23. |
| [9] | Grossman J, Mackenzie FJ. The randomized controlled trial: gold standard, or merely standard? Perspect Biol Med 2005; 48(4): 516-34. |
| [10] | Bluhm R. From Hierarchy to Network: a richer view of evidence for evidence-based medicine Perspect Biol Med 2005; 48(4): 535-47. |
| [11] | Holmes D, Murray SJ, Perron A, Rail G. Deconstructing the evidence-based discourse in health sciences: truth, power and fascism Int J Evid Based Heathc 2006; 4(3): 180-6. |
| [12] | Smith G, Pell J. Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials Int J Prosthodont 2006; 19(2): 126-8. |
| [13] | Borgerson K. Evidence-based alternative medicine? Perspect Biol Med 2005; 48(4): 502-15. |
| [14] | Timmermans S. From autonomy to accountability: the role of clinical practice guidelines in professional power Perspect Biol Med 2005; 48(8): 490-501. |
| [15] | Øvretveit J. Evaluating health interventions. Buckingham: Open University Press 1998. |
| [16] | Ridker PM, Torres J. Reported outcomes in major cardiovascular clinical trials funded by for-profit and not-for-profit organizations: 2000-2005 JAMA 2006; 295(19): 2270-4. |
| [17] | Zuiderent-Jerak T. Preventing implementation: exploring interventions with standardization in healthcare Sci Cult 2007; 16(3): 311-29. |
| [18] | Olsson J, Terris D, Elg M, Lundberg J, Lindblad S. The one-person randomized controlled trial Qual Manag Health Care 2005; 14(4): 206-16. |
| [19] | Muir Gray JA. Evidence based policy making BMJ 2004; 329: 988-. |
| [20] | Berwick DM. Crossing the quality chasm: health care for the 21st Century Washington DC; updated 2004 (www.iom.edu/event.asp?id=12580) [cited 18/03/2004]; |
| [21] | Tomes N. Patient empowerment and the dilemmas of late-modern medicalisation Lancet 2007; 369(9562): 698-700. |
| [22] | Shumway M, Chouljian TL, Rozewicz F. Paraphrase procedures for assessing comprehension of health outcome measures: an illustration from schizophrenia research Eval Health Prof 2003; 26(1): 73-85. |
| [23] | Stoop AP, Berg M. Integrating Quantitative and qualitative methods in patient care information systems evaluation: guidance for the organizational decision maker Methods Inf Med 2003; 4: 458-62. |
| [24] | Cummings E, Turner P. Considerations for deploying web and mobile technologies to support the building of patient self-efficacy and self-management of chronic illness In: Al-Hakim L, Ed. Web mobile-based applications for healthcare management: Idea Group, Inc. 2007. |
| [25] | Nguyen HQ, Cuenco D, Wolpin S, Benditt J, Carrieri-Kohlman V. Methodological Considerations in Evaluating eHealth Interventions Can J Nurs Res 2007; 39(1): 116-34. |
| [26] | Stoop A, Heathfield H, de Mul M. Evaluation of patient care information systems In: Berg M, Ed. Health information management: integrating information technology in health care work. London: Routledge 2004. |
| [27] | Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges BMJ 1998; 316: 1959-61. |
| [28] | Kushniruk A. Evaluation in the design of health information systems: application of approaches emerging from usability engineering Comput Biol Med 2002; 32: 141-9. |
| [29] | Kushniruk A, Borycki E, Kuwata S, Ho F. Emerging approaches to evaluating the usability of health information systems In: Kushniruk A, Borycki E, Eds. Human, social, and organisational aspects of health information systems. Hershey: IGI Global 2008. |
| [30] | Pagliari C, Sloan D, Gregor P, et al. What is ehealth (4): a scoping exercise to map the field J Med Internet Res 2005; 7(1): e9. |
| [31] | Gustafson DH, Wyatt JC. Evaluation of ehealth systems and services -We need to move beyond hits and testimonials BMJ 2004; 328(7449): 1150. |
| [32] | Jadad AR, Delamothe T. From electronic gadgets to better health: where is the knowledge? BMJ 2003; 327(7410): 300-1. |
| [33] | Al-Busaidi A, Gray A, Fiddian N. Personalizing web information for patients: linking patient medical data with the web via a patient personal knowledge base Health Informatics J 2006; 12(1): 27-39. |
| [34] | Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykänen P, Rigby M. STARE-HI--Statement on reporting of evaluation studies in Health Informatics Int J Med Inform 2009; 78(1): 1-9. |
| [35] | Wyatt J. Health Informatics Society of Australia. Evaluating Health Informatics Research In: Keynote Address, Health Informatics Conference; July 25-27, 2004; Brisbane. |
| [36] | Glasgow RE. eHealth evaluation and dissemination research Am J Prev Med 2007; 32(5 Suppl): S119-26. |
| [37] | Cummings E, Turner P. Patient self-management and chronic illness: evaluating outcomes and impacts of information technology Stud Health Technol Inform 2009; 143(1): 229-34. |
| [38] | Kelder JC, Turner P. “Lost in translation: critical reflection on quantitative approaches for informing information systems design” In: Proceedings of QualIT 2005: Challenges for qualitative research; 23-25 November 2005; Brisbane, Australia. |
| [39] | Muir Gray JA. Self-management in chronic illness Lancet 2004; 364(9444): 1467-8. |
| [40] | Gould J, Lewis C. Designing for usability: Key principles and what designers think Commun ACM 1985; 28(3): 300-11. |
| [41] | Shneiderman B. Designing the user interface In: Forester T, Ed. Computers in the human context. Oxford: Basil Blackwell 1989. |
| [42] | DeLone W, McLean E. Information systems success: the quest for the dependent variable Inform Syst Res 1992; 3(1): 60-95. |
| [43] | Sanders G, Garrity E. Dimensions of information systems success In: Sanders G, Garrity E, Eds. Informtion systems success measurement. Hershey, USA: Idea Group Publishing 1998. |
| [44] | Ives B, Olson M, Baroudi J. The measurement of user information satisfaction Commun ACM 1983; 26(10): 785-93. |
| [45] | Nøhr C, Boye N. Towards computer supported clinical activity: a roadmap based on empirical knowledge and some theoretical reflections In: Kushniruk A, Borycki E, Eds. Human, social, and organizational aspects of health information systems. Hershey: IGI Global 2008; pp. 67-83. |
| [46] | Price M. A bio-psycho-social review of usability methods and their applications in healthcare In: Kushiniruk A, Borycki E, Eds. Human, social, and organizational aspects of health information systems. Hershey: IGI Global 2008; pp. 23-48. |
| [47] | Preece J, Rogers Y, Sharp H. Interaction design: beyond human-computer interaction. New York: Wiley 2002. |
| [48] | Sanderson P, Ed. Designing and evaluating healthcare ict innovations: a cognitive engineering view information technology in health care 2007. Proceedings of the 3rd International Conference on Information Technology in Health Care: Socio-technical Approaches; Sydney. IOS Press 2007. |
| [49] | Cummings E, Chau S, Turner P. Assessing a patient-centered e-health approach to chronic disease self-management In: Wilson EV, Ed. Patient-Centered E-Health. Hershey PA: Medical Information Science Reference 2009. |
| [50] | Schneider S, Lane J. The patient from hell: How i worked with my doctors to get the best of modern medicine and how you can too. Cambridge, MA: Da Capo Press 2005. |
| [51] | Howe KR. Getting over the quantitative-qualitative debate Am J Educ 1992; 100(2): 236-56. |
| [52] | Bryman A. The debate about quantitative and qualitative research: a question of method or epistemology? Br J Sociol 1984; 35(1): 75-92. |
| [53] | Onwuegbuzie AJ, Leech NL. On becoming a pragmatic researcher: the importance of combining quantitative and qualitative research methodologies Int J Soc Res Methodol 2005; 8(5): 375 -87. |
| [54] | Olson H, Ed. Qualitative "versus" quantitative research: the wrong question. CAIS/ACSI 95, Annual Conference of the canadian association for information science, connectedness: Information, Systems, People, Organizations; Edmonton, Alberta; June 7-10, 1995; School of Library and Information Studies, University of Alberta. |
| [55] | Bhaskar RA, Ed. The possibility of naturalism. London: Routledge 1979. |
| [56] | Bussey-Smith KL, Rossen RD. A systematic review of randomized control trials evaluating the effectiveness of interactive computerized asthma patient education programs Ann Allergy Asthma Immunol 2007; 98(6): 507-16. |
| [57] | Välimäki M, Nenonen H, Koivunen M, Suhonen R. Patients’ perceptions of Internet usage and their opportunity to obtain health information Inform Health Soc Care 2007; 32(4): 305-14. |
| [58] | Wyatt S, Thomas G, Terranova T. They came, they surfed, they went back to the beach. Conceptualising use and non-use of the Internet In: Woolgar S, Ed. Virtual Society?. Oxford: Oxford University Press 2002. |
| [59] | Robinson AL, Courtney-Pratt HM, Lea EJ, et al. Transforming clinical practice amongst community nurses: mentoring for COPD patient self-management J Clin Nurs 2008; 17: 370-9. |
| [60] | Nwosu CR, Cox BM. The impact of the Internet on the doctor-patient relationship Health Inform J 2000; 6(3): 156-61. |
| [61] | Cummings E. An investigation of the influence of an online patient diary on the health outcomes and experiences of people with chronic obstructive pulmonary disease (COPD) participating in a mentored self-management clinical controlled trial [PhD]. University of Tasmania: Hobart 2008. |
| [62] | Kazis LE, Anderson JJ, Meenan RF. Effect Sizes for Interpreting Changes in Health Status Med Care 1989; 27(Suppl 3): S178-89. |
| [63] | Cohen J, Ed. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Lawrence Earlbaum Associates 1988. |
| [64] | Strauss AL, Corbin J. Basics of qualitative research: grounded theory procedures and techniques. Newbury Park: SAGE Publications 1990. |
