A Sociotechnical Approach to Evaluating the Impact of ICT on Clinical Care Environments
Identifiers and Pagination:Year: 2010
First Page: 202
Last Page: 205
Publisher Id: TOMINFOJ-4-202
Article History:Received Date: 30/11/2009
Revision Received Date: 21/1/2010
Acceptance Date: 25/1/2010
Electronic publication date: 21 /10/2010
Collection year: 2010
open-access license: This is an open access article licensed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted, non-commercial use, distribution and reproduction in any medium, provided the work is properly cited.
Introduction: Process-supporting information technology holds the potential to increase efficiency, reduce errors, and alter professional roles and responsibilities in a manner which allows improvement in the delivery of patient care. However, clashes between the model of health care work inscribed in these tools with the actual nature of work has resulted in staff resistance and decreased organisational uptake of ICT, as well as the facilitation of unexpected and negative effects in efficiency and patient safety. Sociotechnical theory provides a paradigm against which workflow and transfusion of ICT in healthcare could be better explored and understood.
Design: This paper will conceptualise a formative, multi-method longitudinal evaluation process to explore the impact of ICT with an appreciation of the relationship between the social and technical systems within a clinical department.
Method: Departmental culture, including clinical work processes and communication patterns will be thoroughly explored before system implementation using both quantitative and qualitative research methods. Findings will be compared with post implementation data, which will incorporate measurement of safety and workflow efficiency indicators.
Discussion: Sociotechnical theory provides a paradigm against which workflow and transfusion of ICT in healthcare could be better explored and understood. However, sociotechnical and multimethod approaches to evaluation do not exist without criticism. Inherent in the protocol are limitations of sociotechnical theory and criticism of the multimethod approach; testing of the methodology in real clinical settings will serve to verify efficacy and refine the process.
Process-supporting information technology holds the potential to increase efficiency, reduce errors, and alter professional roles and responsibilities in a manner which allows improvement in the delivery of patient care . However, despite large-scale investments in Information Communication Technology (ICT), uptake has been slow [1, 2] whilst recent evidence of ICT-induced errors has called for a more critical examination of the impact of such technology in healthcare [3-6].
Grounded in both social theory and Information Technology (IT), sociotechnical approaches to ICT evaluation focuses on the interrelation between technology and its social environment . Over time, ICT implementation models have developed an emphasis on the alignment between the clinical context where clinical work is carried out and the technology designed to improve the delivery of care [8-10], as well as the contextual differences both between organisations and within – at departmental and individual levels . This paper will propose a methodology for the sociotechnical evaluation of the impact of a computerised information system on a clinical care department.
WHY SOCIOTECHNICAL THEORY?
Contemporary health organisations are complex, composed of deeply interdependent and interrelated social and technical elements where changes to one aspect will affect the other ; the introduction of any new technology into a setting will entail innovation in clinical roles, work processes, and culture change [13-15] whilst the attitudes and utility of technology is simultaneously socially shaped . Rather than viewing computer systems as isolated entities within their intended clinical environment, the contemporary perspective portrays systems as an active component of the clinical team, constantly cooperating and dynamically interacting with the clinical staff and organisational routines [12, 16].
The sociotechnical approach is one which seeks to identify the dynamics between technology and the social, professional, and cultural environment in which it is used . The limitations of a purely technocentric approach to systems evaluation became apparent when early implementation projects were less than successful and purported benefits of ICT rarely realised [17-19]. Causes of implementation failure extend beyond poor system design to the erroneous perceptions and theories about how medical work is conducted [12, 20, 21]. Clashes between the model of health care work inscribed in these tools with the actual nature of work has resulted in staff resistance and decreased organisational uptake of ICT, as well as the facilitation of unexpected and negative effects in efficiency and patient safety [3, 4, 15, 18]. Sociotechnical theory provides a paradigm against which workflow and perfusion of ICT in healthcare could be better explored and understood.
Any form of ICT evaluation in health may be distinguished as either summative or formative. Summative evaluations determine the overall quality or value of an intervention and have traditionally followed the objectivist view which dictates that the merits and worth of ICT should and could be quantified [18, 22]. In healthcare, such evaluations of ICT are conducted post hoc to account for the impacts of the system such as financial benefits or system effectiveness in terms of clinical outcomes . Dominating the literature on health ICT evaluation, ICT is often conceptualised as the “intervention” in summative studies which allude, as in medicine, to the Randomised Controlled Trial (RCT) model of evaluation as the “gold standard” of evaluation [18, 23].
An issue with ICT evaluation in health is the complexity of the evaluation object . The deep integration of ICT with the culture and work practices of the healthcare organisation requires an evaluative focus which expands beyond only software and hardware to the surrounding clinical situation and associated processes to capture the diverse and diffuse nature of system effects in the healthcare setting . Whilst RCT-type studies are valuable for determining the cause and impact of a pre-specified effect, it assumes possibility of isolating system functioning from the social processes that surround it, whilst reality suggests that the design, implementation, and operation of ICT are deeply influenced by organisational and social factors .
Formative evaluations are conducted during the development of a product for the purpose of system improvement . The increase in qualitative studies of health ICT has exhibited a shift towards the subjectivist theoretical orientation to systems evaluation which asserts the responsibility of social and organisational aspects in shaping the outcomes of objective measurement and endorsing a “multimethod” approach for both summative and formative evaluations . The mixed or multimethod approach to health ICT evaluation is one which views qualitative and quantitative research methods as complementary [23, 25, 26]; the combination of methods allows exploration of the what, why, and how of a social phenomenon that qualitative methods can address, and the size, extent or duration (how much) of certain phenomena that quantitative methods establish . Pleiotropic healthcare interventions such as ICT create a hybrid sociotechnical situation requiring a range of integrated multi-perspective research methods to achieve a holistic understanding [23, 27, 28].
ICT is also a dynamic and emerging process, continually evolving across the multiple phases of the implementation process [11, 23]. A longitudinal study design which follows the target of study over a substantial period of time will allow appreciation of this aspect by providing a means of monitoring changes caused by the system within the context it operates [22, 24].
This paper will conceptualise a formative, multi-method longitudinal evaluation process to explore the impact of ICT with an appreciation of the relationship between the social and technical systems within a clinical department.
Pre System Implementation
1. To explore the context of the clinical department.
Implicit in the sociotechnical approach to systems evaluation is consideration for the context in which the system is implemented . Differing socio-organisational settings influence ICT adoption processes and system impact [10, 11, 30-33]. Departmental culture, including roles, clinical work processes and communication patterns will be thoroughly explored before system implementation to identify focus areas and facilitate interpretation of findings post implementation. Focus groups will be held pre system implementation with a purposive sample of participants across all clinical staff groups to obtain a broad representative cross-section. Discussion will concentrate on how work is currently conducted, issues of concern and expectations of the electronic system to gain a general understanding of the nature and needs of the department. Individual roles as well as the perceived roles and contributions of members of other staff groups will be explored and triangulated to establish a sense of departmental culture and dynamics. Significant points or themes arising from analysis of transcripts will be further explored during in-depth interviews with relevant participants or key informants identified from initial focus groups.
Whilst interview and focus groups examine how a situation is perceived, they are limited in their ability to understand and explain behaviour . Observations of actual work processes in real situations divulge more about individual work, successes, failures, and preferences than consultation with participants alone . Emerging themes from interviews and focus groups will also be explored through sufficient hours of non participant observation. This will be supplemented with video recordings which allow reviewing of specific work processes/behaviour multiple times, careful transcription and re-transcription of events to reveal and analyse “fine-grained, ‘seen but unnoticed’” aspects of conduct otherwise unidentified from the grosser examples of work phenomena obtained from plain observation alone . Non-participant observations and video recordings will follow certain individuals or teams during routine work processes with focus on specific items raised during interviews and focus groups.
Whilst qualitative research methods such as interviews, focus groups, and observations yield context-dependent data from the perspective of the participants within their organisational or departmental context, quantitative means for measuring social and departmental characteristics also exist, and may be used in conjunction with qualitative measures to strengthen robustness of data through triangulation . Social network analysis examines communication patterns and interactions between professional groups to elicit the patterns of connections that occur between staff . This may be further quantified using the Team Climate Inventory (TCI) which measures team cohesiveness and support for work innovation which serves to predict the likelihood of departmental acceptance and successful adoption of ICT . For the purposes of this study the entire department should be conceptualised as the team.
Post System Implementation
- To determine the impact of the clinical information system on staff functioning and departmental dynamics
- To quantify the impact of the clinical information system on patient safety, departmental productivity and efficiency indicators
Evaluation will maintain focus on the impact of ICT on departmental culture, work practices and communication post implementation. Focus groups, interviews, and observations conducted pre system implementation will be repeated post implementation to identify changes to the previously defined culture, roles, communication patterns and work processes. Questions will be informed by existing literature on the impact of information systems on culture and work in conjunction with pre implementation findings, including specific areas of concern identified from pre implementation study unique to the department. Social network surveys will be redistributed at this stage to identify changes in communication patterns and individual roles in the exchange of information in the network.
A central argument for the implementation of computerised patient information systems is their potential for improving patient safety and work practice efficiency . Key performance indicators for safety and productivity (e.g. medication errors , diagnostic turnaround times , and patient waiting times ) exist from evaluation literature in the field, and will be selected and measured in conjunction with context-dependent issues or indicators identified from analysis of qualitative data both pre and post system implementation. To elucidate impacts of the computerised system, data from both before and after implementation of the system will be elicited from relevant administration databases as well as paper documentation. Quantitative measurements of indicator data supplement qualitative findings, adding rigour and validation whilst qualitative data place quantitative findings in an appropriate social setting that allows interpretation and a broader understanding of their implications .
ICT implementation is multi-staged [11, 23]. Data collection will be carried out immediately post implementation and continue at appropriate intervals across the various stages throughout the implementation process to monitor impacts and inform system customisation and development to improve system integration into existing work processes and culture. Findings and recommendations should be considered and formulated in light of influence from complex external factors such as resource limitations, government policy, influence from different stakeholders, the economic environment, and features of the health system and the information technology industry [11, 18].
- Quantification of the effects of a clinical information system on a range of safety, productivity, and efficiency indicators
- Identification of the underlying mechanisms which determine optimal systems functioning within the clinical department
The sociotechnical approach to ICT evaluation appreciates the highly complex healthcare setting and emphasises the relationship between the social and technical impacts of clinical information systems. ICT represent technical artefacts which embody the implicit theories about how work is conducted; success of the artefact in real clinical situations relies upon the accuracy and applicability of these theories . ICT evaluation tools [42, 43] exist to guide the process of planning and execution of evaluation projects, with which the proposed protocol may be further developed and fine-tuned to accommodate individual settings; the methodology outlined in the paper supplements existing work in its offering of a unique, practical evaluation procedure which combines the theory and practice of the sociotechnical perspective to attain a more holistic understanding of the impacts of ICT and to ultimately achieve successful implementation.
However, sociotechnical theory does not exist without criticism. Examination of both the social and technical systems within a clinical department seeks to determine the different contextual factors operating in separate settings in order to identify the latent mechanisms that can influence outcomes . Berg  cautions that understanding the interdependency of the social and technical systems within a particular context is insufficient for reconciling the social and technical aspects within that environment. Similarly, multi-method approaches to evaluation could potentially produce contradictory findings which do not, in fact, facilitate mutual interpretation of results produced from the separate methods . The proposed methodology risks inheriting such limitations; efficacy of the process remains to be verified through independent field testing.
Sociotechnical theory provides a paradigm against which workflow and transfusion of ICT in healthcare could be better explored and understood. Incorporating both interactionist and ethnomethodological methods, this paper has outlined a formative, multi-method, longitudinal model for ICT evaluation which aims to inform interpretation of objective system effects in light of how the IT is conceptualised and viewed by users, as well as its impact on staff work practices within the unique context of the clinical department. However, inherent in the protocol are limitations of sociotechnical theory and criticism of the multimethod approach; testing of the methodology in real clinical settings will serve to verify efficacy and refine the process.
|||Fonkych K, Taylor R. The state and pattern of health information technology adoption, Rand Corporation, Santa Monica 2005.|
|||Ash JS, Gorman PN, Seshadri V, et al. Computerized physician order entry in U.S. hospitals: results of a 2002 survey J Am Med Inform Assoc 2004; 11(2 ): 95-.|
|||Koppel R, Metlay J, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors J Am Med Assoc 2005; 293(10 ): 1197-203.|
|||Han Y, Carcillo J, Venkataraman S, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system Pediatrics 2005; 116(6 ): 1506-2.|
|||Garg AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review JAMA 2005; 293(10 ): 1223-38.|
|||Perrow C. Normal accidents: Living with high-risk technologies 1999.|
|||Berg M, Aarts J, Van Der Lei J. ICT in health care: sociotechnical approaches Methods Inform Med 2003; 42(4 ): 297-301.|
|||Goodhue D, Thompson R. Task-technology fit and individual performance MIS Q 1995; 00: 213-36.|
|||Aarts J, Peel V, Wright G. Organizational issues in health informatics: a model approach Int J Med Inform 1998; 52(1-3 ): 235-42.|
|||Ammenwerth E, Iller C, Mahler C, Ammenwerth E, Iller C, Mahler C. IT-adoption and the interaction of task, technology and individuals: a fit framework and a case study BMC Med Inf Decis Mak 2006; 6: 3-0.|
|||Callen J, Braithwaite J, Westbrook J. Contextual implementation model: a framework for assisting clinical information system implementations J Am Med Inform Assoc 2008; 15(2 ): 255-62.|
|||Wears RL, Berg M. Computer technology and clinical work: still waiting for godot JAMA 2005; 293(10 ): 1261-3.|
|||Coiera E. Four rules for the reinvention of health care BMJ 2004; 328(7449 ): 1197-9.|
|||Ash JS, Sittig DF, Campbell E, et al. An unintended consequence of CPOE implementation: shifts in power, control, and autonomy: 2006: Proceedings of the AMIA Annual Symposium. Washington DC, United States of America 2006.|
|||Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: Findings from a mixed methods exploration Int J Med Inform 2009; 78(Suppl1): S69-76.|
|||Anderson J, Aydin C. Evaluating the organizational impact of healthcare information systems 2005.|
|||Whetton S, Ed. Health informatics: a socio-technical perspective Melbourne: Oxford University Press 2005.|
|||Berg M. Health information management: integrating information technology in health care work. Routledge, London, 2004.|
|||Littlejohns P, Wyatt JC, Garvican L. Evaluating computerised health information systems: hard lessons still to be learnt BMJ 2003; 326(7394 ): 860-3.|
|||Aarts J, Gorman P. IT in health care: sociotechnical approaches "To Err is System" Int J Med Inform 2007; 76(Suppl 1): S1-3.|
|||Aarts J, Ash J, Berg M. Extending the understanding of computerized physician order entry: Implications for professional collaboration, workflow and quality of care Int J Med Inform 2007; 76: 4-13.|
|||Scriven M. The logic and methodology of evaluation, Edgepress, Inverness.CA 1981.|
|||Kaplan B. Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism Int J Med Inform 2001; 64(1 ): 39-56.|
|||Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of health information systems—problems and challenges Int J Med Inform 2003; 71(2-3 ): 125-35.|
|||Westbrook JI, Braithwaite J, Georgiou A. Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory J Am Med Inform Assoc 2007; 14(6 ): 746-55.|
|||Stoop AP, Berg M. Integrating quantitative and qualitative methods in patient care information system evaluation: guidance for the organizational decision maker Methods Inform Med 2003; 42(4 ): 458-62.|
|||Scott P, Briggs J. A pragmatist argument for mixed methodology in medical informatics J Mixed Methods Res 2009; 3(3 ): 223-0.|
|||Kaplan B, Shaw N. Future directions in evaluation research: people, organizational, and social issues Methods Inform Med 2004; 43(3 ): 215-31.|
|||Pasmore W. Social science transformed: the socio-technical perspective Hum Relat 1995; 48(1 ): 1-21.|
|||Callen J, Braithwaite J, Westbrook J. Cultures in hospitals and their influence on attitudes to, and satisfaction with, the use of clinical information systems Soc Sci Med 2007; 65(3 ): 635-9.|
|||Aarts J, Berg M. Same systems, different outcomes--comparing the implementation of computerized physician order entry in two Dutch hospitals Methods Inform Med 2006; 45(1 ): 53-61.|
|||Niazkhani Z, van der Sijs H, Pirnejad H, et al. Same system, different outcomes: comparing the transitions from two paper-based systems to the same computerized physician order entry system Int J Med Inform 2009; 78(3 ): 170-81.|
|||Peute L, Aarts J, Bakker P, Jaspers M. Anatomy of a failure: A sociotechnical evaluation of a laboratory physician order entry system implementation Int J Med Inform 2010; 74(4 ): 58-70.|
|||Norman D. Emotional Design: Why we love (or hate) everyday things 2004.|
|||Greatbatch D, Murphy E, Dingwall R. Evaluating medical information systems: ethnomethodological and interactionist approaches Health Serv Manage Res 2001; 14(3 ): 181-91.|
|||Heng HKS, McGeorge WD, Loosemore M. Beyond strategy: Exploring the brokerage role of facilities manager in hospitals J Health Org Manag 2005; 19: 16-31.|
|||Ouwens M, Hulscher M, Akkermans R, Hermens R, Grol R, Wollersheim H. The team climate inventory: application in hospital teams and methodological considerations Qual Saf Health Care 2008; 17(4 ): 275-0.|
|||Georgiou A, Westbrook J, Braithwaite J, et al. When requests become orders—A formative investigation into the impact of a computerized physician order entry system on a pathology laboratory service Int J Med Inform 2007; 76(8 ): 583-91.|
|||Boyce NW, Boyce NW. Potential pitfalls of healthcare performance indicators Med J Aust 2002; 177(5 ): 229-30.|
|||Pawson R. A measure for measures: a manifesto for empirical sociology 1989.|
|||Woods D. Designs are hypotheses about how artifacts shape cognition and collaboration Ergonomics 1998; 41(2 ): 168-73.|
|||Nykänen P, Brender J, Ammenwerth E. Guidelines for Good Evaluation Practices in Health Informatics (v0.16) 2008 [cited 2009 Jan 5]: Available from: http://iig.umit.at/efmi/gep-hi/GEP_HI_v016_Dec012008.doc.|
|||Cusack C, Byrne C, Hook J, McGowan J, Poon E, Zafar A. Health information technology evaluation toolkit: 2009 Update. Rockville, MD: Agency for Healthcare Research and Quality Report No.: AHRQ Publication No. 09-0083-EF. June 2009 2009.|
|||Berg M. Patient care information systems and health care work: a sociotechnical approach Int J Med Inform 1999; 55(2 ): 87-101.|