Process-supporting information technology holds the potential to increase efficiency, reduce errors, and alter professional roles and responsibilities in a manner which allows improvement in the delivery of patient care [1]. However, despite large-scale investments in Information Communication Technology (ICT), uptake has been slow [1, 2] whilst recent evidence of ICT-induced errors has called for a more critical examination of the impact of such technology in healthcare [3-6].

Grounded in both social theory and Information Technology (IT), sociotechnical approaches to ICT evaluation focuses on the interrelation between technology and its social environment [7]. Over time, ICT implementation models have developed an emphasis on the alignment between the clinical context where clinical work is carried out and the technology designed to improve the delivery of care [8-10], as well as the contextual differences both between organisations and within – at departmental and individual levels [11]. This paper will propose a methodology for the sociotechnical evaluation of the impact of a computerised information system on a clinical care department.

Contemporary health organisations are complex, composed of deeply interdependent and interrelated social and technical elements where changes to one aspect will affect the other [12]; the introduction of any new technology into a setting will entail innovation in clinical roles, work processes, and culture change [13-15] whilst the attitudes and utility of technology is simultaneously socially shaped [13]. Rather than viewing computer systems as isolated entities within their intended clinical environment, the contemporary perspective portrays systems as an active component of the clinical team, constantly cooperating and dynamically interacting with the clinical staff and organisational routines [12, 16].

The sociotechnical approach is one which seeks to identify the dynamics between technology and the social, professional, and cultural environment in which it is used [17]. The limitations of a purely technocentric approach to systems evaluation became apparent when early implementation projects were less than successful and purported benefits of ICT rarely realised [17-19]. Causes of implementation failure extend beyond poor system design to the erroneous perceptions and theories about how medical work is conducted [12, 20, 21]. Clashes between the model of health care work inscribed in these tools with the actual nature of work has resulted in staff resistance and decreased organisational uptake of ICT, as well as the facilitation of unexpected and negative effects in efficiency and patient safety [3, 4, 15, 18]. Sociotechnical theory provides a paradigm against which workflow and perfusion of ICT in healthcare could be better explored and understood.

Any form of ICT evaluation in health may be distinguished as either summative or formative. Summative evaluations determine the overall quality or value of an intervention and have traditionally followed the objectivist view which dictates that the merits and worth of ICT should and could be quantified [18, 22]. In healthcare, such evaluations of ICT are conducted post hoc to account for the impacts of the system such as financial benefits or system effectiveness in terms of clinical outcomes [18]. Dominating the literature on health ICT evaluation, ICT is often conceptualised as the “intervention” in summative studies which allude, as in medicine, to the Randomised Controlled Trial (RCT) model of evaluation as the “gold standard” of evaluation [18, 23].

An issue with ICT evaluation in health is the complexity of the evaluation object [24]. The deep integration of ICT with the culture and work practices of the healthcare organisation requires an evaluative focus which expands beyond only software and hardware to the surrounding clinical situation and associated processes to capture the diverse and diffuse nature of system effects in the healthcare setting [24]. Whilst RCT-type studies are valuable for determining the cause and impact of a pre-specified effect, it assumes possibility of isolating system functioning from the social processes that surround it, whilst reality suggests that the design, implementation, and operation of ICT are deeply influenced by organisational and social factors [23].

Formative evaluations are conducted during the development of a product for the purpose of system improvement [22]. The increase in qualitative studies of health ICT has exhibited a shift towards the subjectivist theoretical orientation to systems evaluation which asserts the responsibility of social and organisational aspects in shaping the outcomes of objective measurement and endorsing a “multimethod” approach for both summative and formative evaluations [18]. The mixed or multimethod approach to health ICT evaluation is one which views qualitative and quantitative research methods as complementary [23, 25, 26]; the combination of methods allows exploration of the what, why, and how of a social phenomenon that qualitative methods can address, and the size, extent or duration (how much) of certain phenomena that quantitative methods establish [26]. Pleiotropic healthcare interventions such as ICT create a hybrid sociotechnical situation requiring a range of integrated multi-perspective research methods to achieve a holistic understanding [23, 27, 28].

ICT is also a dynamic and emerging process, continually evolving across the multiple phases of the implementation process [11, 23]. A longitudinal study design which follows the target of study over a substantial period of time will allow appreciation of this aspect by providing a means of monitoring changes caused by the system within the context it operates [22, 24].

This paper will conceptualise a formative, multi-method longitudinal evaluation process to explore the impact of ICT with an appreciation of the relationship between the social and technical systems within a clinical department.

The sociotechnical approach to ICT evaluation appreciates the highly complex healthcare setting and emphasises the relationship between the social and technical impacts of clinical information systems. ICT represent technical artefacts which embody the implicit theories about how work is conducted; success of the artefact in real clinical situations relies upon the accuracy and applicability of these theories [41]. ICT evaluation tools [42, 43] exist to guide the process of planning and execution of evaluation projects, with which the proposed protocol may be further developed and fine-tuned to accommodate individual settings; the methodology outlined in the paper supplements existing work in its offering of a unique, practical evaluation procedure which combines the theory and practice of the sociotechnical perspective to attain a more holistic understanding of the impacts of ICT and to ultimately achieve successful implementation.

However, sociotechnical theory does not exist without criticism. Examination of both the social and technical systems within a clinical department seeks to determine the different contextual factors operating in separate settings in order to identify the latent mechanisms that can influence outcomes [18]. Berg [44] cautions that understanding the interdependency of the social and technical systems within a particular context is insufficient for reconciling the social and technical aspects within that environment. Similarly, multi-method approaches to evaluation could potentially produce contradictory findings which do not, in fact, facilitate mutual interpretation of results produced from the separate methods [28]. The proposed methodology risks inheriting such limitations; efficacy of the process remains to be verified through independent field testing.

Sociotechnical theory provides a paradigm against which workflow and transfusion of ICT in healthcare could be better explored and understood. Incorporating both interactionist and ethnomethodological methods, this paper has outlined a formative, multi-method, longitudinal model for ICT evaluation which aims to inform interpretation of objective system effects in light of how the IT is conceptualised and viewed by users, as well as its impact on staff work practices within the unique context of the clinical department. However, inherent in the protocol are limitations of sociotechnical theory and criticism of the multimethod approach; testing of the methodology in real clinical settings will serve to verify efficacy and refine the process.