RESEARCH ARTICLE


STAT-HI: A Socio-Technical Assessment Tool for Health Informatics Implementations



Philip J Scott*, James S Briggs
Centre for Healthcare Modelling and Informatics, University of Portsmouth, UK


Article Metrics

CrossRef Citations:
7
Total Statistics:

Full-Text HTML Views: 3940
Abstract HTML Views: 2081
PDF Downloads: 252
Total Views/Downloads: 6273
Unique Statistics:

Full-Text HTML Views: 1881
Abstract HTML Views: 1158
PDF Downloads: 187
Total Views/Downloads: 3226



© Scott and Briggs; Licensee Bentham Open.

open-access license: This is an open access article licensed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted, non-commercial use, distribution and reproduction in any medium, provided the work is properly cited.

* Address correspondence to this author at the Centre for Healthcare Modelling and Informatics, School of Computing, University of Portsmouth, Lion Terrace, Portsmouth PO1 3HE, UK; Tel: +44 23 9284 6378; Fax: +44 23 9284 6364; E-mail: Philip.scott@port.ac.uk


Abstract

This paper proposes a socio-technical assessment tool (STAT-HI) for health informatics implementations. We explore why even projects allegedly using sound methodologies repeatedly fail to give adequate attention to socio-technical issues, and we present an initial draft of a structured assessment tool for health informatics implementation that encapsulates socio-technical good practice. Further work is proposed to enrich and validate the proposed instrument. This proposal was presented for discussion at a meeting of the UK Faculty of Health Informatics in December 2009.

Keywords: Socio-technical, checklist, project management, programme management, hospital information system, complex systems.



INTRODUCTION

This paper seeks to address the issue of practical application of socio-technical principles in health informatics implementations. In the UK, this issue has arisen mostly from concerns about the National Health Service (NHS) National Programme for IT (NPfIT) in England. This topic has been a focus for discussion by the UK Faculty of Health Informatics, an informal group comprising a mixture of academics and practitioners in the field.

What is the Problem?

NPfIT was established in 2002 to deliver “twenty-first century IT support” for the NHS, defined as a set of electronic services to provide lifelong health records, direct booking of outpatient appointments and transmission of prescriptions from primary care to pharmacy, all based upon a robust information architecture and secure broadband network [1].

There has been sustained and widespread criticism, not always well-informed, of the perceived failures of NPfIT and its overly centralist, secretive and monolithic approach [2, 3]. For example, the 2002 Gateway Review by the UK Office of Government Commerce (OGC) into the business justification for what was then called the Integrated Care Records Service (only recently published following requests made under the provisions of the Freedom of Information Act [4]) stated that they found “much talk of what the IT programme will achieve, but little recognition of the potential impact of this on current practices, procedures and systems, both technical and organisational” [5].

Similar themes continue to be echoed [6]. The evaluation report on the NPfIT Summary Care Record pilot implementation in North West England suggested that the project’s failure to adopt a socio-technical approach was one reason why the system was not embraced by busy clinicians [7].

Comparable problems have been reported in other major health informatics (HI) implementations [8, 9], so this issue is by no means unique to the English programme.

What is the Socio-Technical Systems View?

The roots of socio-technical thinking can be found in research studies by the Tavistock Institute in the 1950s. For example, investigations in the British coal industry found that autonomous groups of miners had re-invented surprisingly effective innovations in mining methods contrary to “obvious” techniques suggested by increasing mechanization [10]. The socio-technical approach was built upon a philosophy of improving worker satisfaction and quality of life, with the belief that this leads to improved productivity and effectiveness [11].

In this paper, the “socio-technical” perspective is understood to refer primarily to the insight that a technical system implementation inevitably affects and is affected by the interdependent social system within which and upon which it operates [12, 13]. This concept has been used to formulate design principles [14] (see Box 1) and methods [15, 16] for information systems.

This approach requires a more nuanced worldview than the positivist mindset typically associated with reductionist physical science and “hard” technology. It utilizes perspectives such as constructivist or interpretive epistemologies and a social interactionist or negotiated view of reality [17] involving complex adaptive systems [18]. This paper argues that these factors are particularly crucial in HI and introduces a tool that supports adoption of a socio-technical approach in HI.

How does this Tool Aim to Help?

The organization and practice of healthcare is one of humanity’s most complex endeavours. Clinical work has been described as mostly “unpredictable and non-routine” [19] and “highly contingent, ad-hoc and idiosyncratic” [20]. Dealing with multiple uncertainties is a challenge for clinicians [21, 22], so arguably even more so for planners, designers or implementers of HI solutions [23, 24]. In particular, the NHS is a paradox: nominally a centrally-controlled monolith but in fact a coalition of competing confederacies of autonomous interests.

Clegg’s Socio-Technical Design Principles [14]

  1. Design is systemic.
  2. Values and mindsets are central to design.
  3. Design involves making choices.
  4. Design should reflect the needs of the business, its users and their managers.
  5. Design is an extended social process.
  6. Design is socially shaped.
  7. Design is contingent.
  8. Core processes should be integrated.
  9. Design entails multiple task allocations between and amongst humans and machines.
  10. System components should be congruent.
  11. Systems should be simple in design and make problems visible.
  12. Problems should be controlled at source.
  13. The means of undertaking tasks should be flexibly specified.
  14. Design practice is itself a socio-technical system.
  15. Systems and their design should be owned by their managers and users.
  16. Evaluation is an essential aspect of design.
  17. Design involves multidisciplinary education.
  18. Resources and support are required for design.
  19. System design involves political processes.

We argue that this irreducible complexity means that a socio-technical approach is vital to the success of innovations and transformations that are based on HI. The socio-technical approach would help to negotiate these complexities using principles such as minimal process specification, multi-stakeholder participation, user leadership, local adaptability and holistic evaluation [14].

Unfortunately, knowledge of socio-technical principles and their vital importance in HI does not seem to be widespread among practitioners. The socio-technical assessment tool for health informatics (STAT-HI) proposed here aims to help bridge the gap between research and practice by distilling key factors of good practice into a simple, readily accessible and applicable form.

METHODS

Checklist Format

The tool is expressed as a reflective checklist to validate the socio-technical soundness of an HI implementation approach. It does not aim to check every aspect of project fitness for purpose.

This approach seeks to utilize the ‘checklist effect’. Merely having a structured set of tasks or questions can in itself improve outcomes. This effect is so strong that it is a well-known source of bias in studies that compare interventions that use any checklist-based approach (whatever its merits) against those that do not [25]. Checklists have proved very powerful in improving aspects of clinical practice [26].

The checklist was conceived as a categorized summary of principles derived from a synthesis of critical success factors identified in the literature. It emulates the approach and certain principles of the AHRQ [27], GEP-HI [28] and STARE-HI [29] standards for designing and reporting system evaluation in health informatics.

Sources

The primary sources used were seminal UK reports on reasons for success or failure in major IT projects [30, 31] and a selection of HI studies [32-39] encountered in an ongoing literature review of health informatics theory by the present authors and colleagues [40]. This selection does not purport to be comprehensive or to form a systematic review of socio-technical principles in HI, but is presented as a starting point for discussion and development. The sources were selected on face validity as offering immediately relevant and directly applicable socio-technical assessment criteria.

The checklist emphasises particular elements of programme governance, cultural factors, the approach to changes in workflow and practice and the need for strong and balanced clinical leadership and governance. The headings echo the critical success factors identified by Whetton [17] (see Box 2) and Georgiou & Westbrook’s ten questions to consider before implementing computerized physician order entry (CPOE) [41] (see Box 3).

Whetton’s Critical Success Factors [17]

  1. A federal approach.
  2. Senior management support.
  3. Project champion.
  4. Project management.
  5. Project team membership.
  6. End-user input and participation.
  7. Change management.
  8. Communication.
  9. End-user skill development.

PROPOSAL

Table 1 shows the first published version of the STAT-HI checklist, comprising twenty items under the four headings of culture, governance, managing change and technology. The bracketed remarks are intended to illustrate and amplify the implied recommendation taken from the referenced source. Some checklist questions are taken verbatim from the cited source but most are paraphrases.

Table 1.

STAT-HI Checklist


Heading Ref Item Source
Culture 1.1 Does the programme/project give sufficient attention to social factors, ethical considerations and practical workflow issues?
(See principle 13 [14]).
[32-35, 37-39]
1.2 Does the business case and programme/project plan critically evaluate the relevant supporting evidence (or its absence) from the literature?
(Is the “treatment” evidence-based? [32])
[42]
1.3 Is there an open and constructive relationship with suppliers?
(Does the commercial contract make clear where functionality is constrained to a stated specification or what degree of flexibility there is for development or adaptation and how this is costed)?
[31]
1.4 Is the default assumption that the implementation will be evaluated using STARE-HI (or a similar structured methodology) and that the evaluation will be published?
(See principle 16 [14]).
[29]
1.5 Is there an appropriate balance between standardization and respect for local autonomy?
(See principle 13 [14]).
[32]
1.6 Is there an environment in which those who feel the programme/project is starting to go wrong feel able to say so and then get a proper hearing?
(See principle 2 [14]).
[30]
Governance and Risk 2.1 How clearly are the success criteria defined? How widely are they agreed?
(Does the business case include funding to backfill clinicians for project governance and workflow design activities?
Are the participating clinicians solely the enthusiasts, or have opponents been recruited to balance the discussions?
See principles 5 and 18 in [14]).
[30, 37, 39]
2.2 Has the programme been broken into manageable steps?
(Have all stakeholders agreed that the stages are manageable? See principle 5 in [14]).
[30]
2.3 Are there sufficient skills to deliver the full scope of the programme?
(Has the business case and programme/project plan been independently peer reviewed by qualified health informatics practitioners? See principle 18 in [14]).
[30]
2.4 Do the Senior Responsible Owner and programme/project manager have good relevant track records?
(For instance, are they registered with UKCHIP [43] at level 3)?
[30]
2.5 Does the risk management plan include assessment of how to address unforeseen consequences in workflow and emergent change following implementation?
(Is the programme alert for new kinds of errors, negative emotions, disruption to well-established communication patterns? See principle 1 in [14]).
[37, 39]
2.6 Is there a decision-making structure that will ensure strong and effective leadership of the IT-enabled business change?
(Do eventual system users and operational managers own the changes? See principle 15 in [14]).
[31]
2.7 Does the programme/project use an evolutionary approach with rapid learning cycles, recognizing that it is virtually impossible to fully understand and predict the behaviour of complex IT systems at the start of the project? [30, 32, 37, 39]
2.8 Beyond immediate technical success, how will wider benefits be secured? [31]
Managing change 3.1 Does the programme have a complete understanding of its current business processes and how its stakeholders interact both with the business and between themselves and a clear understanding of what it wants the new business process to achieve? [31, 37, 39]
3.2 What incentives exist to drive performance?
(Have clinicians agreed with the desired outcomes of the HI project, not seeing it as an end in itself that is being imposed upon them)?
[31]
[44]
3.3 Does the programme have a good appreciation of the likely impact of the business process change on service levels, productivity and different stakeholders? [31, 37, 39]
3.4 Does the technology exist to deliver the change?
(Is a feasibility study or pilot required to offer proof of concept or proof of technology)?
[31]
Technology 4.1 Does the design and the implementation plan include not just system functionality but how it will affect actual clinical workflow?
(Is there sufficient attention to ‘messy’ real-world practice rather than deceptively neat abstract models? [23-24])
[33-35, 37-39]
4.2 Will system performance be rapid and reliable enough for clinical usage? [36-39]

DISCUSSION

Suggested Application of STAT-HI

The STAT-HI checklist is presented as a tool for project/programme managers, project/programme boards and other professionals who provide external assurance and audit functions.

Georgiou & Westbrook’s Ten Key Questions for CPOE Implementation [41]

  1. What does the organisation/department expect to gain by introducing the new system?
  2. Who wants or needs this new technology and why?
  3. Which groups are most involved in the decision making about implementation and use?
  4. Will the system be technically compatible with current systems in use?
  5. Can it be tailored to fit the specific needs of professionals?
  6. How will the benefits of the new system be measured?
  7. What changes to work practices and processes are required?
  8. Are the lines of accountability for dealing with expected and unexpected problems clear?
  9. What are the drawbacks and risks of system implementation?
  10. Are they being addressed, and are there safeguards for dealing with problems?

It is suggested that a required reading list should be promulgated to HI practitioners alongside this checklist, comprising eight books, papers and electronic sources [30-32, 34, 37-39, 45].

The reading list supports the aim of the checklist to be reflective, rather than simply a tick-box exercise. The checklist is an opportunity to learn, to enlarge one’s worldview and to change practice. Schön [46] argued that reflective practice goes beyond problem-solving by considering problem setting: has it been ‘named and framed’ rightly, or is evidence emerging that it needs to be rethought?

Therefore we suggest that a reviewer (whatever their professional role) making an assessment using STAT-HI should give narrative justification for their considered responses to each of the questions, adducing data to substantiate their commentary as necessary to satisfy both their own ethical standard and the independent critical judgement of the relevant project/programme assurance bodies.

Is this Proposal Novel?

By definition, this checklist utilizes existing evidence and prior good practice recommendations. Its only claim to originality is the condensation and annotated checklist presentation of key guidance and the suggestion of a specific minimum knowledge base for HI practitioners.

Do Existing Methodologies Incorporate Socio-Technical Principles?

Arguably there are socio-technical principles embedded in numerous extant methods, even if not explicitly. Examples of this include the OGC programme management guidelines, Soft Systems Methodology and the ETHICS method.

OGC’s Managing Successful Programmes [45] acknowledges that change programmes involve ambiguity and the need to establish both organizational readiness for change and capability to manage and deliver change. Programmes are described as led by vision not specification. They are most effective when issues are debated freely and risks evaluated openly. The programme team needs staff with the relevant skills and experience to manage cultural and people issues involved in the business change.

Checkland’s Soft Systems Methodology is a process of enquiry that accepts and examines the multiple perceived realities of different stakeholders in a problem scenario. It envisages a group learning cycle that seeks to discover feasible improvements that can be accommodated within the continually re-negotiated social interpretations of individuals and organizations. Its philosophy directly rejects the idea that its conceptual models represent descriptions of an absolute reality, but regards “systems” as devices to think about the purposeful activities of various actors and thus formulate questions of the problem situation [47]. So if used for HI, the soft systems approach might in a sense be regarded as an application of socio-technical principles but with a more strongly constructivist interpretation of what we mean by “system”.

Mumford devised the ETHICS method (Effective Technical and Human Implementation of Computer-based Systems) as an explicitly socio-technical approach [15-16]. It emphasizes participatory diagnosis of problems, setting objectives for both efficiency and work satisfaction and ensuring compatibility of the technical and organizational systems. Mumford’s extensive socio-technical work seems to be little known outside academia, particularly since the 1980s [11].

Why are Socio-Technical Principles Not Routinely Followed in HI Projects?

Given the above, we echo Cobb’s paradox: "We know why projects fail, we know how to prevent their failure – so why do they still fail?" [30].

One factor is that health informatics and project management are not controlled professions with mandatory regulatory standards. Hence, the professional competence of some NHS HI project managers is limited to general IT project experience, which tends to bring the implicit but incorrect view that HI is "just another" industry sector. This has been another general criticism of the NPfIT “commodity” approach to HI. If anything, the unregulated nature of project management and health informatics means that their need for guidance is even greater than a formally controlled profession.

A converse aspect of the problem is information overload. There is simply so much guidance available on project and programme management in healthcare. In some respects this parallels medical practice, where for many years it has been recognized that practicing clinicians cannot be expected to keep track of every new piece of research evidence. Instead, important new knowledge is highlighted through summary publications, clinical guidelines and professional standards. In fact, the latency inherent in the medical profession is such that even solid research evidence only changes clinical practice slowly, patchily and when multiple interventions are used for reinforcement [38, 48].

By contrast, HI benefits and project management methodologies do not have a robust evidence base but are largely aspirational [42] or normative [49]. In fact there are legitimate grounds for caution with respect to HI, given the evidence that products are not equal [50], things can be made worse [51] and that the method of implementation can make all the difference to success or failure [34, 52].

At times HI is used instrumentally, beyond its functional purposes, to enforce “improvements” in ways of working as defined by the project sponsors [53]. This adds yet another layer of complexity and potential opposition, especially if the merits of the given changes are disputed.

Of course, the invidious effects of politics and commercial interests cannot be ignored in this context. The NHS is unavoidably part of the political game, given its funding and structure. A taxpayer-funded IT programme has legitimate expectations of optimum progress from both its political masters and its commercial suppliers. But when these interests seem to sacrifice good practice for the sake of apparent progress or simply meeting contractual milestones, then the taxpayers have sound reason for concern.

It must also be stressed that the anticipated benefits of planned change in healthcare systems, particularly when based on HI, are not deterministic. The inherent complexity of the socio-technical system in healthcare means that apparently minor changes can have profound unpredictable consequences [18, 37]. Indeed, it has been argued that any ICT “solution” actually adds complexity, and hence risk, by increasing the number of information and process interrelationships in the world [54].

Comparison with Organizational Readiness Guidance

NHS Connecting for Health (CFH), the English Department of Health agency responsible for NPfIT, publishes an “Organisational Readiness Assurance Guide” for HI planning [55]. This document seems to assume a linear model of change that comprises work process redesign, system deployment and establishing a new status quo. It does not address unintended consequences or complex adaptive change. Its subtitle “Are you ready?” reinforces the concept of the organization or individual as passively receptive to a “deployment”. It does suggest that new working practices and roles be subjected to an impact analysis and requires that both line managers and “stakeholders” agree to the new working practices and roles, but does not specify whether that includes the clinicians themselves. Arguably, its helpful direction on applying lessons learned from other HI implementations itself implies that the socio-technical perspective needs greater emphasis.

Limitations

The genesis and focus of this paper has been on HI in the UK. However, the essential principles discussed here would intuitively seem rather general across the global HI domain.

At best, this proposed tool can offer a selection of key messages and maxims to practitioners. This is significantly less than a formal theory of HI programme management, but is arguably an improvement on the status quo.

Obviously, this proposed tool will not remove political or contractual pressure to impose systems to meet arbitrary deadlines. However, it could provide a mechanism to create or channel balancing feedback pressure within or upon the governance structures of HI programmes where participants have concerns that good practice may be compromised by inappropriate commercial or governmental timescales.

Further Work

The suggested approach will be debated and the content and format tested with the UK Faculty for Health Informatics. If the tool seems to have face validity based on this expert consensus, the aim will be to apply it to current or recent HI projects to measure its utility. One extension may be to devise a numerically scored assessment rating. Further thought is needed about how in practice the rating would be conducted for it to be credible, and how any proposed remedial action would be validated. Further essential content for the checklist may be added from analysis of the recently published AMIA collection of case studies of failed HI projects [56].

CONCLUSIONS

There is a real need for improving socio-technical awareness among HI project teams. This tool offers a starting point to bridge the gap between research knowledge and routine practice.

Coiera described health informatics as the systems science of healthcare [57]. We argue that a socio-technical approach is fundamental to correct application of that science.

ACKNOWLEDGEMENTS

The authors gratefully acknowledge the support of the UK Faculty of Health Informatics in hosting the discussions that led to the inception of this paper. It should be noted that the Faculty is independent but its meetings are funded by CFH. The authors are grateful for the helpful comments from Faculty members and the anonymous reviewers.

DECLARATION OF INTEREST

PS is a Council member of UKCHIP and actively promotes the need for the professionalization of health informatics.

REFERENCES

[1] UK Department of Health. Delivering 21st Century IT: Support for the NHS National Strategic Programme. [electronic document] Available from: http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/documents/digitalasset/dh_4067112.pdf 2002 [cited 2007 October 15];
[2] Anonymous. Computers and the NHS Private Eye 2006 October 27;
[3] British Computer Society Health Informatics Forum. The Way Forward for NHS Health Informatics [electronic document] Available from: http://www.bcs.org/upload/pdf/BCS-HIF-report.pdf 2006 [cited 2009 December 15 ];
[4] Collins T. Government ordered to publish reviews of risky IT projects Computer Weekly [electronic document] Available from: http://www.computerweekly.com/Articles/2009/06/22/236526/government-ordered-to-publishreviews-of-risky-it-projects.htm 2009 June 22; [cited 2009 Nov 3];
[5] UK Office of Government Commerce. National Health Service - Integrated Care Records Services. Gateway 1 - Business Justification [electronic document] Available from: http://www.connectingforhealth.nhs.uk/about/foi/g1icrs0275.pdf 2002 October; [cited 2009 Nov 3];
[6] Peltu M, Eason K, Clegg C. How a sociotechnical approach can help NPfIT deliver better NHS patient care [electronic document] Available from: http://lubswww2.leeds.ac.uk/COSLAC/index.php?id=54 2008 [cited 2009 August 21];
[7] Greenhalgh T, Stramer K, Bratan T, et al. Summary care record early adopter programme: An independent evaluation by University College London [electronic document] Available from: http://www.ucl.ac.uk/openlearning/documents/scrie2008.pdf 2008 April 30; [cited 2009 Nov 3];
[8] Hanseth O, Jacucci E, Grisot M, Aanestad M. Reflexive integration in the development and implementation of an Electronic Patient Record system In: Hanseth O, Ciborra C, Eds. Risk complexity and ICT. Cheltenham: Edward Elgar 2007.
[9] Aarts J, Doorewaard H, Berg M. Understanding implementation: the case of a computerized physician order entry system in a large Dutch university medical center J Am Med Inform Assoc 2004; 11(3): 207-16.
[10] Trist E. The evolution of socio-technical systems: a conceptual framework and an action research program Occasional paper no 2 [electronic document] Available from: http://orca.ecn.purdue.edu/~grouper/papers/public/evolution_of_socio_technical_systems.pdf 1981 June; [cited 2009 10 November];
[11] Mumford E. The story of socio-technical design: reflections on its successes, failures and potential Inform Syst J 2006; 16(4): 317-42.
[12] Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care - an interactive sociotechnical analysis J Am Med Inform Assoc 2007; 14(5): 542-9.
[13] Coiera E. Four rules for the reinvention of health care BMJ 2004; 328(7449): 1197-9.
[14] Clegg C. Sociotechnical principles for system design Appl Ergon 2000; 31(5): 463-77.
[15] Mumford E. Systems design: ethical tools for ethical change. Basingstoke: Macmillan 1996.
[16] Mumford E. Redesigning human systems. Hershey, PA:: IRM Press 2003.
[17] Whetton S. Health Informatics: A socio-technical perspective. Oxford: OUP 2005.
[18] Plsek P. Redesigning healthcare with insights from the science of complex adaptive systems In: Committee on Quality of Health Care in America, Ed Crossing the quality chasm. Washington DC: National Academy Press 2001.
[19] Greenhalgh T. Implementing the National Programme for IT in the NHS: No easy answers: UK Faculty of Health Informatics 2008.
[20] Aarts J. On Articulation and Localization - Some Sociotechnical Issues of Design, Implementation, and Evaluation of Knowledge Based Systems In: Lecture Notes in Computer Science. Berlin: Springer 2001; 2101: pp. 16-9.
[21] Ghosh AK. Understanding medical uncertainty: a primer for physicians J Assoc Physicians India 2004; 52(9): 739-42.
[22] Moskowitz AJ, Kuipers BJ, Kassirer JP. Dealing with uncertainty, risks, and tradeoffs in clinical decisions. A cognitive science approach Ann Intern Med 1988; 108(3): 435-9.
[23] Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot JAMA 2005; 293(10): 1261-3.
[24] Wears RL, Berg M. Computers and clinical work-reply JAMA 2005; 294(2): 182-b.
[25] Friedman C, Wyatt J. Evaluation methods in medical informatics. London: Springer-Verlag 1997.
[26] Pronovost P, Needham D, Berenholtz S, et al. An intervention to decrease catheter-related bloodstream infections in the ICU N Engl J Med 2006; 355(26): 2725-32.
[27] Cusack C, Byrne C, Hook J, McGowan J, Poon E, Zafar A. Health Information Technology Evaluation Toolkit: 2009 Update Rockville, MD: Agency for Healthcare Research and Quality 2009 June; AHRQ Publication No. 09-0083-EF
[28] Nykänen P, Brender J, Ammenwerth E, et al. Guidelines for Good Evaluation Practices in Health Informatics (v06). [electronic document] Available from: http://iig.umit.at/efmi/gep-hi/GEP_HI_v016_Dec012008.doc 2008 [cited 2009 4 September];
[29] Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykanen P, Rigby M. STARE-HI--Statement on reporting of evaluation studies in Health Informatics Int J Med Inform 2009; 78(1): 1-9.
[30] Royal Academy of Engineering, British Computer Society. The challenges of complex IT projects [electronic document] Available from: http://www.bcs.org/upload/pdf/complexity.pdf 2004 [cited 2009 November 9];
[31] UK National Audit Office. Delivering successful IT-enabled business change [electronic document] Available from: http://www.nao.org.uk/publications/0607/delivering_successful_it-enabl.aspx 2006 [cited 2009 November 9];
[32] Ozdas A, Miller RA. Care provider order entry (CPOE): a perspective on factors leading to success or to failure Yearb Med Inform 2007; 128-37.
[33] Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness J Am Med Inform Assoc 1999; 6(2): 104-4.
[34] Berg M. Medical work and the computer-based patient record: a sociological perspective Methods Inform Med 1998; 37(3): 294-301.
[35] Goorman E, Berg M. Modelling nursing activities: electronic patient records and their discontents Nurs Inq 2000; 7(1): 3-9.
[36] Shabot MM. Ten commandments for implementing clinical information systems Proc Bayl Univ Med Cent 2004; 17(3): 265-9.
[37] Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry Int J Med Inform 2007; 76(Suppl 1): 21-7.
[38] Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality J Am Med Inform Assoc 2003; 10(6): 523-30.
[39] Physician Order Entry Team. POET Recommendations [electronic document] Available from: http://www.cpoe.org/ 2007 [cited 2009 November 11];
[40] Scott PJ, Briggs JS, Wyatt JC, Georgiou A. How Important is Theory in Health Informatics?. A Survey of the UK Health Informatics Academic Community Manuscript submitted for publication 2010.
[41] Georgiou A, Westbrook J. Computerised Order Entry Systems and Pathology Services - A Synthesis of the Evidence Clin Biochem Rev 2006; 27(2): 79-87.
[42] Clamp S, Keen J. Electronic health records: is the evidence base any use? In: Bryant J, Ed. Proceedings of Healthcare Computing Harrogate: BCS Health Informatics Forum. 2006.
[43] UK Council for Health Informatics Professions. Registration with UKCHIP [electronic document] Available from: http://www.ukchip.org/?q=page/Registration-with-UKCHIP 2009.
[44] Greenhalgh T, Stramer K, Bratan T, Byrne E, Mohammad Y, Russell J. Introduction of shared electronic records: multi-site case study using diffusion of innovation theory BMJ 2008; 337: a1786.
[45] UK Office of Government Commerce. Managing Successful Programmes London: OGC 2003.
[46] Schön D. The reflective practitioner: how professionals think in action Aldershot: Ashgate 1991.
[47] Checkland P, Poulter J. Learning for action: a short definitive account of soft systems methodology and its use for practitioners, teachers and students Chichester: John Wiley 2006.
[48] Wyatt J. Clinical knowledge and practice in the information age: a handbook for health professionals. London: RSM Press 2001.
[49] UK Office of Government Commerce In: Managing Successful Projects with PRINCE2. London: OGC 2005.
[50] Murff HJ, Kannry J. Physician satisfaction with two order entry systems J Am Med Inform Assoc 2001; 8(5): 499-509.
[51] Nebeker JR, Hoffman JM, Weir CR, Bennett CL, Hurdle JF. High rates of adverse drug events in a highly computerized hospital Arch Intern Med 2005; 165(10): 1111-6.
[52] Scott JT, Rundall TG, Vogt TM, Hsu J. Kaiser Permanente's experience of implementing an electronic medical record: a qualitative study BMJ 2005; 331(7528): 1313-6.
[53] Broadhurst K, Wastell D, White S, et al. Performing 'Initial Assessment': Identifying the Latent Conditions for Error at the Front-Door of Local Authority Children's Services Br J Soc Work [electronic document] Available from: http://bjsw.oxford journals.org/cgi/content/abstract/bcn162v1 2009 [cited 2009 11 December];
[54] Hanseth O. Integration-complexity-risk: the making of information systems out-of-control In: Hanseth O, Ciborra C, Eds. Risk complexity and ICT. Cheltenham: Edward Elgar 2007.
[55] NHS Connecting for Health. CFH Organisational Readiness Assurance Guide [electronic document] Available from: http://nww.connectingforhealth.nhs.uk/imtplanning/aguide.pdf 2008. [Only accessible within NHS network]
[56] Leviss J, Gugerty B, Kaplan B, Keenan G, Ozeran L, Rose E, Eds. H.T. or miss: lessons learned from health information technology implementations Chicago IL: AHIMA Press 2010.
[57] Coiera E. Guide to health informatics. 2nd. London: Hodder Arnold 2003.