Process-supporting information technology holds the potential to increase efficiency, reduce errors, and alter professional roles and responsibilities in a manner which allows improvement in the delivery of patient care [1Fonkych K, Taylor R. The state and pattern of health information technology adoption, Rand Corporation, Santa Monica 2005.]. However, despite large-scale investments in Information Communication Technology (ICT), uptake has been slow [1Fonkych K, Taylor R. The state and pattern of health information technology adoption, Rand Corporation, Santa Monica 2005., 2Ash JS, Gorman PN, Seshadri V, et al. Computerized physician order entry in U.S. hospitals: results of a 2002 survey J Am Med Inform Assoc 2004; 11(2 ): 95-.] whilst recent evidence of ICT-induced errors has called for a more critical examination of the impact of such technology in healthcare [3Koppel R, Metlay J, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors J Am Med Assoc 2005; 293(10 ): 1197-203.-6Perrow C. Normal accidents: Living with high-risk technologies 1999.].

Grounded in both social theory and Information Technology (IT), sociotechnical approaches to ICT evaluation focuses on the interrelation between technology and its social environment [7Berg M, Aarts J, Van Der Lei J. ICT in health care: sociotechnical approaches Methods Inform Med 2003; 42(4 ): 297-301.]. Over time, ICT implementation models have developed an emphasis on the alignment between the clinical context where clinical work is carried out and the technology designed to improve the delivery of care [8Goodhue D, Thompson R. Task-technology fit and individual performance MIS Q 1995; 00: 213-36.-10Ammenwerth E, Iller C, Mahler C, Ammenwerth E, Iller C, Mahler C. IT-adoption and the interaction of task, technology and individuals: a fit framework and a case study BMC Med Inf Decis Mak 2006; 6: 3-0.], as well as the contextual differences both between organisations and within – at departmental and individual levels [11Callen J, Braithwaite J, Westbrook J. Contextual implementation model: a framework for assisting clinical information system implementations J Am Med Inform Assoc 2008; 15(2 ): 255-62.]. This paper will propose a methodology for the sociotechnical evaluation of the impact of a computerised information system on a clinical care department.

Contemporary health organisations are complex, composed of deeply interdependent and interrelated social and technical elements where changes to one aspect will affect the other [12Wears RL, Berg M. Computer technology and clinical work: still waiting for godot JAMA 2005; 293(10 ): 1261-3.]; the introduction of any new technology into a setting will entail innovation in clinical roles, work processes, and culture change [13Coiera E. Four rules for the reinvention of health care BMJ 2004; 328(7449 ): 1197-9.-15Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: Findings from a mixed methods exploration Int J Med Inform 2009; 78(Suppl1): S69-76.] whilst the attitudes and utility of technology is simultaneously socially shaped [13Coiera E. Four rules for the reinvention of health care BMJ 2004; 328(7449 ): 1197-9.]. Rather than viewing computer systems as isolated entities within their intended clinical environment, the contemporary perspective portrays systems as an active component of the clinical team, constantly cooperating and dynamically interacting with the clinical staff and organisational routines [12Wears RL, Berg M. Computer technology and clinical work: still waiting for godot JAMA 2005; 293(10 ): 1261-3., 16Anderson J, Aydin C. Evaluating the organizational impact of healthcare information systems 2005.].

The sociotechnical approach is one which seeks to identify the dynamics between technology and the social, professional, and cultural environment in which it is used [17Whetton S, Ed. Health informatics: a socio-technical perspective Melbourne: Oxford University Press 2005.]. The limitations of a purely technocentric approach to systems evaluation became apparent when early implementation projects were less than successful and purported benefits of ICT rarely realised [17Whetton S, Ed. Health informatics: a socio-technical perspective Melbourne: Oxford University Press 2005.-19Littlejohns P, Wyatt JC, Garvican L. Evaluating computerised health information systems: hard lessons still to be learnt BMJ 2003; 326(7394 ): 860-3.]. Causes of implementation failure extend beyond poor system design to the erroneous perceptions and theories about how medical work is conducted [12Wears RL, Berg M. Computer technology and clinical work: still waiting for godot JAMA 2005; 293(10 ): 1261-3., 20Aarts J, Gorman P. IT in health care: sociotechnical approaches "To Err is System" Int J Med Inform 2007; 76(Suppl 1): S1-3., 21Aarts J, Ash J, Berg M. Extending the understanding of computerized physician order entry: Implications for professional collaboration, workflow and quality of care Int J Med Inform 2007; 76: 4-13.]. Clashes between the model of health care work inscribed in these tools with the actual nature of work has resulted in staff resistance and decreased organisational uptake of ICT, as well as the facilitation of unexpected and negative effects in efficiency and patient safety [3Koppel R, Metlay J, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors J Am Med Assoc 2005; 293(10 ): 1197-203., 4Han Y, Carcillo J, Venkataraman S, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system Pediatrics 2005; 116(6 ): 1506-2., 15Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: Findings from a mixed methods exploration Int J Med Inform 2009; 78(Suppl1): S69-76., 18Berg M. Health information management: integrating information technology in health care work. Routledge, London, 2004.]. Sociotechnical theory provides a paradigm against which workflow and perfusion of ICT in healthcare could be better explored and understood.

Any form of ICT evaluation in health may be distinguished as either summative or formative. Summative evaluations determine the overall quality or value of an intervention and have traditionally followed the objectivist view which dictates that the merits and worth of ICT should and could be quantified [18Berg M. Health information management: integrating information technology in health care work. Routledge, London, 2004., 22Scriven M. The logic and methodology of evaluation, Edgepress, Inverness.CA 1981.]. In healthcare, such evaluations of ICT are conducted post hoc to account for the impacts of the system such as financial benefits or system effectiveness in terms of clinical outcomes [18Berg M. Health information management: integrating information technology in health care work. Routledge, London, 2004.]. Dominating the literature on health ICT evaluation, ICT is often conceptualised as the “intervention” in summative studies which allude, as in medicine, to the Randomised Controlled Trial (RCT) model of evaluation as the “gold standard” of evaluation [18Berg M. Health information management: integrating information technology in health care work. Routledge, London, 2004., 23Kaplan B. Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism Int J Med Inform 2001; 64(1 ): 39-56.].

An issue with ICT evaluation in health is the complexity of the evaluation object [24Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of health information systems—problems and challenges Int J Med Inform 2003; 71(2-3 ): 125-35.]. The deep integration of ICT with the culture and work practices of the healthcare organisation requires an evaluative focus which expands beyond only software and hardware to the surrounding clinical situation and associated processes to capture the diverse and diffuse nature of system effects in the healthcare setting [24Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of health information systems—problems and challenges Int J Med Inform 2003; 71(2-3 ): 125-35.]. Whilst RCT-type studies are valuable for determining the cause and impact of a pre-specified effect, it assumes possibility of isolating system functioning from the social processes that surround it, whilst reality suggests that the design, implementation, and operation of ICT are deeply influenced by organisational and social factors [23Kaplan B. Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism Int J Med Inform 2001; 64(1 ): 39-56.].

Formative evaluations are conducted during the development of a product for the purpose of system improvement [22Scriven M. The logic and methodology of evaluation, Edgepress, Inverness.CA 1981.]. The increase in qualitative studies of health ICT has exhibited a shift towards the subjectivist theoretical orientation to systems evaluation which asserts the responsibility of social and organisational aspects in shaping the outcomes of objective measurement and endorsing a “multimethod” approach for both summative and formative evaluations [18Berg M. Health information management: integrating information technology in health care work. Routledge, London, 2004.]. The mixed or multimethod approach to health ICT evaluation is one which views qualitative and quantitative research methods as complementary [23Kaplan B. Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism Int J Med Inform 2001; 64(1 ): 39-56., 25Westbrook JI, Braithwaite J, Georgiou A. Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory J Am Med Inform Assoc 2007; 14(6 ): 746-55., 26Stoop AP, Berg M. Integrating quantitative and qualitative methods in patient care information system evaluation: guidance for the organizational decision maker Methods Inform Med 2003; 42(4 ): 458-62.]; the combination of methods allows exploration of the what, why, and how of a social phenomenon that qualitative methods can address, and the size, extent or duration (how much) of certain phenomena that quantitative methods establish [26Stoop AP, Berg M. Integrating quantitative and qualitative methods in patient care information system evaluation: guidance for the organizational decision maker Methods Inform Med 2003; 42(4 ): 458-62.]. Pleiotropic healthcare interventions such as ICT create a hybrid sociotechnical situation requiring a range of integrated multi-perspective research methods to achieve a holistic understanding [23Kaplan B. Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism Int J Med Inform 2001; 64(1 ): 39-56., 27Scott P, Briggs J. A pragmatist argument for mixed methodology in medical informatics J Mixed Methods Res 2009; 3(3 ): 223-0., 28Kaplan B, Shaw N. Future directions in evaluation research: people, organizational, and social issues Methods Inform Med 2004; 43(3 ): 215-31.].

ICT is also a dynamic and emerging process, continually evolving across the multiple phases of the implementation process [11Callen J, Braithwaite J, Westbrook J. Contextual implementation model: a framework for assisting clinical information system implementations J Am Med Inform Assoc 2008; 15(2 ): 255-62., 23Kaplan B. Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism Int J Med Inform 2001; 64(1 ): 39-56.]. A longitudinal study design which follows the target of study over a substantial period of time will allow appreciation of this aspect by providing a means of monitoring changes caused by the system within the context it operates [22Scriven M. The logic and methodology of evaluation, Edgepress, Inverness.CA 1981., 24Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of health information systems—problems and challenges Int J Med Inform 2003; 71(2-3 ): 125-35.].

This paper will conceptualise a formative, multi-method longitudinal evaluation process to explore the impact of ICT with an appreciation of the relationship between the social and technical systems within a clinical department.

The sociotechnical approach to ICT evaluation appreciates the highly complex healthcare setting and emphasises the relationship between the social and technical impacts of clinical information systems. ICT represent technical artefacts which embody the implicit theories about how work is conducted; success of the artefact in real clinical situations relies upon the accuracy and applicability of these theories [41Woods D. Designs are hypotheses about how artifacts shape cognition and collaboration Ergonomics 1998; 41(2 ): 168-73.]. ICT evaluation tools [42Nykänen P, Brender J, Ammenwerth E. Guidelines for Good Evaluation Practices in Health Informatics (v0.16) 2008 [cited 2009 Jan 5]: Available from: http://iig.umit.at/efmi/gep-hi/GEP_HI_v016_Dec012008.doc. , 43Cusack C, Byrne C, Hook J, McGowan J, Poon E, Zafar A. Health information technology evaluation toolkit: 2009 Update. Rockville, MD: Agency for Healthcare Research and Quality Report No.: AHRQ Publication No. 09-0083-EF. June 2009 2009.] exist to guide the process of planning and execution of evaluation projects, with which the proposed protocol may be further developed and fine-tuned to accommodate individual settings; the methodology outlined in the paper supplements existing work in its offering of a unique, practical evaluation procedure which combines the theory and practice of the sociotechnical perspective to attain a more holistic understanding of the impacts of ICT and to ultimately achieve successful implementation.

However, sociotechnical theory does not exist without criticism. Examination of both the social and technical systems within a clinical department seeks to determine the different contextual factors operating in separate settings in order to identify the latent mechanisms that can influence outcomes [18Berg M. Health information management: integrating information technology in health care work. Routledge, London, 2004.]. Berg [44Berg M. Patient care information systems and health care work: a sociotechnical approach Int J Med Inform 1999; 55(2 ): 87-101.] cautions that understanding the interdependency of the social and technical systems within a particular context is insufficient for reconciling the social and technical aspects within that environment. Similarly, multi-method approaches to evaluation could potentially produce contradictory findings which do not, in fact, facilitate mutual interpretation of results produced from the separate methods [28Kaplan B, Shaw N. Future directions in evaluation research: people, organizational, and social issues Methods Inform Med 2004; 43(3 ): 215-31.]. The proposed methodology risks inheriting such limitations; efficacy of the process remains to be verified through independent field testing.

Sociotechnical theory provides a paradigm against which workflow and transfusion of ICT in healthcare could be better explored and understood. Incorporating both interactionist and ethnomethodological methods, this paper has outlined a formative, multi-method, longitudinal model for ICT evaluation which aims to inform interpretation of objective system effects in light of how the IT is conceptualised and viewed by users, as well as its impact on staff work practices within the unique context of the clinical department. However, inherent in the protocol are limitations of sociotechnical theory and criticism of the multimethod approach; testing of the methodology in real clinical settings will serve to verify efficacy and refine the process.