Health care systems in Australia and across the world are facing major challenges in how to maintain the provision of quality care for all, in a manner that is economically sustainable. Ageing populations and the related growth in the incidence of chronic disease are contributing significantly to spiralling health costs. As a consequence system affordability has now become a major driver underpinning initiatives aimed at health care reform. Currently, no health care system has developed a comprehensive response to meet these challenges but there is agreement and support for new models of service delivery that can enhance the efficiency and effectiveness of the system.

Considering the complexity of challenges faced, it is perhaps not surprising that there are different perspectives on what needs to be done and by whom, how, where and when. In this context, debates on health service delivery and quality of care have become increasingly prominent. Unfortunately, these debates have tended to be fragmented with a variety of approaches advocated each based on different measures and types of evidence to support their claims of value. It can however be argued that developing comprehensive solutions to the challenges faced requires a more integrated approach.

“Most quality improvement activities in the world are still largely a reflection of the specific beliefs of specific parties about the best way to improve patient care. Crossing borders among professional pride and self-regulation, external accountability, payer profit, organisational development, and pleasing and involving patients can help us overcome the obstacles to optimal medical care” [1Grol R. Improving the quality of medical care: Building bridges among professional pride, payer profit, and patient satisfaction JAMA 2001; 286: 2578-86.].

In parallel with these developments, the widespread adoption and utilisation of information systems in health care has also been advocated as a major driver for stimulating the changes necessary to respond to the challenges faced.

“Both Australian and international studies support investment in health information management and information & communication technologies (IM&ICT) as a fundamental component of system reform… to improve health outcomes while helping to offset the health cost increases expected as a result of advances in medical technologies and pharmacology, and the inevitable aging of the population. In addition, improving the collection, management and dissemination of health information will boost administrators’ and policymakers’ ability to undertake informed, system-wide planning, resulting in more effective and efficient deployment of health resources” [2Boston Consulting Group. Australian National Health Information Management Information & Communications Technology Strategy Final Report B C Group, National Health Information Group (NHIG) and Australian Health Information Council (AHIC) on behalf of the Commonwealth Government of Australia 2004.].

By opening up opportunities for increased information access, improved information delivery, update and evaluation, the economic case for more information systems is strong. However, this deployment also raises a number of socio-technical, clinical and legal challenges that have become increasingly evident as the implementation of more sophisticated solutions have met with mixed success or failed to generate their forecast benefits [3Harrison M, Koppel R, Bar-Lev S. Unintended Consequences of Information Technologies in Health Care – An Interactice Sociotechnical Analysis J Am Med Inform Assoc 2007; 14(5): 542-9.]. This has led to calls for the utilisation of more sophisticated socio-technical approaches to the design, implementation and evaluation of health projects involving information systems [4Coiera E. Four rules for the reinvention of health care BMJ 2004; 328: 1197-99., 5Jadad AR, Delamothe T. What next for electronic communication and health care? BMJ 2004; 328: 1143-4.].

While debates on the role and impacts of different approaches are likely to continue, from a methodological perspective finding an integrated way to meaningfully evaluate the different types of evidence they use, may contribute to supporting more effective solutions to be developed. To explore these methodological issues, this paper considers how contemporary clinical and IS evaluation discourses on health interventions approach patients. More specifically, the paper focuses on the socio-technical processes by which patients’ interests and outcomes are measured, defined and evaluated within health interventions that involve them using web-based information systems. The paper commences with a critical appraisal of the limitations of current clinical and IS evidence and evaluation processes of patients involved in these types of health interventions. The paper proceeds to outline an integrated approach that aims to generate evidence about the impact of these types of health interventions that are meaningful at both individual patient and patient cohort levels.

With the increasing incidence of chronic illness, health interventions encouraging people with a chronic disease to take a more active role in managing their own condition have become popular [6Coleman MT, Newton KS. Supporting self-management in patients with chronic illness Am Fam Physician 2005; 72: 1503-0.,7Downs JA, Roberts CM, Blackmorel AM, Le Souef PN, Jenkins SC. Benefits of an education programme on the self-management of aerosol and airway clearance treatments for children with cystic fibrosis Chron Respir Dis 2006; 3(1): 19-27.]. To date the evaluation of these interventions has focussed either on the impact of self-management upon health outcomes or on health resource usage and has been based on evidence produced through randomised controlled trials (RCTs). The evidence produced suggests that these health interventions can be as effective as the introduction of new medications [8Lorig K, Ritter P, Stewart A, et al. Chronic disease self-management program: 2-year health status and health care utilization outcomes Med Care 2001; 39(11): 1217-23.]. However, there have been concerns raised about limitations of only evaluating the impact of interventions involving patients through the quantitative techniques utilised in RCTs [9Grossman J, Mackenzie FJ. The randomized controlled trial: gold standard, or merely standard? Perspect Biol Med 2005; 48(4): 516-34., 10Bluhm R. From Hierarchy to Network: a richer view of evidence for evidence-based medicine Perspect Biol Med 2005; 48(4): 535-47.].

The notion that all interventions within the medical field are comparable to a drug intervention and should therefore be evaluated in the same way, uncritically privileges evidence generated through randomised controlled trials over evidence generated in other ways [9Grossman J, Mackenzie FJ. The randomized controlled trial: gold standard, or merely standard? Perspect Biol Med 2005; 48(4): 516-34., 10Bluhm R. From Hierarchy to Network: a richer view of evidence for evidence-based medicine Perspect Biol Med 2005; 48(4): 535-47.]. The persistent claims that RCTs are the ‘gold standard’ of health intervention evaluation results primarily because of the dominance of the scientific research paradigm within which healthcare exists [11Holmes D, Murray SJ, Perron A, Rail G. Deconstructing the evidence-based discourse in health sciences: truth, power and fascism Int J Evid Based Heathc 2006; 4(3): 180-6.]. RCTs merely reflect good experimental design, however there are many potential shortcomings in the design and implementation of RCTs [9Grossman J, Mackenzie FJ. The randomized controlled trial: gold standard, or merely standard? Perspect Biol Med 2005; 48(4): 516-34., 10Bluhm R. From Hierarchy to Network: a richer view of evidence for evidence-based medicine Perspect Biol Med 2005; 48(4): 535-47.]. Indeed the dominance of RCTs as ‘the method’ for evaluating health interventions has begun to attract some significant criticism [12Smith G, Pell J. Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials Int J Prosthodont 2006; 19(2): 126-8.] and the utility and validity of other approaches for use in evaluating health interventions have begun to emerge [13Borgerson K. Evidence-based alternative medicine? Perspect Biol Med 2005; 48(4): 502-15.].

Major proponents of the RCT as the ‘gold standard’ for the evaluation of health interventions are advocates of evidence based medicine (EBM). EBM has the laudable goal of promoting the improvement of patient care through the use of up-to-date high quality, research evidence to make healthcare decisions. However, implicit within the EBM approach is a hierarchy of evidence that places evidence from RCTs and other population level research above other types of research evidence [10Bluhm R. From Hierarchy to Network: a richer view of evidence for evidence-based medicine Perspect Biol Med 2005; 48(4): 535-47.]. For health interventions, particularly those that engage the patient as an active participant, this hierarchy is of concern, as it prioritises population level evidence and reduces emphasis upon individuals. Related to this is the reality that health professional member organisations frequently base their development of clinical practice guidelines on the evidence from RCTs and systematic reviews. While not mandatory, these guidelines often place significant pressure upon individual health professionals to adhere to them, particularly for medico-legal reasons [14Timmermans S. From autonomy to accountability: the role of clinical practice guidelines in professional power Perspect Biol Med 2005; 48(8): 490-501.]. RCTs frequently concentrate on only a small number of measurable outcomes and may ignore patients’ experiences and/or any unexpected effects from interventions that are more difficult to quantify or measure easily. During extended RCTs it is also difficult to isolate and evaluate any single intervention to avoid outside factors adversely affect the stability of the intervention [15Øvretveit J. Evaluating health interventions. Buckingham: Open University Press 1998.].

More broadly, some researchers have raised questions about the dependence of RCTs upon their funding sources to produce outcome bias. This research was based upon a survey of published RCTs between 1990 and 2000 [16Ridker PM, Torres J. Reported outcomes in major cardiovascular clinical trials funded by for-profit and not-for-profit organizations: 2000-2005 JAMA 2006; 295(19): 2270-4.]. It revealed RCTs funded by for-profit organisations more frequently reported positive outcomes than those funded by not-for-profit organisations. These results raise questions around the validity of trial design and ethics. Perhaps most importantly they raise concerns in relation to the quality of the evidence that may now be used to treat patients.

However, aligned to this point is the reality check that even where evidence is produced through RCTs, gaps continue to exist between this evidence and its actual use in clinical practice [17Zuiderent-Jerak T. Preventing implementation: exploring interventions with standardization in healthcare Sci Cult 2007; 16(3): 311-29.]. Others research have argued that where RCTs evidence is used there is a need for more effective mechanisms to tailor this evidence to individual patients [18Olsson J, Terris D, Elg M, Lundberg J, Lindblad S. The one-person randomized controlled trial Qual Manag Health Care 2005; 14(4): 206-16.]. Perhaps of more relevance here however, have been critiques of how RCTs tend to marginalise individual patient experiences, priorities and perceptions [19Muir Gray JA. Evidence based policy making BMJ 2004; 329: 988-., 20Berwick DM. Crossing the quality chasm: health care for the 21st Century Washington DC; updated 2004 (www.iom.edu/event.asp?id=12580) [cited 18/03/2004];]. Most importantly however, the point needs to be made that whilst the goals of RCTs are laudable the practice, particularly where the interventions involve the patients themselves, continue to leave much to be desired. As has been identified:

“Although evidence-based medicine advocates laudably took improving patient care as their goal, the methods they use to define and measure outcomes remained firmly in specialist hands. The collaborations set up by Cochrane and other similar groups were partnerships between researchers and clinicians, not doctors and patients…the most easily measured aspects of physiological change and therapeutic efficacy did not necessarily capture the markers of recovery that patients themselves deemed most important” [21Tomes N. Patient empowerment and the dilemmas of late-modern medicalisation Lancet 2007; 369(9562): 698-700.].

It should be emphasized that the intention of this critical appraisal is not to present a unilateral argument for the discarding of RCTs but rather to highlight that there are some limitations that need to be considered. This is particularly the case where health interventions aim to actively involve patients and aim to generate evidence that meaningfully captures outcomes that reflect individual and cohort impacts [22Shumway M, Chouljian TL, Rozewicz F. Paraphrase procedures for assessing comprehension of health outcome measures: an illustration from schizophrenia research Eval Health Prof 2003; 26(1): 73-85.]. The dominance of RCTs as a means of evaluating health care interventions continues to create a conflict between statistical evidence and the individual patient experience that needs to be more actively addressed if we are not to make the term patient-centred increasingly meaningless.

Schneider and Lane [50Schneider S, Lane J. The patient from hell: How i worked with my doctors to get the best of modern medicine and how you can too. Cambridge, MA: Da Capo Press 2005.] have argued:

“Too often, a patient's treatments are performed "by the book" rather than being tailored to the patient's specific needs. To put it more bluntly, much of the care that today's patients receive is "medicine by numbers," which is best suited for that mythical abstraction, the "statistically average patient"”.

Health interventions involving patients using information systems present challenges to conventional clinical and information system approaches to the generation and evaluation of evidence on impacts and outcomes.

From a clinical perspective the analysis of discrete sets of independent variables within RCTs is purposeful but tends to provide evidence of changes only at the cohort level of those involved in the health intervention. This is despite knowledge of the fact that the interplay of psychological and social factors directly impact upon an individual’s self-perceived health status during their participation in clinical interventions [9Grossman J, Mackenzie FJ. The randomized controlled trial: gold standard, or merely standard? Perspect Biol Med 2005; 48(4): 516-34.-11Holmes D, Murray SJ, Perron A, Rail G. Deconstructing the evidence-based discourse in health sciences: truth, power and fascism Int J Evid Based Heathc 2006; 4(3): 180-6., 15Øvretveit J. Evaluating health interventions. Buckingham: Open University Press 1998., 18Olsson J, Terris D, Elg M, Lundberg J, Lindblad S. The one-person randomized controlled trial Qual Manag Health Care 2005; 14(4): 206-16., 19Muir Gray JA. Evidence based policy making BMJ 2004; 329: 988-., 21Tomes N. Patient empowerment and the dilemmas of late-modern medicalisation Lancet 2007; 369(9562): 698-700.].

From an information systems perspective user-centred approaches do involve the ‘user’ but may miss the patient! Focusing on technological acceptance or usability is sensible but it may often fail to examine in any detail the interplay of personal factors that make up an individual patients technology experiences, attitudes and responses. Further, in the realm of patients’ as ‘users’ the implicit assumption of mandatory use must be discarded as individual patients adopt and use systems in ways that suit themselves and their lifestyles and not necessarily as anticipated by clinicians or information systems professionals [23Stoop AP, Berg M. Integrating Quantitative and qualitative methods in patient care information systems evaluation: guidance for the organizational decision maker Methods Inf Med 2003; 4: 458-62., 24Cummings E, Turner P. Considerations for deploying web and mobile technologies to support the building of patient self-efficacy and self-management of chronic illness In: Al-Hakim L, Ed. Web mobile-based applications for healthcare management: Idea Group, Inc. 2007., 26Stoop A, Heathfield H, de Mul M. Evaluation of patient care information systems In: Berg M, Ed. Health information management: integrating information technology in health care work. London: Routledge 2004., 29Kushniruk A, Borycki E, Kuwata S, Ho F. Emerging approaches to evaluating the usability of health information systems In: Kushniruk A, Borycki E, Eds. Human, social, and organisational aspects of health information systems. Hershey: IGI Global 2008., 31Gustafson DH, Wyatt JC. Evaluation of ehealth systems and services -We need to move beyond hits and testimonials BMJ 2004; 328(7449): 1150., 33Al-Busaidi A, Gray A, Fiddian N. Personalizing web information for patients: linking patient medical data with the web via a patient personal knowledge base Health Informatics J 2006; 12(1): 27-39., 45Nøhr C, Boye N. Towards computer supported clinical activity: a roadmap based on empirical knowledge and some theoretical reflections In: Kushniruk A, Borycki E, Eds. Human, social, and organizational aspects of health information systems. Hershey: IGI Global 2008; pp. 67-83., 49Cummings E, Chau S, Turner P. Assessing a patient-centered e-health approach to chronic disease self-management In: Wilson EV, Ed. Patient-Centered E-Health. Hershey PA: Medical Information Science Reference 2009.].

From a methodological perspective the tendency amongst these researchers to treat epistemology and method as being inseparable is evident [51Howe KR. Getting over the quantitative-qualitative debate Am J Educ 1992; 100(2): 236-56., 52Bryman A. The debate about quantitative and qualitative research: a question of method or epistemology? Br J Sociol 1984; 35(1): 75-92.]. This is despite an awareness of the utility of employing a greater level of pragmatism when engaged in research with human participants [53Onwuegbuzie AJ, Leech NL. On becoming a pragmatic researcher: the importance of combining quantitative and qualitative research methodologies Int J Soc Res Methodol 2005; 8(5): 375 -87.]. Olson [54Olson H, Ed. Qualitative "versus" quantitative research: the wrong question. CAIS/ACSI 95, Annual Conference of the canadian association for information science, connectedness: Information, Systems, People, Organizations; Edmonton, Alberta; June 7-10, 1995; School of Library and Information Studies, University of Alberta. ] argues that the distinction between qualitative and quantitative data is purely a distinction of data types and does not predict how these data should be treated and that research philosophy is of more importance in determining how data is analysed than the data type per se. Therefore in generating and evaluating evidence from health interventions involving patients use of web-based information systems, a case can be argued for analysing numeric and non-numeric data in an effort to ‘keep the patient at the centre’ [55Bhaskar RA, Ed. The possibility of naturalism. London: Routledge 1979.].

Of course, working through the implications of these points is much more difficult in practice than in theory. From a patient’s perspective the huge array of web-based health resources poses its own problems including differentials amongst individual patients to be able to categorise or recognise the validity and reliability of the content provided [33Al-Busaidi A, Gray A, Fiddian N. Personalizing web information for patients: linking patient medical data with the web via a patient personal knowledge base Health Informatics J 2006; 12(1): 27-39.]. Although these resources may improve patient knowledge, their effect on objective clinical outcomes is inconsistent [56Bussey-Smith KL, Rossen RD. A systematic review of randomized control trials evaluating the effectiveness of interactive computerized asthma patient education programs Ann Allergy Asthma Immunol 2007; 98(6): 507-16., 57Välimäki M, Nenonen H, Koivunen M, Suhonen R. Patients’ perceptions of Internet usage and their opportunity to obtain health information Inform Health Soc Care 2007; 32(4): 305-14.]. It is also important to be realistic about the capacity of patients’ to embrace web-based information systems, as financial and skill-based barriers may inhibit access to on-line information. These issues relate to broader concerns about how to maintain a patient focus in health interventions in order to avoid the creation of an ‘eHealth divide’ [49Cummings E, Chau S, Turner P. Assessing a patient-centered e-health approach to chronic disease self-management In: Wilson EV, Ed. Patient-Centered E-Health. Hershey PA: Medical Information Science Reference 2009.]. In the era of rapidly ageing populations and growing chronic illness it is also sobering to consider that it may continue to be those individuals with the most limited access to resources (financial, educational etc. and including Internet access) who are further marginalised by these types of health interventions.

For clinical and information systems researchers these types of health interventions may also challenge some deeply held beliefs about ‘what we think we are doing when we intervene’ versus how this is perceived by our patients (participants). If researchers are really serious about ‘keeping the patient at the centre’ then the reality of discontinued use of these types of health interventions must be faced. Otherwise there is the risk of ending up trying to replace a dependency on health professionals for a dependency on information systems and the professionals who implement them [58Wyatt S, Thomas G, Terranova T. They came, they surfed, they went back to the beach. Conceptualising use and non-use of the Internet In: Woolgar S, Ed. Virtual Society?. Oxford: Oxford University Press 2002.]. There is a need to maintain an awareness of the fact that patients may be unwilling or unable to take on the new responsibilities of ‘empowerment’ and that even if they do, that this will axiomatically lead to improved quality of care whether evaluated at the individual or cohort level [49Cummings E, Chau S, Turner P. Assessing a patient-centered e-health approach to chronic disease self-management In: Wilson EV, Ed. Patient-Centered E-Health. Hershey PA: Medical Information Science Reference 2009.]. To date, evidence on the impact of attempts to ‘empower’ patients, and the actual impact this has on care practices and outcomes remains limited [59Robinson AL, Courtney-Pratt HM, Lea EJ, et al. Transforming clinical practice amongst community nurses: mentoring for COPD patient self-management J Clin Nurs 2008; 17: 370-9.]. Indeed, despite the rhetoric, it appears evident that the impact of the availability of web-based clinical and other health information on patient-doctor relationships is mixed and dependent on a range of other factors that are not related to information or the systems that make it accessible [21Tomes N. Patient empowerment and the dilemmas of late-modern medicalisation Lancet 2007; 369(9562): 698-700., 60Nwosu CR, Cox BM. The impact of the Internet on the doctor-patient relationship Health Inform J 2000; 6(3): 156-61.].

Using this methodology in both of the health interventions led to the generation of some very interesting results. In both instances phase one analysis based on conventional statistical analysis of the cohort clearly and unequivocally revealed that the web-based information system did not have a positive effect and there was the possibility of some minor negative effect. This evidence was common across all statistical measures deployed. However, analyses generated from phases two and three revealed a more complex account of the intervention and the impacts on user experiences, capacities for behavioural change, and the influence of individual disease knowledge and stage.

More specifically, it was highlighted that at the individual level there were a highly varied set of reactions to the introduction of the web-based information systems that were not revealed in phase one analysis. These reactions result from the inter-relationships between many individual factors including the severity of disease, self-image, family context, confidence, social support and disease understanding.

The results demonstrated many different experiences and unintended consequences including a participant with severe disease, who lived alone, who reported a transformation of their life through the use of solitaire, email and changed family status. For this individual patient, this did lead to an upward trend in symptoms for some time even though this trend was not statistically significant. This participant also continued to use the web-based information system for the duration of the project and found some benefit from this in terms of managing their disease and also considered that it made a difference to them in terms of quality of life. Another participant, who initially had elected to adopt the web-based information system, did not manage to use the system or the computer at all during the intervention. This participant had a moderately severe chronic condition and was primary carer for her husband who had a similar but severe chronic condition. Her husband was extremely unsupportive of her participation in the health intervention and refused to allow her access to further training in the use of the web-based information system. Despite these obstacles this participant persevered and acquired a good understanding of the aims of the health intervention and improved her self-efficacy for self-management through her participation [61Cummings E. An investigation of the influence of an online patient diary on the health outcomes and experiences of people with chronic obstructive pulmonary disease (COPD) participating in a mentored self-management clinical controlled trial [PhD]. University of Tasmania: Hobart 2008.].

This paper has considered how contemporary clinical and IS evaluation discourses on health interventions approach patients. More specifically, the paper has focused on the socio-technical processes by which patients’ interests and outcomes are measured, defined and evaluated within health interventions that involve them using web-based information systems.

The paper has presented a critical appraisal of the limitations of current clinical and IS evidence and evaluation processes in relation to patients involved in these types of health interventions. It has also outlined an integrated approach to evaluation that has been used to generate evidence about the impact of these types of health interventions. This approach has produced evidence about the impact of the intervention and the web-based information systems that are meaningful at both individual patient and patient cohort levels. The integrated approach outlined encourages the utilisation of both quantitative and qualitative data across cohorts, groups and individuals to build a detailed view of participants’ experiences and outcomes within this type of health intervention. By combining the analysis of RCT data with interview data this approach allows for evidence to be generated at multiple levels of abstraction that more readily supports ‘keeping the patient at the centre’ in evaluation.

As the number of health interventions directly involving patients and their use of information systems increase it is anticipated that the approach outlined will contribute to more comprehensive evaluations of the impact of interventions on patients and to the accommodation of perspectives that patients themselves recognise, acknowledge and deem relevant.