A dashboard is a visualizing tool which provides awareness, trending, and both planning and actual comparisons, frequently visualized in a slick simplified user interface. With so many dashboards available, choosing an appropriate one often boils down to the users of dashboard tools [1Dashboard OR, Ed. Ed. karami M Dashboard Operational Radiology, Ed. 28^th Iranian congress of Radiology. Iran. 2012].

This intellectual and visual tool has been used in healthcare fields, for example, 1) in the operation roomsto allocate resources and cost management, [2Park KW, Smaltz D, McFadden D, Souba W. The operating room dashboard. J Surg Res 2010; 164(2): 294-300.
[http://dx.doi.org/10.1016/j.jss.2009.09.011] [PMID: 20006345] ] in radiology departments to improve dose management and X-ray usage, [3Morgan MB, Branstetter BF, Lionetti DM, Richardson JS, Chang PJ. The radiology digital dashboard: effects on report turnaround time. J Digit Imaging 2008; 21(1): 50-8.
[http://dx.doi.org/10.1007/s10278-007-9008-9] [PMID: 17334871] ] in emergency departments to decrease patient's length of stay and for improving capacity and workflow management, [4Stone-Griffith S, Englebright JD, Cheung D, Korwek KM, Perlin JB. Data-driven process and operational improvement in the emergency department: the ED Dashboard and Reporting Application. J Healthc Manag 2012; 57(3): 167-80.
[PMID: 22724375] , 5McLeod B, Zaver F, Avery C, et al. Matching capacity to demand: A regional dashboard reduces ambulance avoidance and improves accessibility of receiving hospitals. Acad Emerg Med 2010; 17(12): 1383-9.
[http://dx.doi.org/10.1111/j.1553-2712.2010.00928.x] [PMID: 21122023] ] and in SICU to increase compatibility with ventilator bundle measures, and for decreasing rates of VPA [6Zaydfudim V, Dossett LA, Starmer JM, et al. Implementation of a real-time compliance dashboard to help reduce SICU ventilator-associated pneumonia with the ventilator bundle. Arch Surg 2009; 144(7): 656-62.
[http://dx.doi.org/10.1001/archsurg.2009.117] [PMID: 19620546] ].

The dashboard should have goals set by users and consistently meet their expectations. End user experience is one of the major features of dashboard software [7Malik S. Dashboard Evaluation: What are the Characteristics of a 'Best In Class' Enterprise Dashboard? 2007 2007. Available at: http://www.dashboardinsight.com/articles/digital-dashboards/fundamentals/dashboard-evaluation.aspx]. End users are information consumers who make decisions and drive change management strategies based on presented information [8Benny A. Checklist for Evaluating Dashboarding Tools 2010. Available at: http://bennyaustin.wordpress.com/2010/03/20/ checklist-for-evaluating-dashboarding-tools/].

In this regard, it is necessary for the dashboard to communicate information about the pros and cons of decision alternatives quickly, highlight factors that merit consideration, and provide information in a non-linear format to facilitate its incorporation in decision making deliberations [9Dolan JG, Veazie PJ, Russ AJ. Development and initial evaluation of a treatment decision dashboard. BMC Med Inform Decis Mak 2013; 13: 51.
[http://dx.doi.org/10.1186/1472-6947-13-51] [PMID: 23601912] ].

While dashboards constantly evolve, developing evaluation criteria based on clinical relevance, efficiency, and usability is important [1Dashboard OR, Ed. Ed. karami M Dashboard Operational Radiology, Ed. 28^th Iranian congress of Radiology. Iran. 2012]. The main aim of this study is to determine evaluation criteria of dashboards in the healthcare organizations. The results obtained from this study can help users to determine a set of user interface requirements to inform the building or selecting an appropriate dashboard for operational use.

In this Delphi study, a review of literature was firstly performed to gain a good understanding of the criteria that contributed to evaluation of the dashboard tools. Based on the literature review, a number of features and their relevant elements were extracted to assist and inform user interface requirements. These features were known as category and the elements were known as criteria. It was then set which criterion was relevant for each category.

Next, a panel of information technology consultants composed of experts in health information management, medical informatics, and software engineering along with radiologists as end users (because a prototype of the dashboard was used in the radiology department.) were asked to confirm and rank these criteria with an open ended question included to seek further potential criteria. In this process, the panel was asked to declare their opinions about whether a criterion is necessary or not. They also were asked to rank every criterion based on its priority and the degree of importance in terms of category. In this regard, an electronic checklist including items such as criteria, yes, no, rank, and your comments were designed. In the first round, a list of the criteria to be used was determined. SPSS (ver. 16) was then employed for descriptive statistical analyses.

Out of the 46 experts, 42 completed their task. A total of 56 criteria statements were confirmed in the first round. Since in the first round no offers were gained, the Delphi study was completed with only one round. 56 important criteria for evaluating or building or selecting an effective dashboard were categorized into 7 groups. These are presented in Tables (1-7).

Table 1
Feature of user customization and related criteria.

Table 2
Feature of knowledge discovery and related criteria.

Table 3
Feature of security and related criteria.

Table 4
Feature of information delivery and related criteria.

Table 5
Feature of visual design and related criteria.

Table 6
Feature of alerting and related criteria.

Table 7
Feature of system connectivity & integration and related criteria.

The results showed that the first category “user customization” and its four main subgroups were highly rated, except for “discussion forum” which was ranked at the level of 5. The “knowledge discovery” category covered 5 criteria with the same degree of importance. All the criteria ranked as 1. In the category of “security”, 7 criteria were of which only the “version control” criterion gained the lowest rank as 5.

There were 8 criteria included in the “information delivery” of which the “Sorting the report” was found at the lowest rank of 6. The “visual design” entailed 10 criteria of which the “single screen with no scrolling” gained the lowest ranking of 6. The “alert” category contained 10 criteria in which the “delivering alert through pager” obtained the lowest priority of 3. The “system connectivity and integration” category consisted 6 criteria of which the “integrating with portal” was considered as the lowest rank of 3.

Based on the results, the dashboard evaluation criteria were grouped into 7 major categories. The first category included “user customization”. This criterion can empower the users to response rapid changes in environment and turn their departments to agile department [10karami M. Application of Data Mining and Text Mining in agility of healthcare organization. Health Manage 2007; 10(30): 15-20.]. In this category, “discussion forum” gained the lowest priority. The discussion forum is “an application that allows a thread of communication among several users”. Low priority does not meant that this criterion is not important because this feature can enhance the collaboration between users and ultimately make better decisions, especially in times of crisis. In fact, collaboration converts the role of the dashboard from passive information interface to an active management console [7Malik S. Dashboard Evaluation: What are the Characteristics of a 'Best In Class' Enterprise Dashboard? 2007 2007. Available at: http://www.dashboardinsight.com/articles/digital-dashboards/fundamentals/dashboard-evaluation.aspx, 11Eckerson W. Evaluation criteria for dashboards and scorecard 2005. Available at: http://adtmag.com/articles/2005/01/01/ evaluation-criteria-for-dashboards-and-scorecards.aspx].

The second category was knowledge discoveryin which all the criteria obtained high priority, because they allow users to conduct root analysis, especially when diving to discover cause of incidents or problems [11Eckerson W. Evaluation criteria for dashboards and scorecard 2005. Available at: http://adtmag.com/articles/2005/01/01/ evaluation-criteria-for-dashboards-and-scorecards.aspx, 12Karami M, Safdari R, Rahimi A. Effective radiology dashboards: key research findings. Radiol Manage 2013; 35(2): 42-5.
[PMID: 23638580] ].

Among the security criteria, “version control” found the lowest priority. The definition of version control is a “repository of files, often source code files of computer programs, with monitored access. All changes applied on source files are tracked along with who made such changes, why they did it, and references to problems fixed, or enhancements introduced” [13Yeates S. What is version control? 2013. Available at: http://oss-watch.ac.uk/resources/versioncontrol]. Therefore, it is an important feature.

Although, dashboard is not a reporting tool, however, its main use is reporting. Therefore, the way of “information delivery” is very important for dashboard design. The dashboard should meet the objectives that are defined and understood by the users on an ongoing basis. And also, the context of the contents being displayed in the dashboard should be in clarity [12Karami M, Safdari R, Rahimi A. Effective radiology dashboards: key research findings. Radiol Manage 2013; 35(2): 42-5.
[PMID: 23638580] , 14Fox D. Applying Heuristic Principles to Dashboard Design & Evaluation 2011. Available at: http://www.claricent.com/2011/06/ applying-heuristic-principles-to-dashboard-design-evaluation/].

The options such as sorting, filtering, exporting, inserting, deleting, scheduling, and updating are the user’s interactions with the dashboard. Pleasurable and respectful user interaction can enhance user’s quality of work-life because it enables the users to access to information at the right time with the least amount of effort and also keeps the dashboard stable dynamically [14Fox D. Applying Heuristic Principles to Dashboard Design & Evaluation 2011. Available at: http://www.claricent.com/2011/06/ applying-heuristic-principles-to-dashboard-design-evaluation/-16Few S. Common Pitfalls in Dashboard Design - Perceptual Edge 2006 2006. Available at: www.perceptualedge.com/articles/ Whitepapers/Common_Pitfalls.pdf].

The sixth group was “visual design”. The dashboard should be visually appealing and engaging without overwhelming the users but make them feel comfortable and enable them to change the information layout [14Fox D. Applying Heuristic Principles to Dashboard Design & Evaluation 2011. Available at: http://www.claricent.com/2011/06/ applying-heuristic-principles-to-dashboard-design-evaluation/]. It is necessary to adopt a concise and minimalist design in order to avoid overloading the user with information, components, contents, and navigation steps that are unnecessary. This is lead to avoid from fragmenting the information by having to scroll. The capability of “Single screen with no scrolling” caused to display useful information without screen scrolling and users can overview them quickly [14Fox D. Applying Heuristic Principles to Dashboard Design & Evaluation 2011. Available at: http://www.claricent.com/2011/06/ applying-heuristic-principles-to-dashboard-design-evaluation/-19Few S. Information Dashboard Design: The Effective Visual Communication of Data 1^st ed. 2006.].

The criterion “metadata and help” informs the user about the dashboard and instructions for its use. This information should be visible or easily retrievable whenever appropriate. In this category, visual intelligence means the capability of software to provide better insight through intelligently highlighting relevant areas and values on the dashboard in response to a user's cursor movement. Intelligent presentation improves the user's ability to extract information from data” [14Fox D. Applying Heuristic Principles to Dashboard Design & Evaluation 2011. Available at: http://www.claricent.com/2011/06/ applying-heuristic-principles-to-dashboard-design-evaluation/].

“Alerting” was the seventh category. The alerts are a mechanism to turn the focus to the exceptions, outliers and data highlights. Whether embedded in the dashboard or presented separately, alerts can be used as extra layers of abstraction to make a dashboard more useful [20Gemignani Z. A dashboard alerts checklist 2008. Available at: http://www.juiceanalytics.com/writing/dashboard-alerts-checklist]. The dashboard should be able to manage events to evaluate the performance of department against predefined goals using indicators [11Eckerson W. Evaluation criteria for dashboards and scorecard 2005. Available at: http://adtmag.com/articles/2005/01/01/ evaluation-criteria-for-dashboards-and-scorecards.aspx, 12Karami M, Safdari R, Rahimi A. Effective radiology dashboards: key research findings. Radiol Manage 2013; 35(2): 42-5.
[PMID: 23638580] ]. It is better to present the alerts by color coding to show levels of threats. These alerts are defined based on performance target thresholds which are derived from the yearly goals and objectives. Thresholds are defined as a target zone in green, a warning zone in yellow, and a trouble zone in red [21Karami M, Safdari R. From information management to information visualization: Development of radiology dashboards. Appl Clin Inform 2016; 7(2): 308-29.
[http://dx.doi.org/10.4338/ACI-2015-08-RA-0104] [PMID: 27437043] ].

Also, agents can be used in data models for registering and managing alerts if alerts are to be shown in groups or individual [21Karami M, Safdari R. From information management to information visualization: Development of radiology dashboards. Appl Clin Inform 2016; 7(2): 308-29.
[http://dx.doi.org/10.4338/ACI-2015-08-RA-0104] [PMID: 27437043] ]. Agent is a computing entity which is located in dynamic and complex environment and can autonomically sense the environment and act accordingly to complete its tasks or goals [22karami M, Hosseini S.A. Applying agent-based technologies in complex healthcare environment. Iran J Public Health 2017; 46. [in-press].]. In ranking, the criterion “deliver alert through pager” found the lowest rank. It should be noted that in the design of alerts, unnecessary alarm must be avoided [3Morgan MB, Branstetter BF, Lionetti DM, Richardson JS, Chang PJ. The radiology digital dashboard: effects on report turnaround time. J Digit Imaging 2008; 21(1): 50-8.
[http://dx.doi.org/10.1007/s10278-007-9008-9] [PMID: 17334871] ].

Finally, the seventh category was “system connectivity and integration”. The dashboard software must be coordinated with the system infrastructure in organization. It must support different versions of all operating systems used within the organization and also must fall into the domain of organization’s application servers. On the other side, the dashboard should be able to capture live data from various data sources, and if data values in the specific data source have changed, those changes should be reflected in the dashboard. It should be able to interact with standard databases such as relational or multidimensional data bases [7Malik S. Dashboard Evaluation: What are the Characteristics of a 'Best In Class' Enterprise Dashboard? 2007 2007. Available at: http://www.dashboardinsight.com/articles/digital-dashboards/fundamentals/dashboard-evaluation.aspx]. Integrating with portal obtained the lowest rank while this is a main feature to achieve the virtual dashboard [23Aydin CE, Bolton LB, Donaldson N, Brown DS, Mukerji A. Beyond Nursing Quality Measurement: The Nation's First Regional Nursing Virtual Dashboard Assessment) Henriksen K, Battles J, Keyes M, editors Advances in Patient Safety: New Directions and Alternative Approaches 1 2011/01/21 ed Rockville Agency for Healthcare Research and Quality 2011.].

Since healthcare is a data-intensive and technology-driven environment, choosing good criteria to select effective dashboards for using in such environment is critical. This study tried to reveale the importance of key criteria which could inform the development of a checklist for evaluating the dashboards. Align with progression in dashboard technology, comparative study and user feedback will lead to further improvement in this field.