Welphi as tool to inform the shape of value functions– 2nd step in the process of an index creation; Combining the multi-criteria method MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique) with Web-Delphi social processes.

Nowadays the process of creating an index, or the process of decision making, has drawn-out of its classical approach, having to consider aspects such as multiple criteria and stakeholders’ opinions in the same scope, which usually leads to a satisfactory outcome rather than an optimal one. 

Such setups have been recognized as requiring innovative ways of combining evidence with the views and values of multiple stakeholders like decision makers, experts and ultimately the population affected by future decisions. This evolution and consciousness regarding index creation, and decision making, features has led to the Multi-Criteria Decision Analysis (MCDA). 

MCDA has been defined as “an umbrella term, which describes a collection of formal approaches that seek to take explicit account of multiple criteria in helping individuals or groups explore decisions that matter”. MCDA is a structured framework with two components: a technical and a social component. The technical component entails the employment of a set of technics to support the different steps of the development of an index creation, or multi-criteria evaluation model, whereas the social component is meant to capture the points of view of the participants involved, in order to create a “shared understanding of the issue”. 


Welphi has proven to be a great game changer ensuring the social strand of these work frames, beening a rich and effective way to collect information from an enlarged and geographically dispersed number of participants. Having set the necessary structuring work leading to the structuring components, i.e indicators or criterion, to be considered in an index creation, or decision making process, follows the task of articulating and modeling preferences.  

Let us resort to our latest introduced Welphi case study – The Euro-Healthy project – to broadly explain how Welphi was used to articulate and model preferences, in a 1^st instance accessing panelists’ views about the added value of improvements in different levels of indicator, informing the shape of the value functions resorting to the MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique) socio-technical methodological approach, a novel methodology specially developed for the Euro-Healthy project’s needs.

In the scope of the Euro-Healthy project, and at this stage, Welphi processes, web-based Delphi processes, technically sound in MACBETH were conducted to inform the shape of value functions. The MACBETH based intuitive protocols of questioning were key to promote transparency along the processes, avoiding the eventual difficulty and cognitive uneasiness experienced by evaluators when trying to express their preference judgments numerically. These processes were developed in three sequential rounds. Along the three rounds, an “identity card” for each indicator was always available for on-line search of information and scientific evidence, as was also always available, for each indicator, the range of performance across the European regions. This was done resorting to Welphi’s features which you can find more about at http://support.welphi.com/article/add-indicators/. 

|Aim: To determine the shape of the value function on each indicator.

|Objects of study: A set of previously selected indicators, through Welphi processes. (https://welphi.blogspot.com/2019/01/welphi-for-selection-or-identification.html).

|Panel: 58 experts and stakeholders from different countries, distributed by four panels as follows Socio-economic, Demographic change and Health behaviors, Physical and Built environment and Healthcare and Mortality, according to their area of knowledge and expertise. A common design for the Welphi's value function processes was implemented simultaneously with the four panels.

|1^st round: The range of performance on each indicator was divided in three equal pieces representing three changes in performance. For each indicator, panelists judged the contribution to population health of each one of those three changes in performance, answering the question:

“To improve population health in Europe, what is the contribution of this change in performance on the indicator?”

Panelists’ answers were provided with the MACBETH qualitative judgement scale (from very weak contribution, to extreme contribution). A “Don’t know/Don’t want to answer” option was available and comments could be given on each indicator. To the sequence of three judgements given by each panelist as attributed an implicit a value function shape on each indicator.

|2^nd and 3^rd rounds: In the 2^nd round panelists were presented with the implicit value function of their first round judgments for each indicator. Moreover, feedback with the results of the 1^st round was provided on each indicator: the number and percentage of respondents for each type of implicit value function shape and the comments made. In this 2^nd round, panelists were given the chance to either confirm or change their implicit value functions shapes, at the light of the group information provided. At the beginning of the 3^rd round, panelists received again information synthesizing the answers of the panel after the 2^nd round. Accordingly, these had the opportunity to maintain or revise their answers.

|Results: The results of the process, along with other project’s outcomes guided a strategic group to set the value functions for the indicators in a decision conference.

Check out our support page (http://support.welphi.com/video-tutorials/) and watch our video tutorials to help setup your Welphi process today for articulating and modeling preferences towards your project’s goals, whichever your framework might be!

Welphi for the selection, or identification, of criteria and indicators – 1st step in the process of an index creation

The selection, or identification, of criteria and indicators is the first step in any process of creation of an index or any evaluation of options. The Delphi method, as a participatory process to accomplish the task of selecting, or identifying, criteria and indicators has proven to increase the validity and acceptance of the project’s results independently of its area of application, as both criteria and indicators are key structuring components for projects. Selecting, or identifying, criteria and indicators benefits form a multidisciplinary sphere often including the involvement of multiple stakeholders like decision makers, experts and ultimately the population affected by future decisions. The engagement of all these actors in the process increases outcomes’ chances of being deemed more credible, scientific, commonly understood and technically useful. Is has been established that Delphi enables the involvement of all the actors in an iterative and anonymous way, towards a shared understanding undoing all social pressures to conform. However, its characteristics of successive interrogation and questioning format make it a highly resource and time consuming technique to employ. Not anymore with Welphi’s web-based environment and automation features enabling for the development of Delphi processes in an easy way.

Let us resort to our latest introduced Welphi case study – The Euro-Healthy project – to broadly explain how Welphi was used for the selection of indicators to build a population health index.

In the scope of the Euro-Healthy project, Welphi processes were developed, comprising a total of three rounds, to involve a multidisciplinary panel and to establish their views on the relevance of the identified indicators, with specific rules, resulting to Welphi’s rule building features, in place for dealing with differences in opinion and for measuring the level of agreement¹.

|Aim: to apply a participatory process to inform the selection of a set of indicators considered relevant combining scientific evidence and the points of view of experts and stakeholders.

|Objects of study: 130 indicators, previously identified through literature review.

|Panel: 51 experts and 30 stakeholders from different countries, with applicable knowledge in a variety of domains and a keen level of interest in the field of European population health.

|1^st round: Panelists were required to indicate their level of agreement or disagreement with the following statement

“This indicator is relevant to the evaluation of Europe’s population health”

Panelists must choose one of the 5 following options, belonging to a 5-level Likert scale: “Strongly disagree”, “Disagree”, “Neither agree nor disagree”, “Agree” and “Strongly Agree”.

|2^nd and 3^rd rounds: The extracted results of the 1^st round were presented at the beginning of the 2^nd round, allowing panelists to maintain or revise their answers.  At the beginning of the 3^rd round, panelists received again information synthesizing the answers of the panel after the 2^nd round. Accordingly, these had the opportunity to maintain or revise their answers.

|Results: The group opinion (aggregate of individual opinions) was defined by calculating the percentage of responses given in each Likert item, for each indicator. The group agreement for indicator approval was determined by absolute majority (agreement above 50% and disagreement below 33.3%) and qualified majority (agreement above 75%)¹.

Check out our support page (http://support.welphi.com/video-tutorials/) and watch our video tutorials to help setup your Welphi process today for selecting, or identifying, your project’s structuring components, whichever your framework might be! Start your free trial today to find our Welphi process “Demo – Selection of indicators for health index”, based in the Welphi case-study – Euro-Healthy.

¹ A. Freitas and P. Santana “The selection of indicators to evaluate European population health”, http://www.euro-healthy.eu/documents/indicator-selection.

Using Welphi to build a Population Health Index – The Euro-Healthy case

Welphi was used to integrate the Euro-Healthy (Shaping EUROpean policies to promote HEALTH equitY) project, a three-year Horizon 2020 research project launched in January 2015 which aimed to advance knowledge of policies that have the highest potential to enhance health and health equity across European regions. 

The project envisioned the development of tools – based on a Population Health Index (PHI) – to evaluate and monitor overall health as well as interactions between health and multiple dimensions at different geographical levels. The PHI was then used to foresee and discuss the impact of multi-level policies and combinations of policies in population health (PH) and health equity across European regions, thus providing a basis for policy dialogue.

Euro-Healthy brought together 15 multidisciplinary institutions from 12 European countries assuring a multi-sectoral approach required to employ cross-cutting determinants of population health and it was developed through eight research work packages (WP).

Welphi integrated the work developed in WP6 – Decision support for multicriteria modeling of the population health index and evaluation, foresight and selection of policies – proving to be a major asset in the approach taken towards the research work, answering to the project’s specific needs of developing inclusive non-face-to-face participatory processes, to be carried out with both health experts and stakeholders. Welphi was mainly used for the activities of:

1| Indentification of indicators;

2| Construction of the Indicators’ Value Functions;

3| Weighting the Indicators.

To ensure that the Euro-Healthy PHI considered the multiple dimensions that PH entails and was informed by evidence, a specific socio-technical design was built combining concepts of multicriteria value measurement with participatory processes through the development of several modified Web-Delphi processes technically sound in MACBETH (MACBETH - Bana e Costa, C.A., De Corte, J.M., Vansnick, J.C. (2012), “MACBETH”, International Journal of Information Technology and Decision Making, 11, 2, 359-387), resorting to Welphi.

The Web-Delphi processes conducted through Welphi, making use of its friendly and attractive interface, proved to be in fact an inclusive and effective way of collecting information from a high number of geographically dispersed experts and stakeholders aiding the building of the PHI with their insights thus allowing for the collection of first-hand knowledge and experience regarding the development and application of methods and tools for the health evaluation context.

Welphi’s processes results were then used to inform and assist a smaller and strategic group of participants to solve the questions in hand, as an effective way of including the views of an enlarged number of participants in the decision making process.

As a result of all the work developed in the Euro-Healthy’s scope the Euro-healthy Web-GIS (https://healthyregionseurope.uc.pt) now provides a wide range of robust information to analyse PH and health inequities across European regions.

Adapted from http://www.euro-healthy.eu/, where you can find more about the Euro-Healthy project. Test Welphi for yourself at http://www.welphi.com/.

Welphi benefits

While exploring Welphi one can pinpoint its most beneficial features for both the groups of actors in a Welphi process, the participants involved in the process and the team conducting the study. These are the benefits of Welphi, amongst others, that best meet the needs of both parties and are listed next!

For the participants engaging in the process Welphi:

| Ensures anonymity, avoiding social pressure;

| Provides controlled feedback and statistical summary of answers, engaging participants in a non-face-to-face format;

| Enables the response of each participant at its own pace;

| Presents a friendly and attractive interface.

For the team conducting the study Welphi:

| Is suitable for large evaluation groups, geographical dispersed and/or without time to meet participants;

| Allows a rapid execution process;

| Monitors in real time the response rate;

| Automatically sends reminder e-mails for the participants that have not participated in the round to the questionnaire;

| Automatically produces summary statistics of each round;

Don’t let time pass you by, see for yourself these and other benefits in the vast Welphi horizon! Visit us at http://www.welphi.com and start your trial today at https://app.welphi.com