Welphi as tool to inform the shape of value functions– 2nd step in the process of an index creation; Combining the multi-criteria method MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique) with Web-Delphi social processes.

Nowadays the process of creating an index, or the process of decision making, has drawn-out of its classical approach, having to consider aspects such as multiple criteria and stakeholders’ opinions in the same scope, which usually leads to a satisfactory outcome rather than an optimal one. 

Such setups have been recognized as requiring innovative ways of combining evidence with the views and values of multiple stakeholders like decision makers, experts and ultimately the population affected by future decisions. This evolution and consciousness regarding index creation, and decision making, features has led to the Multi-Criteria Decision Analysis (MCDA). 

MCDA has been defined as “an umbrella term, which describes a collection of formal approaches that seek to take explicit account of multiple criteria in helping individuals or groups explore decisions that matter”. MCDA is a structured framework with two components: a technical and a social component. The technical component entails the employment of a set of technics to support the different steps of the development of an index creation, or multi-criteria evaluation model, whereas the social component is meant to capture the points of view of the participants involved, in order to create a “shared understanding of the issue”. 


Welphi has proven to be a great game changer ensuring the social strand of these work frames, beening a rich and effective way to collect information from an enlarged and geographically dispersed number of participants. Having set the necessary structuring work leading to the structuring components, i.e indicators or criterion, to be considered in an index creation, or decision making process, follows the task of articulating and modeling preferences.  

Let us resort to our latest introduced Welphi case study – The Euro-Healthy project – to broadly explain how Welphi was used to articulate and model preferences, in a 1^st instance accessing panelists’ views about the added value of improvements in different levels of indicator, informing the shape of the value functions resorting to the MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique) socio-technical methodological approach, a novel methodology specially developed for the Euro-Healthy project’s needs.

In the scope of the Euro-Healthy project, and at this stage, Welphi processes, web-based Delphi processes, technically sound in MACBETH were conducted to inform the shape of value functions. The MACBETH based intuitive protocols of questioning were key to promote transparency along the processes, avoiding the eventual difficulty and cognitive uneasiness experienced by evaluators when trying to express their preference judgments numerically. These processes were developed in three sequential rounds. Along the three rounds, an “identity card” for each indicator was always available for on-line search of information and scientific evidence, as was also always available, for each indicator, the range of performance across the European regions. This was done resorting to Welphi’s features which you can find more about at http://support.welphi.com/article/add-indicators/. 

|Aim: To determine the shape of the value function on each indicator.

|Objects of study: A set of previously selected indicators, through Welphi processes. (https://welphi.blogspot.com/2019/01/welphi-for-selection-or-identification.html).

|Panel: 58 experts and stakeholders from different countries, distributed by four panels as follows Socio-economic, Demographic change and Health behaviors, Physical and Built environment and Healthcare and Mortality, according to their area of knowledge and expertise. A common design for the Welphi's value function processes was implemented simultaneously with the four panels.

|1^st round: The range of performance on each indicator was divided in three equal pieces representing three changes in performance. For each indicator, panelists judged the contribution to population health of each one of those three changes in performance, answering the question:

“To improve population health in Europe, what is the contribution of this change in performance on the indicator?”

Panelists’ answers were provided with the MACBETH qualitative judgement scale (from very weak contribution, to extreme contribution). A “Don’t know/Don’t want to answer” option was available and comments could be given on each indicator. To the sequence of three judgements given by each panelist as attributed an implicit a value function shape on each indicator.

|2^nd and 3^rd rounds: In the 2^nd round panelists were presented with the implicit value function of their first round judgments for each indicator. Moreover, feedback with the results of the 1^st round was provided on each indicator: the number and percentage of respondents for each type of implicit value function shape and the comments made. In this 2^nd round, panelists were given the chance to either confirm or change their implicit value functions shapes, at the light of the group information provided. At the beginning of the 3^rd round, panelists received again information synthesizing the answers of the panel after the 2^nd round. Accordingly, these had the opportunity to maintain or revise their answers.

|Results: The results of the process, along with other project’s outcomes guided a strategic group to set the value functions for the indicators in a decision conference.

Check out our support page (http://support.welphi.com/video-tutorials/) and watch our video tutorials to help setup your Welphi process today for articulating and modeling preferences towards your project’s goals, whichever your framework might be!

Welphi for the selection, or identification, of criteria and indicators – 1st step in the process of an index creation

The selection, or identification, of criteria and indicators is the first step in any process of creation of an index or any evaluation of options. The Delphi method, as a participatory process to accomplish the task of selecting, or identifying, criteria and indicators has proven to increase the validity and acceptance of the project’s results independently of its area of application, as both criteria and indicators are key structuring components for projects. Selecting, or identifying, criteria and indicators benefits form a multidisciplinary sphere often including the involvement of multiple stakeholders like decision makers, experts and ultimately the population affected by future decisions. The engagement of all these actors in the process increases outcomes’ chances of being deemed more credible, scientific, commonly understood and technically useful. Is has been established that Delphi enables the involvement of all the actors in an iterative and anonymous way, towards a shared understanding undoing all social pressures to conform. However, its characteristics of successive interrogation and questioning format make it a highly resource and time consuming technique to employ. Not anymore with Welphi’s web-based environment and automation features enabling for the development of Delphi processes in an easy way.

Let us resort to our latest introduced Welphi case study – The Euro-Healthy project – to broadly explain how Welphi was used for the selection of indicators to build a population health index.

In the scope of the Euro-Healthy project, Welphi processes were developed, comprising a total of three rounds, to involve a multidisciplinary panel and to establish their views on the relevance of the identified indicators, with specific rules, resulting to Welphi’s rule building features, in place for dealing with differences in opinion and for measuring the level of agreement¹.

|Aim: to apply a participatory process to inform the selection of a set of indicators considered relevant combining scientific evidence and the points of view of experts and stakeholders.

|Objects of study: 130 indicators, previously identified through literature review.

|Panel: 51 experts and 30 stakeholders from different countries, with applicable knowledge in a variety of domains and a keen level of interest in the field of European population health.

|1^st round: Panelists were required to indicate their level of agreement or disagreement with the following statement

“This indicator is relevant to the evaluation of Europe’s population health”

Panelists must choose one of the 5 following options, belonging to a 5-level Likert scale: “Strongly disagree”, “Disagree”, “Neither agree nor disagree”, “Agree” and “Strongly Agree”.

|2^nd and 3^rd rounds: The extracted results of the 1^st round were presented at the beginning of the 2^nd round, allowing panelists to maintain or revise their answers.  At the beginning of the 3^rd round, panelists received again information synthesizing the answers of the panel after the 2^nd round. Accordingly, these had the opportunity to maintain or revise their answers.

|Results: The group opinion (aggregate of individual opinions) was defined by calculating the percentage of responses given in each Likert item, for each indicator. The group agreement for indicator approval was determined by absolute majority (agreement above 50% and disagreement below 33.3%) and qualified majority (agreement above 75%)¹.

Check out our support page (http://support.welphi.com/video-tutorials/) and watch our video tutorials to help setup your Welphi process today for selecting, or identifying, your project’s structuring components, whichever your framework might be! Start your free trial today to find our Welphi process “Demo – Selection of indicators for health index”, based in the Welphi case-study – Euro-Healthy.

¹ A. Freitas and P. Santana “The selection of indicators to evaluate European population health”, http://www.euro-healthy.eu/documents/indicator-selection.

Using Welphi to build a Population Health Index – The Euro-Healthy case

Welphi was used to integrate the Euro-Healthy (Shaping EUROpean policies to promote HEALTH equitY) project, a three-year Horizon 2020 research project launched in January 2015 which aimed to advance knowledge of policies that have the highest potential to enhance health and health equity across European regions. 

The project envisioned the development of tools – based on a Population Health Index (PHI) – to evaluate and monitor overall health as well as interactions between health and multiple dimensions at different geographical levels. The PHI was then used to foresee and discuss the impact of multi-level policies and combinations of policies in population health (PH) and health equity across European regions, thus providing a basis for policy dialogue.

Euro-Healthy brought together 15 multidisciplinary institutions from 12 European countries assuring a multi-sectoral approach required to employ cross-cutting determinants of population health and it was developed through eight research work packages (WP).

Welphi integrated the work developed in WP6 – Decision support for multicriteria modeling of the population health index and evaluation, foresight and selection of policies – proving to be a major asset in the approach taken towards the research work, answering to the project’s specific needs of developing inclusive non-face-to-face participatory processes, to be carried out with both health experts and stakeholders. Welphi was mainly used for the activities of:

1| Indentification of indicators;

2| Construction of the Indicators’ Value Functions;

3| Weighting the Indicators.

To ensure that the Euro-Healthy PHI considered the multiple dimensions that PH entails and was informed by evidence, a specific socio-technical design was built combining concepts of multicriteria value measurement with participatory processes through the development of several modified Web-Delphi processes technically sound in MACBETH (MACBETH - Bana e Costa, C.A., De Corte, J.M., Vansnick, J.C. (2012), “MACBETH”, International Journal of Information Technology and Decision Making, 11, 2, 359-387), resorting to Welphi.

The Web-Delphi processes conducted through Welphi, making use of its friendly and attractive interface, proved to be in fact an inclusive and effective way of collecting information from a high number of geographically dispersed experts and stakeholders aiding the building of the PHI with their insights thus allowing for the collection of first-hand knowledge and experience regarding the development and application of methods and tools for the health evaluation context.

Welphi’s processes results were then used to inform and assist a smaller and strategic group of participants to solve the questions in hand, as an effective way of including the views of an enlarged number of participants in the decision making process.

As a result of all the work developed in the Euro-Healthy’s scope the Euro-healthy Web-GIS (https://healthyregionseurope.uc.pt) now provides a wide range of robust information to analyse PH and health inequities across European regions.

Adapted from http://www.euro-healthy.eu/, where you can find more about the Euro-Healthy project. Test Welphi for yourself at http://www.welphi.com/.

Welphi benefits

While exploring Welphi one can pinpoint its most beneficial features for both the groups of actors in a Welphi process, the participants involved in the process and the team conducting the study. These are the benefits of Welphi, amongst others, that best meet the needs of both parties and are listed next!

For the participants engaging in the process Welphi:

| Ensures anonymity, avoiding social pressure;

| Provides controlled feedback and statistical summary of answers, engaging participants in a non-face-to-face format;

| Enables the response of each participant at its own pace;

| Presents a friendly and attractive interface.

For the team conducting the study Welphi:

| Is suitable for large evaluation groups, geographical dispersed and/or without time to meet participants;

| Allows a rapid execution process;

| Monitors in real time the response rate;

| Automatically sends reminder e-mails for the participants that have not participated in the round to the questionnaire;

| Automatically produces summary statistics of each round;

Don’t let time pass you by, see for yourself these and other benefits in the vast Welphi horizon! Visit us at http://www.welphi.com and start your trial today at https://app.welphi.com

Welphi - The survey app to build consensus implementing the Delphi method

How does Welphi work?

Here’s how to setup a Welphi process in 9 easy steps!

1| Create questionnaires

Creating questionnaires and setting up the entire process is quick and easy. Just let the app know who will participate, what will be analysed, which possible answers can the participants choose and that’s it.

2| Introducing and inviting participants

Leave all the heavy-lifting administrative tasks to Welphi. Just let it know who the participants are and, on the click of a button, they will receive an invitation e-mail. Further ahead in the process, reminder emails can also be sent to all participants or only to those who still haven’t completed the questionnaires.

3| Introduce study objects

Easily insert the indicators/goals/criteria/scenarios or whichever objects you will be asking participants to analyse.

4| Creating response scales

Welphi allows the definition of any kind of closed scales like for example Likert scales, or any other you feel better capture your participants’ opinion towards the study elements.

5| Create rounds

In order to start getting the participants answers and feedback, you will need to create a round. To move towards a consensus, you can set up additional rounds at will.

6| Producing questionnaires statistics

Throughout the entire process, Welphi provides detailed stats of each participant’s status as well as the current results for each study object on each round.

7| Creating rules for approval and rejection

At the end of each round, Welphi will enable you to define acceptance or rejection rules based upon which any study object can be either approved, rejected or be left unclassified for further rounds

8| Sharing round results with participants

In the beginning of each round, participants will have the opportunity to check the previous round results as well as participants’ anonymous comments. This information can help them revise their answers or keep the same answer as in the previous round

9| Producing statistical graphs for each round and for the whole process

Welphi will give you the tools to analyse your rounds, test different rules, and check which indicators/criteria/goals got the higher and lower approval rates

Get to know Welphi for yourself! Start a free trial at https://app.welphi.com

The Delphi method & The Welphi

The Delphi method is a tool for structuring an effective group communication process, allowing a group of individuals, as a whole, to deal with a complex problem and it is based on the assumption that group opinion is more valid and reliable that individual opinion. Delphi was developed at the RAND Corporation in the 1950s to forecast the impact of technology on warfare and it roughly consists in the employment of a series of intensive questionnaires interspersed with a controlled opinion feedback. There are four key features that characterize a Delphi process, as follows:

1 | Anonymity of the individual responses;

2 | Iteration that allows participants to revise their perspectives and values;

3 | Controlled feedback;

4 | Statistical aggregation.

By not making mandatory the physical presence of the members of the group, Delphi avoids social pressure (as it is an anonymous process) and it allows for a more efficient use of resources, such as time and money. This is valid for all group sizes, but is especially relevant considering large groups and/or geographically dispersed participants.


Since its development, and given its underlying flexibility, the Delphi method has undergone different modifications and today there are several types of Delphi methods. This ability to adapt to different real-world problems makes it possible to find examples of Delphi processes on: risk management, investment analysis, planning health services, policy and trend analysis and policy formulation. The Delphi method has been used for meeting several challenges, including: establishing objectives, planning for budget allocations, collecting current and historical data, exploring perceptions associated with potential policy options, exploring personal motivations and values and forecasting by exploring what is likely to happen, what may change, and what effects can be anticipated. Thus, Delphi has been applied in a large number of domains including business, economics, finance, health care, sales and strategic planning.


The Welphi is an online survey platform implementing the Delphi method. The platform allows for the creation of questionnaires aiming at the acquisition of the opinion of several participants engaging in the process, which may or may not be geographically disperse and/or with a busy agenda, in an easy and affordable way due to its web-based nature. Through a system of questionnaire rounds, it is possible, in an anonymous way, to confront each participant with the opinions of his peers, engaging participants in a non-face-to-face format and promoting consensus among the group on issues that lack the support of quantitative data. Thus, the Welphi is positioned between a face-to-face meeting – where the confrontation of ideas is possible but the physical presence of all participants is required– and a simple online questionnaire – where the opinion of participants is requested but not shared with others and no opportunity is given to each participant to revise his option in light of the opinion of his peers. 

Therefore, the Welphi permits to confront ideas in an asynchronous, online, participatory and iterative way. Moreover, by permitting to anonymously share the participants’ comments, each participant can change his initial position, thus promoting group consensus. The Welphi was developed to facilitate the process of:

 | Creating questionnaires;

 | Creating rounds;

 | Introducing the object of study (indicators, criteria, objectives, policies, etc.);

 | Creating response scales;

 | Introducing and inviting participants;

 | Producing questionnaire statistics;

 | Creating rules for approval and rejection;

 | Sharing rounds results with participants;

 | Producing statistical graphs for each round and for the whole process.

Stay tuned for further information on the features of Welphi.

In the meantime, test the welphi at https://app.welphi.com