Friday, March 8, 2019

2nd round – Providing Controlled Feedback to the active participants in a Welphi process

If you’ve been following our weekly Welphi updates, it is completely acknowledged by now that the Welphi app is a tool for implementing Delphi processes. With this new update we hope to get more specific about Welphi and Controlled Feedback.

Controlled Feedback is one of the four key-characteristics of the Delphi method as a forecasting technic, and it takes action on the 2nd round of a Delphi process and onwards, in all the subsequent rounds.

In a Delphi process context Controlled Feedback is associated with the provision of anonymous feedback to all the active participants in the process, as to the number or density of responses to the question(s) in the previous rounds plus any new information/suggestion, typically in the form of comments provided by the participants. This anonymous feedback is provided through rounds to remove differences and try to streamline the opinions towards a focus topic, reducing variance in responses to the question(s), in order to improve the process’ output efficiency.

Here's how Welphi presents Feedback throughout the rounds of a Welphi process!

Starting the 2nd round, participants are presented with a screen containing all the percentages of anonymous answers provided in the 1st round of the process in a table, and are reminded of their individual answers, given in the 1st round, which appear pre-selected in a dark-grey cell. Assess to the previous round’s comments is also made available, anonymously, to the participants. To view these, participants simply have to click the “VIEW PREVIOUS ROUND COMMENTS” button, on the upper right corner of the questionnaire. By doing this, a window pops up showing participants’ comments relating to each criteria/indicator.  


In this 2nd round, participants are invited to either keep or change their answers, provided in the previous round, at the light of the group information provided. After revising its previously given answers, for a participant:

|To keep previous round’s answers: Press the “SAVE AND NEXT” button, throughout the questionnaire’s pages, to conclude the round. 

|To change the previous round’s answers: This can be done by clicking the “EDIT” button. By doing this all the answer’s options get automatically unlocked, enabling the participant to select new answers to the question(s). 

Comments can also always be provided at any point. If a participant chooses to do so, by clicking the “BALLOON” button, the balloon icon appears “painted” instead of blank, giving indication that a comment was placed for that criterion/indicator. It should be noted that a participant’s answers provided in the previous round stays highlighted in dark-grey during all the duration of the subsequent round, acting as a reminder. 

This example of providing Feedback to the participants in a Delphi process with Welphi was based on a 2nd round where aggregated data from the 1st round is presented to participants. However, Welphi always behaves in the same way for every subsequent round i.e, always showing the data from the precedent round.

Start your Welphi processes today and see for yourself these and other features that Welphi has to offer. Visit our continuously improving website at www.welphi.com and get registered today to start your Welphi experience with a free trial at www.app.welphi.com.  

Friday, February 22, 2019

Welphi - Enabling synchronicity in any asynchronous way


Time is a funny thing, isn't it? This video, by Nathan Yau, is but ONE exemple of how our day-to-day lives differ from one and other, in terms of time availability. Imagine just adding the geographical disperness to the picture as another time constrain... This and other set-ups make it hard, sometimes, to gather fundamental players, to help you cross your finish lines... Welphi is here to give you a helping hand ;) Welphi allows you to develop your Delphi processes, i.e. Welphi processes, synchronously in any asynchronous situation. Like we said, funny who time works! Check out our website for more on Welphi and it's features, including the ability to develop processes breaking time and geographical constrains, at www.welphi.com! Get in touch with us, we'd love to be a part in the process of achieving your goals! As always, stay tunned!

Friday, February 15, 2019

Designing a Welphi Questionnaire


An indicator/criteria can have several custom fields which you can create to add information about your indicators/criteria. These indicators/criteria custom fields can also be used to help you achieve your desired Welphi questionnaire design as tools to implement the overall follow and layout of your questionnaire, and to make information available to your Welphi panel of respondents.

Well, you may be asking: But how? Following is introduced a simple step-by-step guide on how you can use custom fields to help implement your desired Welphi questionnaire design!

1|Creat your custom fields: You should create your custom fields before you create any indicator/criteria. To add a field, go to the Indicators’ page, click the “ADD/EDIT COSTUM INDACATOR FIELD” button and then the “create indicators field” button. Each field must have a name and can have a description.


2|Select and customize your custom fields to get your desired Welphi questionnaire design. A custom field can be used to:

     2.1| Group indicators, making each different group to appear in a different page in your Welphi questionnaire. To do this choose the “Group all the indicators with the same value in this field. Each group will appear in a separate page.” when adding or editing your indicators field. The design achieved by this action is as shown in the screen featured in Figure 1, below. 

Figure 1 - Welphi screen, displaying groups of indicators appearing in different pages of a Welphi questionnaire. 

2.2|Subgroup indicators, creating different sections in each group page of your Welphi questionnaire. To do this choose the “Subgroup all the indicators with the same value in this field. Each subgroup will appear in a separate section.” when adding or editing your indicators field. The design achieved by this action is as shown in the screen featured in Figure 2, below.

Figure 2 - Welphi screen, displaying subgrouped indicators appearing in different sections on the same page of a Welphi questionnaire. 

2.3|Make information available on your indicator/criterion. If you don’t want to use the field values to group indicators, select the option “Don’t use this field to group indicators”. By doing this you will not be grouping or subgrouping your indicators/criterion as explained above in 2.1 and 2.2.. You will however be adding information to your Welphi questionnaire that you deem as valuable or even essential, to help your panel of respondents to provide their answers in a more informed way. You can input whichever information you see fit. For instance, you can make available tp the participants in the process descriptions of your indicators/criterion or scientific information on them. Information added to this custom field while be displayed on the Welphi questionnaire screen provided the respondents click the “EYE” button highlighted in the screen bellow (Figure 3).  

Figure 3 - Welphi screen, emphasizing the "EYE" button in a page of a Welphi questionnaire.

Note that in order to belong to the same group or subgroup, a set of indicators must have the same value in the selected field and that you can only select one field to group indicators and one field to or sub-group them in each process.

Every field can be edited later in the process by clicking the field’s name in the costum indicators field page. You can also add new fields if you want to. After creating the desired field, you can add the indicators adding values to its fields.

For more on this topic watch our video on how to add indicators on our support page at http://support.welphi.com/article/add-indicators/!

Friday, February 8, 2019

Using Welphi to build a multicriteria model for the selction of software in the oil&gas industry




Wondering if Welphi is the tool to help your decision problems while meeting corporate environments? We can help answer that question for you: definitely yes! Read below to find out how Welphi was successfully used to solve a real decision problem of the leading oil & gas company in Portugal regarding the selection and integration of a much needed software: a data integration platform.

Welphi was used in this project as a tool to collect information to latter inform the building of a multicriteria model meant to assess vendor’s proposals regarding the evaluation and selection of an Enterprise Management System, i.e. a data integration platform.  

Adopting a framework combining concepts of multicriteria value measurement with participatory processes, two Welphi processes where developed and implemented, which where both technically sound in MACBETH, in order to make clear the value system of the actors engaging in the processes these Welphi processes where used to either,
  • Determine the added value from a set of performance levels in their respective categories, i.e. within each criterion (criteria value functions).
  • Determine the partial value of each criterion considered in the model (weighting criteria).

 |Aim
  •  Delphi for Criteria value functions: Collect qualitative pairwise comparison judgments between performance scale levels on each one of multiple evaluation criteria.
  •  Delphi for weighting criteria: Collect qualitative judgments of importance of swinging between least and most preferred performance levels on the criteria.

|Objects of study: A previously selected set of evaluation criteria, determined by the company, considering both the functional and technical stands of software solutions applicable in the context.

|Panel: A total of 73 experts and future users of the platform belonging to district working areas within the company, distributed by two panels: Functional and Technical, according to their area of knowledge and expertise. A common design for the Welphi value function process was implemented simultaneously with the two panels.

|1st round:
  • Delphi for criteria value functions: For each of the considered criteria the respective performance intervals were displayed, each one representing an increase in performance. The participants were asked to answer the following question:

“With regard to this criterion, which do you consider to be the increase in preference between each two levels of performance?”

Answers were provided according to the MACBETH qualitative judgment scale (from “no increase” to “extreme increase”). The sequence of the participants’ answers enabled to extract their implicit main concerns for each criterion.
  • Delphi for weighting criteria: The participants were asked to answer the following question:

“Regarding the selection of the proposal for a data integration platform that best meets the company’s needs, suppose there is a proposal with neutral performances in all criteria. What would be the importance of improving it from neutral to good on each of the criteria?”
Answers were provided with the MACBETH qualitative judgment scale (from “no importance” to “extreme importance”).
In both Welphi processes a “don’t know/don’t want to answer” option was also available for selection and comments could be provided.
|2nd and 3rd rounds: The extracted results of the 1st round, participant’s percentage of answers and comments, were presented at the beginning of the 2nd round, allowing panelists to maintain or revise their answers. Specifically, for the Welphi for criteria value functions, participants were presented with information about the percentage of participants corresponding with each criterion and each one of the extracted implicit main concern. At the beginning of the 3rd round, panelists received again information synthesizing the answers of the panel after the 2nd round. Accordingly, these had the opportunity to maintain or revise their answers.

|Results: Resulting outcomes of this processes served as feed in information for the company to construct the aimed multicriteria evaluation model, having a strategic group make final decisions based in the provided compilation of judgments form the participants in the Delphi processes.

Friday, February 1, 2019

Welphi as tool to understand weighting coefficients – 3rd step in the process of an index creation; Combining the multi-criteria method MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique) with Web-Delphi social processes




Following up the introduction made in our last article (https://welphi.blogspot.com/2019/01/welphi-as-tool-to-inform-shape-of-value.html), this week we pick up where we left off, broadly explaining how Welphi was used in our latest introduced Welphi case study – The Euro-Healthy project – to articulate and model preferences. This time Welphi was used to explore panelists’ views about the importance of closing gaps in indicators to qualitatively understand weighting coefficients. 

In the scope of the Euro-Healthy project Welphi processes, web-based Delphi processes, technically sound in MACBETH were conducted to qualitatively understand weighting coefficients. Once more, the MACBETH based intuitive protocols of questioning were key to promote transparency along the processes, avoiding the eventual difficulty and cognitive uneasiness experienced by evaluators when trying to express their preference judgments numerically. These processes were developed in three sequential rounds. Along the three rounds, an “identity card” for each indicator was always available for on-line search of information and scientific evidence, as was also always available, for each indicator, the gap between the worst and best performance across the European regions. This was done resorting to Welphi’s features which you can find more about at http://support.welphi.com/article/add-indicators/.  

|Aim: To collect qualitative weighting judgments on each indicator.

|Objects of study: A set of previously selected indicators, through Welphi processes (https://welphi.blogspot.com/2019/01/welphi-for-selection-or-identification.html).

|Panel: 58 experts and stakeholders from different countries, distributed by four panels as follows Socio-economic, Demographic change and Health behaviors, Physical and Built environment and Healthcare and Mortality, according to their area of knowledge and expertise. A common design for the Welphi weighting process was implemented simultaneously with the four panels.

|1st round: Panelists were presented with a list of indicators and their gaps between the worst and best performance across the European regions. Each panelist individually was asked to answer a question concerning the gap on each indicator. The question was:

“To reduce inequalities in Europe, how important is to close this gap?”

While answering panelists considered “how big is the gap and how important it is to develop policies to close the gap in view of reducing health inequalities in Europe”. Panelists’ answers were provided with the MACBETH qualitative judgement scale (from not important, to extremely important). A “Don’t know/Don’t want to answer” option was available and comments could be given on each indicator.

|2nd and 3rd rounds: The extracted results of the 1st round were presented at the beginning of the 2nd round, allowing panelists to maintain or revise their answers.  At the beginning of the 3rd round, panelists received again information synthesizing the answers of the panel after the 2nd round. Accordingly, these had the opportunity to maintain or revise their answers.

|Results: The results of the process, along with other project’s outcomes guided a strategic group to assign quantitative weights for the indicators in a decision conference.

In the scope of the Euro-Healthy project, the outputs of both our introduced and distinct Welphi processes, to inform the shape of value functions and to understand weighting coefficients, together with scientific evidence collected, latter informed a decision conferencing face-to-face process in which a strategic group composed of 13 members, representing the diversity of viewpoints related to PH and covering the different areas of expertise and interest, successfully participated in the process of building the PHI model. The result was a set of multi-level PH indices based on the hierarchical multicriteria model. The PHI model underwent a set of testing, adjustment and validation procedures by the strategic group, until the PHI model has shown to be able to be a proper tool to assess PH across European regions.

Check out our support page (http://support.welphi.com/video-tutorials/) and watch our video tutorials to help setup your Welphi process today for articulating and modeling preferences towards your project’s goals, whichever your framework might be!

Friday, January 25, 2019

Welphi as tool to inform the shape of value functions– 2nd step in the process of an index creation; Combining the multi-criteria method MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique) with Web-Delphi social processes.




Nowadays the process of creating an index, or the process of decision making, has drawn-out of its classical approach, having to consider aspects such as multiple criteria and stakeholders’ opinions in the same scope, which usually leads to a satisfactory outcome rather than an optimal one. 

Such setups have been recognized as requiring innovative ways of combining evidence with the views and values of multiple stakeholders like decision makers, experts and ultimately the population affected by future decisions. This evolution and consciousness regarding index creation, and decision making, features has led to the Multi-Criteria Decision Analysis (MCDA). 

MCDA has been defined as “an umbrella term, which describes a collection of formal approaches that seek to take explicit account of multiple criteria in helping individuals or groups explore decisions that matter”. MCDA is a structured framework with two components: a technical and a social component. The technical component entails the employment of a set of technics to support the different steps of the development of an index creation, or multi-criteria evaluation model, whereas the social component is meant to capture the points of view of the participants involved, in order to create a “shared understanding of the issue”. 


Welphi has proven to be a great game changer ensuring the social strand of these work frames, beening a rich and effective way to collect information from an enlarged and geographically dispersed number of participants. Having set the necessary structuring work leading to the structuring components, i.e indicators or criterion, to be considered in an index creation, or decision making process, follows the task of articulating and modeling preferences.  

Let us resort to our latest introduced Welphi case study – The Euro-Healthy project – to broadly explain how Welphi was used to articulate and model preferences, in a 1st instance accessing panelists’ views about the added value of improvements in different levels of indicator, informing the shape of the value functions resorting to the MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique) socio-technical methodological approach, a novel methodology specially developed for the Euro-Healthy project’s needs.

In the scope of the Euro-Healthy project, and at this stage, Welphi processes, web-based Delphi processes, technically sound in MACBETH were conducted to inform the shape of value functions. The MACBETH based intuitive protocols of questioning were key to promote transparency along the processes, avoiding the eventual difficulty and cognitive uneasiness experienced by evaluators when trying to express their preference judgments numerically. These processes were developed in three sequential rounds. Along the three rounds, an “identity card” for each indicator was always available for on-line search of information and scientific evidence, as was also always available, for each indicator, the range of performance across the European regions. This was done resorting to Welphi’s features which you can find more about at http://support.welphi.com/article/add-indicators/. 

|Aim: To determine the shape of the value function on each indicator.

|Objects of study: A set of previously selected indicators, through Welphi processes. (https://welphi.blogspot.com/2019/01/welphi-for-selection-or-identification.html).

|Panel: 58 experts and stakeholders from different countries, distributed by four panels as follows Socio-economic, Demographic change and Health behaviors, Physical and Built environment and Healthcare and Mortality, according to their area of knowledge and expertise. A common design for the Welphi's value function processes was implemented simultaneously with the four panels.

|1st round: The range of performance on each indicator was divided in three equal pieces representing three changes in performance. For each indicator, panelists judged the contribution to population health of each one of those three changes in performance, answering the question:

“To improve population health in Europe, what is the contribution of this change in performance on the indicator?”

Panelists’ answers were provided with the MACBETH qualitative judgement scale (from very weak contribution, to extreme contribution). A “Don’t know/Don’t want to answer” option was available and comments could be given on each indicator. To the sequence of three judgements given by each panelist as attributed an implicit a value function shape on each indicator.

|2nd and 3rd rounds: In the 2nd round panelists were presented with the implicit value function of their first round judgments for each indicator. Moreover, feedback with the results of the 1st round was provided on each indicator: the number and percentage of respondents for each type of implicit value function shape and the comments made. In this 2nd round, panelists were given the chance to either confirm or change their implicit value functions shapes, at the light of the group information provided. At the beginning of the 3rd round, panelists received again information synthesizing the answers of the panel after the 2nd round. Accordingly, these had the opportunity to maintain or revise their answers.

|Results: The results of the process, along with other project’s outcomes guided a strategic group to set the value functions for the indicators in a decision conference.

Check out our support page (http://support.welphi.com/video-tutorials/) and watch our video tutorials to help setup your Welphi process today for articulating and modeling preferences towards your project’s goals, whichever your framework might be!

Friday, January 18, 2019

Welphi for the selection, or identification, of criteria and indicators – 1st step in the process of an index creation


The selection, or identification, of criteria and indicators is the first step in any process of creation of an index or any evaluation of options. The Delphi method, as a participatory process to accomplish the task of selecting, or identifying, criteria and indicators has proven to increase the validity and acceptance of the project’s results independently of its area of application, as both criteria and indicators are key structuring components for projects. Selecting, or identifying, criteria and indicators benefits form a multidisciplinary sphere often including the involvement of multiple stakeholders like decision makers, experts and ultimately the population affected by future decisions. The engagement of all these actors in the process increases outcomes’ chances of being deemed more credible, scientific, commonly understood and technically useful. Is has been established that Delphi enables the involvement of all the actors in an iterative and anonymous way, towards a shared understanding undoing all social pressures to conform. However, its characteristics of successive interrogation and questioning format make it a highly resource and time consuming technique to employ. Not anymore with Welphi’s web-based environment and automation features enabling for the development of Delphi processes in an easy way.

Let us resort to our latest introduced Welphi case study – The Euro-Healthy project – to broadly explain how Welphi was used for the selection of indicators to build a population health index.

In the scope of the Euro-Healthy project, Welphi processes were developed, comprising a total of three rounds, to involve a multidisciplinary panel and to establish their views on the relevance of the identified indicators, with specific rules, resulting to Welphi’s rule building features, in place for dealing with differences in opinion and for measuring the level of agreement1.

|Aim: to apply a participatory process to inform the selection of a set of indicators considered relevant combining scientific evidence and the points of view of experts and stakeholders.

|Objects of study: 130 indicators, previously identified through literature review.

|Panel: 51 experts and 30 stakeholders from different countries, with applicable knowledge in a variety of domains and a keen level of interest in the field of European population health.

|1st round: Panelists were required to indicate their level of agreement or disagreement with the following statement

“This indicator is relevant to the evaluation of Europe’s population health”

Panelists must choose one of the 5 following options, belonging to a 5-level Likert scale: “Strongly disagree”, “Disagree”, “Neither agree nor disagree”, “Agree” and “Strongly Agree”.

|2nd and 3rd rounds: The extracted results of the 1st round were presented at the beginning of the 2nd round, allowing panelists to maintain or revise their answers.  At the beginning of the 3rd round, panelists received again information synthesizing the answers of the panel after the 2nd round. Accordingly, these had the opportunity to maintain or revise their answers.

|Results: The group opinion (aggregate of individual opinions) was defined by calculating the percentage of responses given in each Likert item, for each indicator. The group agreement for indicator approval was determined by absolute majority (agreement above 50% and disagreement below 33.3%) and qualified majority (agreement above 75%)1.

Check out our support page (http://support.welphi.com/video-tutorials/) and watch our video tutorials to help setup your Welphi process today for selecting, or identifying, your project’s structuring components, whichever your framework might be! Start your free trial today to find our Welphi process “Demo – Selection of indicators for health index”, based in the Welphi case-study – Euro-Healthy.

1 A. Freitas and P. Santana “The selection of indicators to evaluate European population health”, http://www.euro-healthy.eu/documents/indicator-selection.