How to evaluate community engagement
(with serious credit to Dr Crispin Butteriss & Bang the Table)
The most common question asked of community engagement practitioners upon completion of their engagement exercise is: “How many people participated in your process?”
How many people did you hear from?
Not, “Who were they?” Not, “What did they say?” Usually, “How many?”
“How many?” is a valid question. But there are three immediate problems:
This question has become a proxy for the validity of a community engagement process and becomes the first port of call to discredit it;
There is often no context surrounding the expectations for how many people should have been engaged; and
There may be other considerations for evaluating a process that are equally if not more valid
Upon spending years seeing countless in-house and consultant engagement practitioners exasperated trying to get past the ‘how many?’ problem with internal stakeholders, Bang the Table began a process of brainstorming and investigating a range of other measures that might be as important, or more important (as context dictates) than “How many?”
Six measures of successful community engagement are in testing. Each in their own way more or less applicable to each situation, dependent on the goals of the organisation conducting the engagement.
Each measure is distinctly different and can be measured either qualitatively or quantitatively. I have tried to indicate ways in which each indicator could be measured or assessed. As follows:
1. Sociability
The degree to which participants interact between each other. This could include:
‘sharing’ their post or the whole process with others
‘liking/disliking’ each other’s contributions
Replying to each other and beginning a discourse
Measure: Quantitative; Count: shares, likes, replies
2. Tone
A measure of sentiment, emotion or controversy within an engagement process. Importantly sentiment can be nuanced between issues and parties (stakeholders) to that issue within an engagement process. Measuring Tone is one measure of the range and depth of positions associated with multiple issues surrounding a proposal.
Nature: Qualitative; Sentiment, Persistence of debate, Emotional content
3. Depth
Of the participants willing and able to be included in an engagement process, how involved were Engagement is often described as ‘shallow’ or ‘deep’ depending on the level of commitment required by participants to make their contribution (could be described as ‘high/low’ involvement).
Measure: Quantitative; Assessment of effort level made to participate
4. Impact
Both the level of change that is possible and achieved through an engagement process. Was there an explicit goal of the engagement (i.e. an outcome or decision) and to what degree was it achieved or another outcome reached?
Measure: Qualitative; Assessment of change initiated
5. Sustainability
The extent to which an ongoing relationship is formed and maintained between the organisation and participants through engagement process. This is exemplified by the degree to which participants return to repeatedly participate in an engagement (or range of engagements) either with an organisation or each other.
Measure: Quantitative; Frequency of participation
6. Reach
The number of people who were reached or communicated to by the engaging party through the process. Their ‘representativeness’ is also critical. They should be the “right” people, an appropriate population sample able to allow their views to stand on their own merits. In some cases these people will represent the ‘interested’ or ‘affected’ stakeholders, but consideration should also be given to including the ‘hard to reach’.
These are the people that have been made ‘aware’ of the process via appropriate communication processes.
Measure: Quantitative; Number of participants, Representativeness of participants
Please note that depending on the situation (and importantly the goal set by the engagement practitioner/organisation) these measures might be either: singularly important, used in combination or not important at all. What is important needs to be taken in context. I will provide some examples to illustrate how these measures could work toward the end of the article.
Some quick examples:
State/Provincial Government Statutory Engagement Process
A government agency with responsibility for waste collection and management engages government and industry stakeholders about industry reform. This is a statutory process (defined by a regulator), it is not designed for public participation, the content is reasonably technical and the stakes are important, but not critical.
How do each of the measures rank in this particular engagement process?
Sociability: This could be important, if the engagement process is designed to encourage participants to contribute through discussion.
Tone: Low. It is unlikely that the topic would be controversial or emotive.
Depth: Important. Participants may be required to contribute by making lengthy submissions, or to contribute in person at workshops (high-effort).
Impact: This will vary depending on the proponent’s willingness to state their goals.
Sustainability: Could be important. Over a period of time it would be likely that organisations, if not individuals, may contribute multiple times to a range of engagement initiatives. The agency may want to track its relationship building efforts.
Reach: Limited, as there are a select number of interested and sufficiently informed parties to participate in a process such as this one.
Conclusion
In this case, “How many?” is of very limited consideration. More likely sociability, depth and sustainability become the main drivers of the value of this engagement process.
National Infrastructure Development Engagement Process
The Nation Building Authority (NBA) is planning to build a new railway line, affecting thousands of commuters, but only requiring a small number of compulsory land acquisitions. It is statutory, the public will be involved, the content has been tailored to a public and technical audiences and the stakes are very high.
How do each of the measures rank?
Sociability: This could be important, especially if the NBA wants to assess whether there are cohorts of supportive or unsupportive stakeholders. The extent to which people share their interactions on social media or enter into debate with other participants will indicate factions around the project.
Tone: Highly important. This project has the potential to be highly emotive and controversial – which the NBA will be seeking to manage in order to progress the project
Depth: This will depend on the design, but here depth will be an indication of how concerned the participants are. If they are given a range of options (regarding depth), you may see ambivalent participants not contributing (i.e. just learning) or lightly engaging, while others more concerned may well go to great lengths to contribute. Depth will indicate level of investment. Depth may be highly related to Tone.
Impact: Important. Stakes are high for this process, so impact should matter. Is the NBA willing to be open about its goal and any changes to the design based on community feedback?
Sustainability: Maybe. Unless there are multiple rounds of engagement, this might not matter as much. It could be considered more of a one off, unless people will be requested to participate about iterations of the project design over months or years.
Reach: Projects of national significance need big audiences, and typically transport projects have no problem garnering them. The NBA will want to demonstrate that a sufficient sample of people (with similar diversity to the broader population) participated in the engagement, or at least came into the process to learn enough about it to be satisfied not to participate.
Conclusion
“How many?” does become more important in this scenario, but so does “Who are they?” The NBA will be interested in managing outrage and gaining approvals without delays and political damage, making the project into a ‘win’ if possible. Gaining a clear understanding of whether a group of stakeholders who are highly emotive, willing to engage deeply, and either strongly for or against the proposed project will be crucial to the government managing the issues surrounding their program.
————
Finally
I believe engagement practitioners can provide more value to the organisations they work with, and ultimately to the communities they represent, by expanding and enriching the understanding of engagement practice beyond specialist engagement practitioners. Moving beyond “How many?” seems to be, from my perspective, a useful first step.
I must credit Bang the Table for the genesis of these measures, I only intend to help practitioners to measure them. Bang the Table are the producers of digital engagement tools, but these measures are not intended for online use only. I see no reason why they could not be used to evaluate off-line and hybrid engagement processes.
I’d love your feedback on how these measures might affect your engagement projects. Are they relevant? Are they able to be measured? Are there more that have not been considered?
If you have any feedback, or you would like help measuring your engagement against these metrics, please feel free to contact me.