Executive Summary of Results from Online Survey of Planners

Research Context

The online survey to planners practicing in Ontario represents one of four research mechanisms designed to present the opportunities and limitations inherent to current public consultation practices.

In particular, many of the questions within this survey were designed to help elaborate responses from a public survey on consultation conducted in the Town of Pelham in March and April 2006.

We launched this online survey on May 4, 2006 and closed in on June 1, 2006. No reminder e-mails were sent to any of the participants during this four week window.

We expected this survey to evidence which consultation formats planners found to be most and least effective, and by extension, how well these favoured formats meshed with public expectations.

Survey Sample Size and Response Rate

With the determination that approximately thirty completed responses would be necessary to provide validity for a number of questions within the survey, a list of 799 names was selected from the OPPI membership list.

We selected the names with an eye for job titles that would indicate a greater degree of experience, and presumably, a greater familiarity with public consultation.

The majority of our respondents fell under one of five main job titles, as evidenced in Table 1 below:

Table 1: Breakdown of Potential Respondents by Most Frequently Cited Job Titles

Based on a number of literary sources, we anticipated a response rate between three and five percent. Removing the forty-five e-mail addresses that bounced back to us, we measured a final response rate of 5.17% from thirty-nine completed surveys in a sample of 754 recipients.

In terms of the top five job titles, the response rates varied significantly from the mean in a number of cases. Specifically, as Table 2 shows, a disproportionally low number of senior planners completed the survey, compared with a surprisingly high number of presidents and planning directors who did us the honour. Please note that due to the difficulties of correlating the e-mails that bounced with job title, these calculations are based on the number of potential participants. This produces a slightly lower average response rate of 4.88%.

Table 2: Response Rates for the Most Frequently Cited Job Titles

There seems little to hypothesize about this phenomenon except to acknowledge the possibility that public consultation is a topic area of significant expertise or interest to these cohorts of planning professionals.

Survey Data and Observations

Our survey consisted of nine questions, requiring participants to answer the first six and allowing them to complete the final three at their discretion.

The first question asked participants to categorize the length of their experience in the planning profession. The most frequently cited response (mode) and the median can be found in the 21-30 years of experience category. As can be seen in Table 3, no respondent had less than six years of experience.

Table 3: Breakdown of Years of Experience for Survey Respondents

Our second question required participants to indicate what percentage of their staff hours were devoted to different consultation formats. Thirty-seven of thirty-nine gave valid responses (94.87%).

As Table 4 indicates, public meetings, stakeholder meetings and workshops placed first, second and third respectively in terms of the highest percentage of hours devoted. These same formats were also the most likely to reach the minimum thresholds of one, ten and twenty percent.

Table 4: Frequency and Percentage of Hours Devoted to Various Consultation Formats

All respondents indicated that at least some percentage of staff hours were devoted to public meetings as a consultation mechanism. Similarly, thirty-five of thirty-seven (94.59%) respondents cited stakeholder meetings as a format for which at least one percent of staff hours had been devoted, while thirty-two reached the same conclusion regarding workshops.

Also interesting is that unlike opinion polls and the ‘other’ category, for which there is a single digit average percentage and relatively infrequent citations, there are a large number of 1%+ citations for interviews, surveys and task forces.

This leads us to conclude that outside of the main three mechanisms (public meetings, stakeholder meetings and workshops), planners may be willing to experiment with other formats – but are much less likely to lean on them exclusively for their policy decisions.

The third question asked participants to provide their own definition of effective public consultation. We used a keyword analysis to redefine the data into more manageable quantities. Words such as ‘opportunity’, ‘methods’ and ‘input’ fell under the category describing the process of consultation. On the other hand, words like ‘average citizen’, and ‘representative’ lumped together to describe the participants of consultation, while words like ‘consensus’ and ‘solutions’ came under the array of terms describing the products of consultation.

In the end, as Table 5 shows, the definitions offered by the thirty-nine participants leaned heavily toward process-orientated words, as opposed to ones steeped in the terminology of participants and results.

Table 5: Distribution of Words Cited in Definitions of Effective Public Consultation

Just two of the definitions^1,2 possessed an equivalent number of references to each of process, participants and product.

The fourth question in this online survey requested the participants to rank consultation formats against each other in terms of their effectiveness for achieving the previously defined effective public consultation. Thirty-four of the thirty-nine respondents (87.18%) ranked at least some of the formats provided.

The results, as Table 6 shows, confirm that public meetings, stakeholder meetings and workshops continue to receive the lion’s share of first, second and third place rankings. However, the inverse relationship between the first and second place rankings of public and stakeholder meetings is interesting to note, as are the overall ranking scores.

Table 6: Frequency and Average Ranking of Various Consultation Formats

In particular, we find that – despite only being ranked in the top three twelve times – focus groups are perceived to be the second most effective consultation format. Its ascent throws public meetings and workshops into a tie for third place with an average ranking score of 4.4.

Stakeholder meetings, despite coming a distant second in terms of average staff hours devoted, collect more top three rankings than any other format (19) and have the lowest average ranking score (3.9).

Also noteworthy are the number of first place rankings associated with consultation formats that hold comparatively high average ranking scores. Two prime examples include opinion polls (four first place ranks and an average score of 6.8) and citizen panels (two first place ranks and an average score of 6.0).

Combined with the data received in question two, these observations lead us to believe two things. First, planners tend to spend more time with public meetings – perhaps because they feel legislatively required to hold them or because they demand more of a staff presence – than they do with any other consultation mechanism.

However, they seem to agree that stakeholder meetings are more efficient in terms of achieving their definition of effective public consultation and appear apt to use them when it is time to get things done.

Second, there is a fairly even spread of first place rankings across the consultation formats presented. Only task forces failed to receive any top rankings, while even opinion polls and citizen panels – which were otherwise disdained – received four and two first place ranks respectively.

This seems to indicate that what is one planner’s trash is another planner’s treasure, and that in the context-specific recollections of the professionals surveyed, each of the methods has proven successful.

In short, while planners may indicate a general preference for certain formats, it cannot be said that they have agreed on a catch-all mechanism for receiving public input.

We designed the fifth and sixth questions to work in tandem to help discover if planners perceive a difference between the way they view the effectiveness of their consultation methods versus the way the public views them. In order to gain a measure of the public’s evaluation of current consultation mechanisms, we simultaneously asked this question using the same 1 to 10 scale to the fifty-six Town of Pelham residents that had previously attended a consultation exercise.

In this self-evaluation, the planner suggested an average score of 7.31 with both a mode and median of 8.00. However, when asked to assume the viewpoint of citizens evaluating their methods, the mean fell dramatically to 5.92, with a mode of 7.00 and a median of 6.00 (see Chart 1).

Chart 1: Relationship of Planner and Public Evaluations of Consultation Methods

This change in opinion is derived largely from the twenty responses which indicated a drop of either one or two points from the self-evaluation score. As can be seen in Chart 2, only two planners suggested that the public held a higher opinion of their consultation methods than they did.

Chart 2: Change in Respondent Scores Between Self-Evaluation and Public Perception

The data suggest that planners are cognizant of a divide between the public’s perception of consultation efforts and their own. However, despite the fact that the mean score of public perception offered by planners (5.92) rested considerably closer to the true public viewpoints (5.12) than their own (7.31), not a single planner suggested that the public might assess them with a score of one out of ten. This is a curious finding in the face of information from Chart 1 that the public’s second most frequent evaluation of public meetings was a score of one.

The seventh question was optional and requested participants to describe their most positive experience with public consultation. Again, we used a simple keyword count to analyze the data, picking out references to specific consultation formats.

Thirty planners responded to this question, making thirty-two references to various consultation methods in a positive light against five framed in the negative.

Once again, workshops, stakeholder and public meetings received the greatest number of favourable experiences, as Table 7 displays. The strong showing of the ‘other’ category is largely aided by three references to open houses. However, in the absence of specific wording, and because open houses can contain elements of many of the other format options presented, they have not been considered for one of the top three placements.

Table 7: Distribution of Consultation Format References in Best Experiences

What is especially intriguing is the number of references made to public meetings in a negative fashion. The equivalent number of positive and negative responses is as telling as the language in which these remarks are couched. Examples include: “…something a public meeting can’t generate…” and “(m)ost people do not like to speak in front of large groups at public meetings.” In all cases, the respondent indicates using an alternative consultation format as a reaction to a negative experience with a public meeting.

Yet again, the kudos for various consultation styles is reasonably well disseminated. Only opinion polls go unmentioned by planners in this question.

The eighth question parallels the previous query in that it asks planners to describe their worst consultation experience. Again, thirty participants completed this question, twenty-six of which had responded to question seven.

The answers generated nineteen references to consultation formats in a negative light, and zero framed in the positive. As Table 8 reveals, the greatest amount of disdain is held for public meetings with workshops and other (open house) formats in a distant tie for second.

Table 8: Distribution of Consultation Format References in Worst Experiences

It should be noted that while we did not count a single, specific reference condemning stakeholder meetings, there were four warnings of the effect that single or narrow interest groups can have on consultation as a process.

That in mind, it becomes slightly more logical to suggest that those formats which are most frequently used (workshops, stakeholder and public meetings) are also the most likely to garner the greatest amounts of both praise and scorn. Be that as it may, the dramatically large gap between overall positive (4) and overall negative (14) comments regarding public meetings seems strangely out of keeping with the high percentage (32.3%) of staff hours devoted and high number (9) of most effective rankings that it received.

Our ninth and final question sought a description from planners as to the factors that might be limiting high quality consultation. Though optional, thirty-four of the thirty-nine (87.18%) respondents provided their thoughts on this query. We then broke these responses into three categories that assigned the limiting factors to either the planner, public or political processes.

As Table 9 evidences, our survey respondents placed most of the factors that would limit good planning from being achieved through consultation on the public – followed by the political processes and planners respectively.

Table 9: Frequency of Limiting Factors Cited by Category

Naturally, we found certain types of responses mentioned more frequently than others within the categories. For example, within the public category, the idea that consultation brings out only input from narrow points of view was cited five times. Honourable mention goes to the notions that the public – in the opinion of our respondents – believes that the public fails to react until the point of crisis and exhibits a lack of understanding.

The most frequent response within the political process category suggests that there simply isn’t enough time and money made available to achieve good planning through consultation. Three references recognized that consultation is just one part of a larger process and cannot alone produce good planning.

Finally, our survey participants made a total of six statements which indicated that the planners need to effect changes themselves in order to create good planning through consultation. Three of these assertations held that good planning is being held back by inefficient or ineffective consultation techniques.

What this data appears to show is that planners, the front line administrators of the democratic political model for land issues, are somewhat more likely to suggest that the factors which limit good planning through consultation are outside of their control.

Perhaps this is a natural human response, or perhaps it is absolute fact. In the absence of further investigative data, this distinction is certainly very difficult to make. However, what it should open our eyes to are the opportunities to improve consultation and its general perception with the public that lie beyond simply altering consultation formats.

Final Conclusions

In the end, the major data observations can be summarized by the following points:

1. Planners cite public meetings as the consultation format for which they devote the greatest percentage of staff hours and are the most likely to consider effective;



2. However, this same group made fourteen statements that paint public meetings in a negative light versus just four which speak of them in more glowing terms;



3. Workshops and stakeholder meetings ranked relatively high across the board in terms of staff hours devoted, ranking scores and the number of positive references in previous consultation experiences;



4. Planners generally assessed their consultation methods in a relatively positive manner, evidenced by the mean score of 7.31. However, they seem to recognize that the public holds a lesser opinion of their efforts, estimating that the public would give them an average score of 5.92;



5. Finally, a high percentage of respondents acknowledged that there are at least one or more factors which limit the ability to achieve good planning through consultation, but tended to distance themselves from them.