How to run a defensible public consultation analysis
Public consultations carry a legal duty. Every response must be conscientiously considered. That standard doesn't bend for volume. Whether a consultation attracts three hundred responses or thirty thousand, the analysis behind the final report has to be able to withstand a legal challenge and a public one.
Where volume breaks manual analysis
Spreadsheet-based coding works for small consultations. It breaks down once volume climbs, not because the themes become more complex, but because consistency does. Two analysts working through thousands of free-text responses will code similar comments differently unless the process is tightly controlled, and that inconsistency is exactly what a legal challenge looks for.
A hybrid approach
The answer isn't to remove people from the process, and it isn't to leave everything to a language model either. AI is genuinely useful for the first pass, ingesting responses, clustering similar comments, and surfacing candidate themes far faster than manual reading. But AI also hallucinates and drifts, and a first-pass theme list is not a finding.
The findings that go into a ministerial briefing need a second layer. An analyst independently reviews the AI-assisted clustering, checks edge cases, and confirms that minority or dissenting views haven't been smoothed away by the model's preference for the majority pattern. That verification step is what makes a finding defensible rather than merely plausible.
What good practice looks like
A few principles tend to separate consultation analysis that holds up from analysis that doesn't.
- Design the survey with analysis in mind. Free-text questions that are too broad produce responses that are hard to theme consistently, no matter how good the tooling is.
- Keep a visible audit trail. Every theme should be traceable back to the responses that generated it, so a challenge can be answered with evidence rather than assertion.
- Report minority views explicitly. A consultation report that only reflects the majority view invites the strongest form of legal challenge. Dissenting themes need their own space in the reporting.
- Separate the AI-assisted pass from the verified pass. Don't let automation quietly become the final word. Someone accountable should sign off on every theme that reaches a decision-maker.
The payoff
Done well, this hybrid approach doesn't just protect against challenge, it changes what's operationally possible. Teams that used to be constrained by how many responses they could manually read can now handle consultation spikes without a proportional increase in headcount, while still producing findings that a policy team can stand behind in public.