Why Vision-Language Models Need a New Urban Playbook

Vision-language models (VLMs) are getting a lot of attention for their ability to generate structured descriptions of street-level imagery. They're becoming tools of choice for urban planning tasks like streetscape auditing and public consultation. But here's the catch: these models aren't about binary facts. They're about interpreting subjective appraisals, and that's a whole different ballgame.

Disagreement and Abstention: The Real Data

urban perception, disagreement and abstention should be viewed as important measurement outcomes. The study focused on a benchmark of 100 Montreal street scenes, analyzed along 30 dimensions by 12 participants from seven community organizations. This is a great start, but let's be real. How do we measure success when humans themselves can't reach a consensus?

We often treat model alignment like it's gospel. But what if the label space and scoring policy were negotiable artifacts? It's key when these outputs guide urban governance. The builders never left. They're just being asked to dance on shifting sands.

Model vs. Human: The Agreement Gap

The research revealed that model agreement with human consensus varies. For instance, in the appraisal dimension dubbed 'Overall Impression,' models and humans were sometimes on different planets, showing mismatched distribution patterns and varying rates of labeling something as 'Not applicable.'

This isn't just a technical glitch. It's a fundamental question about how we define and measure consensus in subjective settings. Are we asking machines to do what even humans struggle with?

Recommendations for a Clearer Picture

It's time to rethink benchmarks. Actions for benchmark creators, model developers, and institutions should focus on making uncertainty and assumptions visible in evaluation reports. Why? Because floor price is a distraction. Watch the utility. Models should adapt as much as the cityscapes they aim to interpret.

Urban governance is evolving, and vision-language models can play a important role. But without addressing how we benchmark their outputs, we're just throwing darts in the dark. The meta shifted. Keep up.