John Ganz’s essay from a few weeks back, “Against Polling,” isn’t really about polling. Or at least it isn’t exclusively about polling. Instead it’s an attack on what he calls, in a later post, “vulgar positivism”:1 a sort of reductively empiricist approach to politics that privileges issue polling and statistical analysis over the more humanistic virtues of rhetoric and values-based discourse. Or as he put it in “Against Polling”:
Ultimately, the worldview of the data guys is based on a giant mistake: there’s an objective world out there, and it doesn’t change. You grasp it with a technical means and then try to apply that objective knowledge. This will create constant befuddlement and surprise as the people don’t behave the way you want them to. This, in turn, will create pseudoscientific behavior like altering the theory to say you were always right and how this anomaly proves it if you look at it the right way. But politics is based on a fundamentally changing world: of opinions, of historical events, of the public’s feelings and imagination on issues. A great politician recognizes changing tides and gradually shapes their public: they go from speaking to a crowd to leading it. The way to learn how to do this used to be obvious: study the words and actions of politicians past, and try to get practical lessons. This is what the humanities teach: how to deal with the world of human affairs as it is, not as it’s been abstracted and dissected by the scientists. Study the classics and the modern imitators of the classics. Go back to an era before endless polling, when politicians were relying on the responses of their audience.
The “data guys” are, of course, a major force in Democratic Party politics. They are also both inveterate posters and an object of hate for a certain other class of inveterate poster. So, needless to say, Ganz’s essay—apparently to his mild regret—instigated a whole lot of posting.
The most detailed rebuttal I’ve seen was from Eric Levitz, another writer I happen to admire. Levitz seems genuinely distressed by what he takes to be the main implication of Ganz’s argument: that it “serves to insulate progressives’ intuitions about electoral politics from any empirical challenge.” In other words, without the tools of the data boys, we have no grounds on which to judge the truth or falsity of political propositions. Abandon polling and “everything is permitted,” as Levitz writes — paraphrasing, perhaps a little revealingly, Ivan Karamazov’s line about what it means to live in a world without God.
Before I get to articulating my own view on this whole thing, let me present my credentials. I tend to think of myself as a writer first and foremost, but I do have a foot in the data guy world. I took classes on economics and econometrics in grad school, and I learned to code in Stata. (I tried to teach myself R too, but it didn’t stick.) For my capstone project, I evaluated a pilot program for the California Department of Social Services, which involved a lot of data cleaning and some basic OLS regressions. (Conclusion: null result.)
Later, when I was working for the California state legislature’s budget office, I was part of a team that had to model enrollment in the state’s Medicaid program, Medi-Cal, to calculate its impact on the state budget. I’m not a data scientist by any stretch of the imagine, and I don’t have any particular aptitude when it comes to numbers, but I at least know enough to interpret the work of actual data people and discuss it semi-intelligently. More to the point, I’ve actually used the tools of data analysis in an applied setting to help make policy.
It’s because of those experiences that I find myself closer to Ganz’s position (and Steve Randy Waldman’s) than Levitz’s. But I have some quibbles, mainly having to do with the terms of the debate. Both Ganz and Levitz fall into the conceptual trap of treating quantitative analysis as a distinct field of human reason, separate from the messier world of debate, argument, and rhetoric.
Ganz complains that “people try to apply scientific principles to the arts, and vice versa, with the result being a giant muddle.” Levitz writes that “scientific methods impose far greater constraints on motivated reasoning than humanistic inquiry does.” My own experience has taught me that data science — at least when applied to politics and policymaking — is very much an art, and doing it well requires a pretty firm grounding in what Levitz calls humanistic inquiry. Quantitative analysis is not a superior, more objective form of reason, but it’s also not just obfuscatory bullshit. It is nothing more or less than an extension of older forms of deliberative argumentation.
What this means in practice is that quantitative analysis typically involves a fair amount of humanistic inquiry both on the front end (model design) and the back end (interpretation). What happens in between could be more reasonably described as “pure mathematics” or “pure science,” but even there the disciplinary borders are a little fuzzy. When I was an undergrad philosophy major, I took a class called First-Order Logic, which taught us how to express philosophical arguments in terms of formal proofs. But while the specific form these arguments took was drawn from mathematics, the proofs themselves could have easily been converted into plain language arguments. The same underlying structure governs both a rigorously logical argument and a mathematical model.
Of course, logic on its own isn’t everything. Model design and interpretation both involve subjective judgment calls. Whether a researcher makes the right judgment calls depends on some combination of experience, good instincts, subject matter expertise, and luck: the very same properties that someone needs to successfully engage in Ganz’s preferred mode of reasoning.
In polling, some of those design questions might include: What weights do you assign to different demographic factors such as race, age, and party affiliation? How do you adjust your weights based on the evolving face of the electorate? How do you deal with selection bias caused by high non-response rates? Even the values used to calculate margins of error are somewhat arbitrary; the commonly accepted p-value threshold of 0.05 is more a matter of convention than anything else.
Pollsters can (and almost certainly should) defer to professional practice when it comes to error bars. But coming up with plausible answers to the other questions requires exercising some political judgment and doing a fair amount of educated guesswork. The nature of this guesswork was largely obscured in 2024 by the fact that pollsters herded around a dead heat consensus; they didn’t have to defend competing theories of the electorate because their models were all producing more or less the same results. But every poll is essentially extrapolating from a theory of the electorate. Good data scientists are upfront about their theories so that their colleagues—including subject matter experts who aren’t necessarily data scientists—can stress test them.
You also need to understand the theory in order to properly interpret results—and the limits of those results. I’ve been thinking a lot about interpretation since reading Matt Bruenig and Kelsey Piper’s exchange regarding universal basic income in The Argument. It wasn’t Piper’s argument and Bruenig’s counter-argument that struck me so much as Piper’s reliance on randomized control trials to make her point.
RCTs — where some people are randomized into a treatment group and others into a control group — are incredibly powerful studies in the right context. All else being equal, they can tell us whether a particular medication is effective in treatment irritable bowel syndrome, or whether — again, all things being equal — homeless people who are randomized into a particular type of housing are more likely than those receiving usual care to be housed in twelve months. But the key assumption here is the all things being equal part. RCTs are designed to test an isolated treatment’s effect on an individual; by their very nature, they cannot test something that alters the whole background context in which both the treatment group and the control group exist.
All of which is to say you can’t actually do an RCT of universal basic income. You can run an RCT of unconditional cash transfers, but that’s not exactly the same thing. UBI is by definition universal: instituting something like a real UBI program anywhere in the United States would transform the entire local economy. You might isolate a “control group” of a few hundred people who were not subject to the treatment, but that wouldn’t be a real control group, because you wouldn’t be able to isolate them from the systemic consequences that would follow from everyone else in the area benefitting from a universal basic income.
Another example: Imagine if we tried to do an RCT of congestion pricing before its implementation in New York City. So you create a treatment group of people who get charged some congestion fee whenever they drive into the city, and a control group who are not subject to the fee. We can reasonably surmise that the members of the treatment group would probably drive into the city less, and maybe they would opt to take transit more frequently. What we would not be able to test in this model is the congestion charge’s effect on congestion; that’s a large-scale systemic impact, not an individual treatment. If someone in the treatment group still drives to work, they’ll inevitably face the same delays in the Lincoln Tunnel as the control group.
It’s not anti-empiricist to point out these limitations. In fact, acknowledging such limitations is precisely what a good quantitative social scientist does, which is why many academic papers in the discipline include a litany of caveats and theories about possible confounding factors. But many consultants and professional data pundits don’t have such scruples, journalists mostly ignore the caveats, especially when it comes to polling.
This creates an unhealthy dynamic where some people (who are usually not very well versed in quantitative methods) treat statistical analysis as if it were prophecy, while others (who are also not well versed in quantitative methods) treat it like obscurantist snake oil. But when used properly, it’s neither. It’s just a subset of human reason, with all its weaknesses and occasional flashes of ingenuity. It’s a tool among many other tools, imperfect and difficult to use correctly, but occasionally very useful.
1 The title of this post is lifted from that follow-up post. Ganz writes:
I think the reason people enjoyed my piece is precisely because it was highly rhetorical. Levitz is flabbergasted that I could dare be Against Polling or dare say that it’s “garbage” or “bullshit,” and then later qualify my claims. Yes, I was exaggerating for effect, and it worked! People paid attention. Then I explained more thoroughly what I meant, although not to Eric Levitz’s satisfaction. If I wrote, “Some Slight Issues with the Positivist Approach to Politics,” it would put people to sleep.
I don’t make my living off this newsletter, so I have fewer compunctions about putting people to sleep.