Ep 05 – Polling Roundtable Pt. 2

[0:00:00 Speaker 0] is pulling an art or a science. Welcome back to the connector, where we bring together innovative, groundbreaking and collaborative research inside the U. T. Austin political science universe. I’m your host, Stewart Tendler. This episode is the second conversation we’re having about polling its accuracy, zin, inaccuracies, uses and misuses. It has become popular perception that the science behind polling is flawed. But our guests Robert Luskin, Darren Shaw and Chris Malaysian have a different take on things is actually the art of the polling, not the science, which poses the biggest challenge. How should questions be worded? In what order should questions be asked? It turns out that the answers to questions such as these can create variation and polling results.

[0:00:52 Speaker 2] We

[0:00:52 Speaker 0] then move on to a discussion of elections with specific comments about the 2016 election cycle recorded in March. In the run up to Super Tuesday, we explore what the polls might have been telling us about November 2016. At the time, polls were suggesting that the strongest candidates for each party come November would have been Bernie Sanders and Marco Rubio. We know now that contest will never happen, however, because it was so early in the campaign season. We also know that the polls were revealing relatively little about eventual election outcomes. There is a timeline to presidential elections are guests. Chris Malaysian wrote a book about it. And as the election gets closer, the polls tell us more. Darren and Chris closed the episode with some of their thoughts on the question, pointing out, for example, that when an election is a dead heat, small swing could be the difference between winning and losing.

[0:01:50 Speaker 1] What is the biggest problem facing polling? And as we’re thinking about this, we’ve focused on technical issues, this kind of responding to the Trump quote about the science behind polling. If I were asked, I would actually go off the board and responding to what I think is the biggest problem with Ploy, I would actually suggest that it’s unmeasurable as opposed to measurable air and, in other words, response options. Question wording. You know, these sorts of things that can bias and influence results. I think we spend actually a lot of time and the sort of people who kibitz from the sidelines on poles talk a lot about sample sizes. The sample frames and non response issues and things like all of which are issues. But I think simple things. You know, if you’re talking about the science versus the art, I think there’s more air introduced by the art of polling in many ways and the science of it. And I want to push this argument too far. But, you know, all of us teach classes in which we will put questions side by side that purport to measure the same thing and produce extremely different results because of the way the questions crafted or the response options. And I think this is underrated and under discussed and, you know, creates problems that the news media don’t like to talk about these things because there’s a sense that if you discuss these things, you undermine the results. And of course, they paid a lot of money and want to talk about these things on, and they don’t really want to raise these questions. But that’s I think a more intelligent discussion of polling would probably focus a lot on these things because they do matter. Would

[0:03:16 Speaker 2] you say that’s equally a problem for candidate preference, which seems a more straight forward thing to ask about as compared to opinions.

[0:03:24 Speaker 1] You know? Not as much, but think about the complexities of a candidate question. Where do I put the ballot in the context? Ah, standard pulled, I put it after the presidential approval measure before, after I talked about hard immigration or party idea afterwards. Do I have the vice presidential candidate listed as a response option, which is what you have in the ballot? You know? Do I replicate the ballot entirely? In which case? Now I’ve got a state by state problem because in some states Daffy Duck can get on the ballot in other states. It’s only the top two candidates, you know. Do I push? People say they’re undecided initially, do I say, Well, if you had to say, Which way would you say you lean? These are all questions, even in a I agree with Bob’s point, even a straightforward question where I think we’re going to get If Chris and Bob and I do a poll independently, I think we’re gonna get pretty similar results. But you think about this multitude of decisions that you make. All of them potentially could influence the results that you get

[0:04:16 Speaker 2] there. There’s an interesting study done by one of our former graduate students. Partly by him has actually done before he came here. This is not your usual and it was co written with Patrick Fournier. And they did a experiment within the questionnaire of the Canadian National Election Study some years back. And what they varied from respondent to respondent randomly was the placement of the vote choice item. And they found that if you ask it at the beginning, it’s not nearly as predictable as it becomes when you ask him at the end. That’s because the process of being interviewed of answering questions, getting tidbits of information they pick up, incidentally, from the questions it both informs them to agree and still more so stimulates them to think about the choice. And so it’s a slightly better informed, slightly better reason choice. By the end,

[0:05:17 Speaker 3] I think one of the things that we haven’t covered well, I’m a little surprised, more than a little surprised. Actually, we haven’t is what’s going on in 2016 with the polls now are telling us about what’s happening in the course of the primary season, but also what the polls now are telling us about the matchups, these hypothetical matchups or trial heats that we survey organizations regularly acts. Ask, um, tell us about what’s gonna happen in November, assuming we get the right candidates in these trial eight matchups that some That’s interesting. I think

[0:05:47 Speaker 1] it is. The thing that strikes me about immediately after Chris’s observation is the difference between Bernie Sanders as the Democratic candidate versus Hillary Clinton. Sanders pulls terrific Lee against potential Republican nominees as opposed to Hillary Clinton and then the variance in Trump as a Republican nominee that I’ve seen him fairly consistently polling least well. But I’ve seen a few instances in which he’s actually pulled very well,

[0:06:15 Speaker 3] typically at least

[0:06:16 Speaker 1] typically, least well, although I would guess the greatest variants would probably be associated with Trump moving forward, that is, I could see Trump staying where he is. I could see him gaining 10 points in the ballot. I could see him losing 10 points in the ballot. I wouldn’t wouldn’t probably guess that for Cruz or Rubio or cases, but the Democratic maybe, but Sanders maybe, Yeah, I do. Yeah, that’s

[0:06:38 Speaker 2] the, uh,

[0:06:39 Speaker 3] likely to gain as much as he’s likely to lose,

[0:06:41 Speaker 2] I think he’s more likely to lose sight. I mean, I think that these choices hypothetical, though they currently are for November. Uh, you know, probably the polling results probably are reasonably accurate if for what we’d see if the election were held today. But there’s a lot of time between now and then a lot of campaigning and counter campaigning to do. I think in the case of Sanders, the Republicans have given him a free what ride so far. That’s partly because they think killer is going to be the nominee. But it’s also partly, I think, because they they like Sanders to be the nominee, thinking him a weaker opponent. And at any rate, whatever the motivation wants, the campaign for the general election begins of Sanders is the Democratic nominee. That freedom free right? It will be completely over, and he’ll be hit hard. So my guess is his his poll results not not because the poll is inaccurate for preferences as they stand, but those poll results will be significantly off for November.

[0:07:53 Speaker 0] Let’s be a little more explicit than what are the polls saying right now? Who are the best candidates for each party today. Here the polls are saying Sanders is

[0:08:01 Speaker 3] the illusory Bubo Rubio. It’s the

[0:08:04 Speaker 0] Rubio on Sanders

[0:08:04 Speaker 1] nationally right now, the Sanders polls significantly better than Hillary Clinton on the Democratic side. By significantly, I mean, in a ballot race where Clinton is, you know, roughly even say, with With Ted Cruz, Sanders will be up nine points. I mean, is

[0:08:20 Speaker 3] the differences eyes wide on the other parallels? But still, it’s pretty big 56 points.

[0:08:24 Speaker 1] And what we’ve seen now is it. Rubio tends to pull strongest amongst the four remaining Republican candidate at this juncture. You’re talking about Trump, Kasich, Cruz and Rubio? Um, Kasich, I think, largely because people don’t really know who he is. Um, and so you get a wide number of undecideds and the undecideds to come disproportionately from Republicans or independent. We don’t know who is, whereas the Democrats know who their candidates are. So, uh, so you know, the obvious conclusion would be the Kasich probably has the most ground toe pick up. Most Republicans know who cruises and a sort of on board. I think the ballot that’s most kind of instructive and informed would be a Hillary versus Cruz matchup because Republicans basically know who cruises and Democrats know who Hillary is and they start assert themselves accordingly. Uh, Rubio and Kasich a little more wild cards, and Trump is just extremely difficult to figure. Some Republicans aren’t on board with him, and so he loses support compared to a generic Republican candidate. He may be gains back, a slight amount because there’s some independents or Democrats who may have already crossed over. But it’s very difficult is Chris and Bob was suggesting to figure out what what, ultimately, a Trump Clinton or Trump Sanders matchup looks like a Trump Sanders matchup. You beat a delight from us from a process and research point of view. May not be with the Republican wants, but

[0:09:47 Speaker 3] but we know that polls at this point in the election year tell us relatively little about what’s gonna happen on Election Day. Right? So I mean, beginning you election your based on history. The relationship between the Poles and the vote is literally zero, so you really, really can tell who’s gonna whether one candidate is gonna win or another, regardless of what the polls tell us. But by this point in time and the cycle, we’re really starting to ramp up. We go from, like, zero to 50 in the 1st 60 to 90 days, which is what we’re right in. The middle of eso were part of the way there, but we’re not probably about 1/3 of the way there, maybe 1/4 of the way there. It could be more or less, depending on the candidates, right? We don’t know who they’re gonna be for sure.

[0:10:22 Speaker 1] I do think that there’s, you know, when people have the observation well, there’s nothing to be in. This not is not what Chris is saying, but when people will say like, Well, there’s nothing to learn from the polls. Well, that’s somewhat nonsensical. I mean, what you do know is is in an open election, and this is an open race with the incumbent. Leaving Hillary Clinton is is a fairly well known commodity. Um, you know, there’s not gonna be a lot of change. I mean, she may be able to reinvent herself a little bit of the convention. You never know. But she’s such an established product that by the standards of an open election and these would be, you know, 1988. Um, you know, 2000 and again in 2008. We don’t have many cases, really, In recent history. Um, you know, she’s a pretty well known commodity, and I think it will be hard for her to move the dial very much on that. You know, she’s fairly well known on the Republican side, it’s more kind of typical of an open race, and I’m definitely Chris on that. Who knows whether these Republican candidates should they grab the nomination, be able to consolidate the base, expand the base or, you know, completely underperform?

[0:11:25 Speaker 2] Yeah, I agree with that. I think that for the Republicans, probably the safest choice is not one they’re likely to make is Kasich. I don’t think you do badly. Uh, the others. There’s much more uncertainty about, I think, and, um, they have greater strengths and weaknesses. Of course, Mitt Romney thinks the safest choice with Mitt Romney.

[0:11:51 Speaker 0] Cruz is back to your point. You’re saying it right now. The polls are telling us relatively a little about how the election’s going to turn out, but at some point they tell us a lot is kind of what? What you’ve discovered correct right in that. At some going, there’s a time there’s a time to it. There’s a timeline.

[0:12:13 Speaker 1] I believe it’s time line, right, Chris? Presidential elections. Yes, that would be a good book.

[0:12:21 Speaker 0] What? So when I mean is there a month? I mean, are we talking like July or something like,

[0:12:26 Speaker 3] Well, if the beginning of the year zero by, say, it’s April early April mid April, it’s were about what? About 50% of the story. So it’s a pretty pretty linear, pretty. That kind of we kind of get new a quiet season, typically when we’ve started out the candidates and maybe this year will be a little different. I think is pretty least I once I may be on both sides, which case things will drag on as we’re sorting out who were matching up by by the end. By the end of March, we usually know who the candidates are, and this year we probably will know. It looks like, well, no one be less clear. We’ll know the other how it makes it harder for people to actually be. You know, as Bob said, you know what the, you know, the degree to which ones ah was focused on the decision that really matters right beyond is going to affect the seriousness, the reliability of their of their responses. And so we don’t know really what, You know, they’re gonna mean something. Maybe a little bit less when the candidates are unclear that when they are clear, usually we nobody in the march and then things got quiet down, and then they wrap back up. Ah, in the summer, with with the conventions and people you know, people are picking their vice presidential candidates that conventions actually occur. People are really re focused on politics. And, um, you know, they may flirt early on, right? Oh, yeah. I’m a Republican And me how consider voting for the Democratic candidate until one finds out that they really are a Democrat? Uh, yeah, they’re they’re saying democratic kinds of things, and that’s what conventions are good for.

[0:13:51 Speaker 0] I mean, my last question be that sort of, ah, campaign becomes a campaign story less than a polling story because now we’re talking about that’s sort of the other side write how much campaigns can move public opinion. Right? So you’re saying. At some point, the polls are giving an accurate demonstration of what opinion really is and how it’s gonna be expressed at the polls because the campaigns are no longer going to move that is that

[0:14:22 Speaker 3] right? I think I think the way turn and I would characterize it is it’s not. It’s not like we have a static world, right? That the polls are becoming increasingly informative about right is a dynamic world, and it’s not dynamic, necessarily in the sense that people are changing their positions right. But they would they could be. But so that in that sense they may be static, maybe the same candidates with the same positions. But that’s not just what the campaigns inform us. Engage us, um, ask us to pay attention and and and and think seriously about what we’re doing and we do. And so, as a result, yes, polls become more informative. So campaigns and one you are doing that are having that kind of effect. But maybe not in the same way. I think that other people might think of campaigns and the way that it’s sort of distortion and gaffes and, you know, errors, which is a way that I think some, some people characterize campaign effects.

[0:15:26 Speaker 1] People think of the phrase reason political science sometimes is enlightened preferences, which is not quite what it sounds like, but it is. A useful is a useful phrase, I think, and that is that we assume that people are gonna move in some sense in a predictable way. The Republicans are gonna figure out who their nominee is, and Democrats will figure out who their nominee is. That, um, you know, people will get a sense of the economy and whether the in party has done a good job of managing the managing the economy and that prior to people focusing on the election in this period from, say, January to March that Chris was talking about not knowing who the candidates are and not really clawing into politics, that there’s a lot of noise Penis to these polls, and so they’re not terribly predictive. But once people focus in begin to sort themselves once you know, they kind of move in the places that we sort of expect them to given conditions, then the polls become very accurate and you can get, you know within within the sort of smallish band, you know, of 2 to 3 to 4 to 5 points. You can see these movements in the polls that occur in response to the events of the campaign, whether it’s a convention or, um, you know, some sort of change in the dynamic a candidate picks up on a particular issue theme. Or there’s a gaffe. Um, and

[0:16:40 Speaker 0] and there’s a

[0:16:41 Speaker 1] question about how much we care about those things you know, are these sorts of effects that the polls oftentimes pick up on short live or, you know, do the result in kind of a sea change in which you know a two or three point difference becomes the new baseline against which we judge the expectation of the election and, you know, so from the outside, people will look at what the conversations in political science and say that what you guys are talking about, these really small changes and Artie within margin of eras were talking about earlier. It’s like Well, yeah, yes, but if they’re consistent, and if you’re talking about a country that’s essentially split down the middle when it comes to national elections, they become extremely consequential. Um, and there are the there. There are elections where people don’t get to the places we think they’re going to be getting to, you know, in 2000 is the famous example where all the external conditions suggested that Al Gore should, you know, capitalize on Bill Clinton’s Ah, popularity and the relatively robust economy. Or there’s some dispute about that at the end. But, you know, and be a very strong candidate to carry on the Democratic controlled White House. And yet ah, he ended up in a tie, basically with George W. Bush. And so is so these air tendencies and things that we we expect to happen more often than not happened. But as as we’ve seen from the last six months, Ah, you know, you go out of business predicting politics, and, you know, I feel like a weatherman here, right? You know, we’re predicting the weather and we said it was gonna be sunny and it’s rained. And now people are questioning our ability, our knowledge of any kind of political phenomena and you know well on the one. And I think that’s kind of unfair. On the other hand, we probably should have been making those predictions in the first place.

[0:18:18 Speaker 3] If there’s like predicting the weather in San Diego,

[0:18:20 Speaker 1] yes, exactly. It’s like predicting the weather. And you know, Tulsa, Uh, you know, but we set ourselves up for it. We can’t resist making these predictions sometimes,

[0:18:28 Speaker 0] and that means you’re

[0:18:28 Speaker 1] gonna be wrong. And politics is Ah, is a social science. And, you know, we we’ve seen so much. There’s so much to be humble about from our perspective in the last six months. But But we do think we have some core understandings, and we’re moving forward with those

[0:18:45 Speaker 0] that’s it for the connectors. First take on polling. I hope you enjoyed the conversation. And if the candidates have not read you up, perhaps we have you feeling that much more excited to be in the midst of a presidential election year. And remember at the University of Texas at Austin, we have our own polling outfit to asserts online for the Texas Politics Project, and I promise you’ll find it as always, to send feedback. You can find me at Gov dot utexas dot . edu