Gravis Marketing: A Deeper Analysis
Editor’s note: This is another article from dawolf, who had previously analyzed some polling data from Gravis Marketing. As always, we encourage our readers to contribute articles to Logarchism.
On the October 14, I examined a poll of Gravis Marketing and found some dubious numbers in the crosstabs. Doug Kaplan, CEO of Gravis Marketing, has stated that demographic adjustments account for these anomalous results, and that demographic adjustments are being continuously adjusted and may vary between weeks.
Unfortunately, Kaplan has declined to provide any raw numbers or weightings so it is not possible to verify his claim. But we can have a look at what Gravis polls of Florida actually say, and if there is evidence of demographic weighting. It should be noted that the poll examined does not contain the words “Demographic” or “Weighting” or indeed any mention of any such adjustment. However, the most recent poll of Florida (posted since the earlier article) does mention this.
The basic concept of demographic weighting is that a typical sample might have too many respondents in one category, and not enough in another category. For example, maybe you estimate that Hispanics will form ten percent of your final voters: but in a poll of 1,000 people, only 50 Hispanics were polled. One way of adjusting for this is to weight the Hispanic respondents to ten percent in your final results.
Gallup for instance states that:
After Gallup collects and processes survey data, each respondent is assigned a weight so that the demographic characteristics of the total weighted sample of respondents match the latest estimates of the demographic characteristics of the adult population available from the U.S. Census Bureau. Gallup weights data to census estimates for gender, race, age, educational attainment, and region.
Essentially, what one tries to do is reduce demographic variability between polls, and make sure that low-response groups don’t get disregarded.
Demographic weighting can take many forms. But the only ones that seem possible given Gravis Marketing’s survey questions are:
- Religious Affiliation
- Age Group
Of these, Kaplan stated that only Race, Age Group and Gender are considered.
For this analysis, I’ve compared several different polls conducted by Gravis Marketing. I’ve summarized the results for clarity.
The poll that I examined in the last article was conducted on September 29–30, 914 likely voters, examined in the previous article, shorthand 1001). This will be compared to the last Gravis poll of Florida before September 29, and the two polls of Florida that have taken place since (although they do not have as detailed crosstabs).
If demographic weighting is used, we’ll expect to see similar weightings in each poll, so let’s see what we find.
Typically respondents to polls are generally:
- More female
Demographic adjustments are therefore generally likely to:
- Reduce the weight of older respondents, and increase the weight of younger respondents
- Reduce the weight of White respondents, increasing the weights of other groups
- Reduce the weight of female respondents, increasing the weight of other groups
There is a lot of variation between each poll and certainly no evidence of demographic weighting. Lets focus down on two particular results, racial variation in the Other/Unsure and Asian groups.
Here’s the breakdown just for those two groups. It should be pointed out that the census result for Florida has Asians at 2.4% and Other at 3.6%.
Since any demographic weighting should tend to increase the size of Asian and Other responses, any demographic weighting should increase our number from 0.109%.
Crosstabs from here come from the September 29–30 poll (the poll that was examined in the previous article). Circled are two numbers.
The first number circled shows that only Asian women were polled. There are several very unusual results of this type (statistically, very unlikely) scattered throughout this poll, but I don’t intend to focus on them here (I address one, about Muslim respondents, at the end of this article in brief). The second number shows that Other/Unsure (Male) has a number of 0.05%. The lowest number we can get is 0.109% without any demographic weighting being applied so we’re looking for a big change.
How about Age weighting?
No respondents age 65 and up for the Asian subgroup, and demographic weighting should, if anything, make the lower age groups bigger.
Here are examples of anomalous results for both Asian and Other/Unsure
For the Asian group, there should be
- No Racial weighting making the percentage reduce (any effect will tend to be in the opposite direction)
- No Age weighting making the percentage reduce (any effect will tend to be in the opposite direction)
- Possible Gender weighting making the percentage reduce.
Since we have a value of 0.05%, this implies a minimum of 75 percent women in the sample, and the other weightings having no effect.
But if that is true, then what’s going on here? Lets go back a couple of steps
75 percent female means 25 percent male; any adjustment you make downwards for too many women has to be balanced by an upward adjustment for too few men. We’re no longer looking at a minimum of 0.109% for Other/Unsure (Male), but instead over double that: 0.22%+. We need an absolutely huge demographic weighting to make this work. Specifically, we’d need about four times too many people in an age category; the only possible option is that 90 percent of the respondents would need to have been age 65+.
There are a few major options. Take your pick:
- The poll did not take place;
- The original sample was both 75 percent female, and 90 percent age 65+, or some combination which includes massive demographic penalties to racial groups without likely cause;
- After the poll took place, errors were introduced which corrupted the results;
- This analysis is flawed (I’d love to have a professional statistician look at this).
Just a last thought. In a world where Muslims intend to vote 68–7 for Obama over Romney, how likely is this result?
Take your pick, 1, 2, 3 or 4.
- A Gravis Fraud
- Matt Towery: Flaws in polling are hiding Romney’s surge
- Leftist bloggers cyber bully pollster for calling Romney momentum early
- Quick Note on the Peril of Crosstabs
- A Gravis Response