Editor’s note: This is another arti­cle from dawolf, who had pre­vi­ously ana­lyzed some polling data from Gravis Mar­ket­ing. As always, we encour­age our read­ers to con­tribute arti­cles to Log­a­rchism.

On the Octo­ber 14, I exam­ined a poll of Gravis Mar­ket­ing and found some dubi­ous num­bers in the crosstabs. Doug Kaplan, CEO of Gravis Mar­ket­ing, has stated that demo­graphic adjust­ments account for these anom­alous results, and that demo­graphic adjust­ments are being con­tin­u­ously adjusted and may vary between weeks.

Unfor­tu­nately, Kaplan has declined to pro­vide any raw num­bers or weight­ings so it is not pos­si­ble to ver­ify his claim. But we can have a look at what Gravis polls of Florida actu­ally say, and if there is evi­dence of demo­graphic weight­ing. It should be noted that the poll exam­ined does not con­tain the words “Demo­graphic” or “Weight­ing” or indeed any men­tion of any such adjust­ment. How­ever, the most recent poll of Florida (posted since the ear­lier arti­cle) does men­tion this.

The basic con­cept of demo­graphic weight­ing is that a typ­i­cal sam­ple might have too many respon­dents in one cat­e­gory, and not enough in another cat­e­gory. For exam­ple, maybe you esti­mate that His­pan­ics will form ten per­cent of your final vot­ers: but in a poll of 1,000 peo­ple, only 50 His­pan­ics were polled. One way of adjust­ing for this is to weight the His­panic respon­dents to ten per­cent in your final results. 

Gallup for instance states that:

After Gallup col­lects and processes sur­vey data, each respon­dent is assigned a weight so that the demo­graphic char­ac­ter­is­tics of the total weighted sam­ple of respon­dents match the lat­est esti­mates of the demo­graphic char­ac­ter­is­tics of the adult pop­u­la­tion avail­able from the U.S. Cen­sus Bureau. Gallup weights data to cen­sus esti­mates for gen­der, race, age, edu­ca­tional attain­ment, and region.

Essen­tially, what one tries to do is reduce demo­graphic vari­abil­ity between polls, and make sure that low-​​response groups don’t get disregarded.

Demo­graphic weight­ing can take many forms. But the only ones that seem pos­si­ble given Gravis Marketing’s sur­vey ques­tions are:

  • Party
  • Race
  • Reli­gious Affiliation
  • Age Group
  • Gen­der

Of these, Kaplan stated that only Race, Age Group and Gen­der are considered.

For this analy­sis, I’ve com­pared sev­eral dif­fer­ent polls con­ducted by Gravis Mar­ket­ing. I’ve sum­ma­rized the results for clarity.

The poll that I exam­ined in the last arti­cle was con­ducted on Sep­tem­ber 29–30, 914 likely vot­ers, exam­ined in the pre­vi­ous arti­cle, short­hand 1001). This will be com­pared to the last Gravis poll of Florida before Sep­tem­ber 29, and the two polls of Florida that have taken place since (although they do not have as detailed crosstabs).

If demo­graphic weight­ing is used, we’ll expect to see sim­i­lar weight­ings in each poll, so let’s see what we find.

 

 

 

Typ­i­cally respon­dents to polls are generally:

  • Older
  • Whiter
  • More female

Demo­graphic adjust­ments are there­fore gen­er­ally likely to:

  • Reduce the weight of older respon­dents, and increase the weight of younger respondents
  • Reduce the weight of White respon­dents, increas­ing the weights of other groups
  • Reduce the weight of female respon­dents, increas­ing the weight of other groups

There is a lot of vari­a­tion between each poll and cer­tainly no evi­dence of demo­graphic weight­ing. Lets focus down on two par­tic­u­lar results, racial vari­a­tion in the Other/​Unsure and Asian groups.

Here’s the break­down just for those two groups. It should be pointed out that the cen­sus result for Florida has Asians at 2.4% and Other at 3.6%.

Race

Poll Date

Sep­tem­ber 15th–16th

Sep­tem­ber 29th–30th

Octo­ber 13th–14th

Octo­ber 24th
Asian

3.3

1.9

1

1
Other/​Unsure

4.1

0.5

4

3

Since any demo­graphic weight­ing should tend to increase the size of Asian and Other responses, any demo­graphic weight­ing should increase our num­ber from 0.109%.

Crosstabs from here come from the Sep­tem­ber 29–30 poll (the poll that was exam­ined in the pre­vi­ous arti­cle). Cir­cled are two numbers.

The first num­ber cir­cled shows that only Asian women were polled. There are sev­eral very unusual results of this type (sta­tis­ti­cally, very unlikely) scat­tered through­out this poll, but I don’t intend to focus on them here (I address one, about Mus­lim respon­dents, at the end of this arti­cle in brief). The sec­ond num­ber shows that Other/​Unsure (Male) has a num­ber of 0.05%. The low­est num­ber we can get is 0.109% with­out any demo­graphic weight­ing being applied so we’re look­ing for a big change.

How about Age weighting?

No respon­dents age 65 and up for the Asian sub­group, and demo­graphic weight­ing should, if any­thing, make the lower age groups bigger.

Here are exam­ples of anom­alous results for both Asian and Other/​Unsure

For the Asian group, there should be

  • No Racial weight­ing mak­ing the per­cent­age reduce (any effect will tend to be in the oppo­site direction)
  • No Age weight­ing mak­ing the per­cent­age reduce (any effect will tend to be in the oppo­site direction)
  • Pos­si­ble Gen­der weight­ing mak­ing the per­cent­age reduce.

Since we have a value of 0.05%, this implies a min­i­mum of 75 per­cent women in the sam­ple, and the other weight­ings hav­ing no effect.

But if that is true, then what’s going on here? Lets go back a cou­ple of steps

75 per­cent female means 25 per­cent male; any adjust­ment you make down­wards for too many women has to be bal­anced by an upward adjust­ment for too few men. We’re no longer look­ing at a min­i­mum of 0.109% for Other/​Unsure (Male), but instead over dou­ble that: 0.22%+. We need an absolutely huge demo­graphic weight­ing to make this work. Specif­i­cally, we’d need about four times too many peo­ple in an age cat­e­gory; the only pos­si­ble option is that 90 per­cent of the respon­dents would need to have been age 65+.

There are a few major options. Take your pick:

  1. The poll did not take place;
  2. The orig­i­nal sam­ple was both 75 per­cent female, and 90 per­cent age 65+, or some com­bi­na­tion which includes mas­sive demo­graphic penal­ties to racial groups with­out likely cause;
  3. After the poll took place, errors were intro­duced which cor­rupted the results;
  4. This analy­sis is flawed (I’d love to have a pro­fes­sional sta­tis­ti­cian look at this).

Just a last thought. In a world where Mus­lims intend to vote 68–7 for Obama over Rom­ney, how likely is this result?

Take your pick, 1, 2, 3 or 4.

Related arti­cles