Seth's Rant for November 20th, 2002

Survey Says...


Someone just sent me a link to an article about a survey decrying the poor geographic knowledge of American youths, and asked "So how can we maintain a functioning society in light of such massive ignorance?"

This really set off one of my pet peeves:

A better question is, how can we have a functioning society in light of such massive incompetence and rumor mongering on the part of the media, our supposed watch-dogs of democracy?

Does this survey actually have ANY statistical significance?

According to the article, they spoke "face-to-face" with "at least 300" people scattered across nine countries. So the geographic knowledge of tens of millions of Americans is being judged by about 30 self-selected people who had nothing better to do than answer geography questions. What did this "survey" consist of? One guy in each country hounding teenage mall-rats?

While I have no doubt that the level of geographic understanding among American youth is unacceptably low, I find it far more disturbing that the level of statistical understanding among American "journalists" is non-existent.

It severely ticks me off when some talking head natters on about how "74.3% of Americans believe..." or "32.9% say". These are almost always based on poorly worded questions, asked of a few hundred volunteers, who were only permitted to choose from a few often ambiguous answers. People who have jobs, kids, or just plain don't answer the phone for telemarketers, don't count.

I especially love it around election time when multiple news agencies will publish survey results, each with a supposed error margin of +/-3% and results spanning a range of 10% to 20%.

I've participated in maybe a dozen opinion surveys over the years, almost all by telephone. In each case I did so only becaue a) I wanted to feel that my opinion counted, and b) I had nothing more important to do at the time. In each case, many of the questions were so badly worded that I had to ask the surveyor what was meant. (I especially love the, "Do you disagree with Mr. X's position against the referendum opposing the proposal to repeal the legislation banning the refusal do Y?" multi-negative opinion questions. Try parsing that on the phone.) If the surveyor answers at all, it's clearly nothing more than their personal guess. When the results of the survey are reported in the news, the results cited are almost always reworded in a way that changes the meaning (or at least changes the way I would have answered).

I now refuse to answer verbal surveys (hell, I refuse to answer my phone).

It's an old axiom that statistics can be manipulated to say anything the statistician wants. What really ticks me off is that the media has become so lazy and ignorant that statistics have become nothing more than another form of press release for whomever has an issue to hammer.



This rant solely reflects the opinion of the author, probably while he was half asleep, drunk, or otherwise incapacitated. It does not necessarily reflect the opinion of his employer, his friends, or possibly the author in a more conscious state. Hate mail will be prosecuted. Constructive criticism may be posted or ignored. Have a nice day.


Seth B. Noble - Rant - sbnoblewheelies.net - November 20, 2002