David Johnson and a troubled Strategic Vision promise crosstabs with every poll

Mark Twain divided falsehoods into three categories: Lies, damned lies, and statistics. Had he lived to see them, Twain might have added a fourth: Public opinion polls.

Atlanta has two political polling firms that routinely tell us who is ahead this race or that, whether for mayor of Atlanta or governor.

Front-running candidates often use numbers generated by these companies to raise campaign cash. Journalists sometimes use the same statistics to help determine which candidates are worth their time.

One of these survey companies, Republican-oriented Strategic Visions LLC, has just finished a very bad week.

The firm was censured by a national organization of pollsters. And Nate Silver, a prominent political statistician, publicly suggested, in a series of posts on his much-read blog, fivethirtyeight.com, that Strategic Vision might be making up its results.

A clearly exhausted David Johnson, CEO of seven-year-old Strategic Vision, on Friday called the intimations libelous. “We categorically deny it. And yes, we are going to be taking legal action,” he said.

Johnson’s troubles are rooted in the New Hampshire presidential primary of 2008, when Democrat Hillary Clinton surprised one and all with a victory. Nearly every pollster had predicted a win for Barack Obama, the winner of the Iowa caucuses five days earlier.

The statistical debacle bothered the 2000-member American Association for Public Opinion Research, which is worried that more and more people in this math-challenged society are coming to view polls with Twain-like suspicion.

The 62-year-old organization – George Gallup was an early president – embarked on a year-long investigation, requesting internal methodology from 21 polling outfits that had operated in New Hampshire and other states with primaries.

In a noticed posted on its Web site last Wednesday, AAPOR announced that Strategic Vision was the only firm that “repeatedly refused to release essential facts.”

The AAPOR said it was not passing judgment on the quality of Strategic Vision’s work. “We’re saying we can’t know anything about quality if we don’t know what you did,” said Peter Miller, president of the organization.

Johnson, 42, said he isn’t a member of AAPOR and shouldn’t be subject to its policing. Miller said his organization has an obligation to point out murkiness in the polling industry wherever it is found.

The economics of political polling have changed a great deal in the last several years. Newspapers and other media outlets were once a major source of funding. This is no longer the case.

Some polling firms, including those in Atlanta – Strategic Vision and InsiderAdvantage – have adapted by giving polls away, or charging only nominal fees. They consider the surveys to be loss-leaders, and use them to attract paying clients.

In a world where you get what you pay for, quality has become a concern.

One measure of quality is the demographic breakdown behind every poll. We call them “crosstabs.” Those with the proper math skills can use them to help gauge a poll’s validity.

Whenever it has released a statewide poll in Georgia, Strategic Vision has kept the crosstabs to itself, as proprietary information. “Because we were trying to sell them,” Johnson said. Other firms do the same, he noted.

But not all. InsiderAdvantage CEO Matt Towery says he now releases background data with every survey – or to any news organization that asks.

“You have to. If you can’t show crosstabs, you can’t prove that you’ve got a poll,” said Towery, a former state lawmaker.

InsiderAdvantage has partnered with WSB-TV to poll the Atlanta mayor’s race. The background data provided by Towery has led some to criticize the surveys as distorted. The polls have given too much weight to young voters, critics say.

But even if it increases the second-guessing, the added transparency boosts public confidence in the results, Towery said.

Events of the last week have caused Strategic Vision to come to the same conclusion about its future polling.

“We’re going to release all the crosstabs, and put an end to this right now,” Johnson said. “That will squelch anybody from saying anything.”

For instant updates, follow me on Twitter.

15 comments Add your comment

Jeremy Jones

September 27th, 2009
4:48 pm

This group did a poll that showed me at 25% in the 9th District, and while I appreciate such high numbers, I know it can’t be true…I also know that Mr. Hawkins is not in the lead. He has no support across the district.

Base

September 27th, 2009
5:27 pm

Polls can’t be trusted everybody should know that.Its only a guess and not worth the money payed.

Squash

September 27th, 2009
5:31 pm

Disraeli, not Twain, is attributed to the ‘lies, damned lies, and statistics’ quote.
:-)
squash

roger

September 27th, 2009
5:57 pm

i would guess that the overall number of surveys would have to be more than 1000 to be accurate; strategic vision eventually had Obama winning in 2008 but it was late in the game

jconservative

September 27th, 2009
7:00 pm

In other words, we will no longer do what we are accused of doing that we did not do.

Chris Mathews is covered in his own spittle

September 27th, 2009
8:39 pm

I am eagerly awaiting the story in the NYTIMES/CBS poll that regularly poll twice as many number of democrat voters as republican.

Matthew

September 27th, 2009
9:38 pm

What a milktoast article!! You call this journalism? Galloway didn’tprobe Johnson on ANY of his answers. Also, he failed to mention that Johnson promised cross tabs 3 times before in his post today. I think he may be too dumb to actually understand the criticism/issue at hand here. He takes everything Johnson says at face value, which is why the situation arose in the first place!

Bert

September 28th, 2009
7:06 am

Matt Towery and his WSB-TV/Insider Advantage poll are lies, lies and damned lies. Even though Towery makes his crosstabs available, he pulls them directly out of his butt.

WSB-TV Channel 2, through its alliance with Towery, is proving itself to be more and more like the right wing nuts at Fox news.

It’s time to begin thinking about organizing a boycott of Channel 2.

Joey

September 28th, 2009
8:50 am

Political Polls have two primary purposes.
First they are used by left-wing and right-wing media in their efforts to influence public opinion (CNN, FOX, USAToday, etc.)
Second they are used by politicians to determine what, if any, position they should take on an issue.
It is not just the choosing whom to poll (cross-tabs), it is also the way questions are worded and the answer options provided.

GoOx

September 28th, 2009
8:56 am

Dave Johnson is an awesome pollster. He has my man Ox in the lead.

dmac

September 29th, 2009
7:42 am

Enter your poll numbers here.

Thank you Mr. Galloway for posting this and linking to Mr. Silver’s site.

Great comment jconservative. Although, I think you are a little off.

We will start doing what we should have been doing but don’t need to do because we never did anything wrong even though we refuse to prove otherwise.

xoff

September 30th, 2009
1:46 pm

What would clear the air, of course,would be to release the cross tabs from past polls — if any really exist.

JohnJay60

October 1st, 2009
6:25 pm

Nate from fivethirtyeight.com showed in the 2008 elections that polls, IN AGGREGATE, were highly predictive of the outcomes. What it also showed was that certain pollsters consistently rate one party higher than another. This consistency of biase (think Rasmussen) is mathematically provable.

The StrategicVision case is a separate instance, where Mr. Silver lays out a clear case where the results obtained by SV are so improbable as to be nearly impossible to imagine taking place in a true poll.

I would ask readers of these comments to distinguish between “precision” (as in Rasmussen being consistently higher on Republican candidates than others) and “accuracy” (as in Rasmussen being inaccurate, though consistent) and add a third category “fraudulent” (as in Strategic Vision releasing results that require amazing faith in highly improbable results).

[...] And the firm, though it didn’t respond to Wall Street Journal requests for comment, has defended its work in comments to other publications. “I stand by our work,” Mr. Johnson told [...]

[...] “for extreme misconduct; substantial likelihood that polls were fabricated.” See the background here. Dave Johnson, the CEO of Strategic Vision, has denied wrongdoing, but has severely cut back his [...]