When Numbers Go Bad
A longer-than-average Monday Memo, but worth it.
Are you one who believes the reliability of research is assured when the sample size is adequate and the respondents are properly qualified? If so, "research" will likely lead you to some tragic conclusions if it hasn't done so already.
The problem with most research is that it's done by mathematical types who have little appreciation of the nuances of language. Ask a witness, "How fast were the cars going when they smashed into each other?" and they will name a much higher speed than if they are asked, "How fast were the cars going when they made contact?" (This is not a speculative assertion. The full report can be found in Essentials of Human Memory by Dr. Alan Baddeley.)
What's missing in most survey writers is an understanding of the illogic that we humans call logic.
Neurologist Richard Cytowic was nominated for a Pulitzer in 1982. This is what he had to say in The Man Who Tasted Shapes: "My innate analytic personality had been reinforced by twenty years of training in science and medicine. I reflexively analyzed whatever passed my way and firmly believed that the intellect could conquer everything through reason. 'You need an antidote to your incessant intellectualizing,' Clark had once suggested, 'something to put you in touch with the irrational side of your mind.'… I had never considered that there might be more to the human mind than the rational part that I was familiar with. It had never once occurred to me that a force to balance rationality existed, let alone that it might be a normal part of the human psyche."
When Cytowic began to study this "force to balance rationality" he learned: "…some of our personal knowledge is off limits even to our own inner thoughts! Perhaps this is why humans are so often at odds with themselves, because there is more going on in our minds than we can ever consciously know."
"If a new soft drink came along that you thought tasted better than your current favorite, would you switch to it?"
"Which of these two colas tastes better to you?"
"Thank you for your opinion. You have been very helpful."
But when New Coke was introduced, America hated it. We were outraged, You're messing with our heritage! New Coke wasn't a genius marketing ploy to remind us of how much we loved old Coke. It was a genuine screw-up, fueled by millions in research.
Joey Reiman, a founding partner of the BrightHouse Institute, (one of Coca-Cola's research partners) gave an interview to the New York Times on Oct 26, 2003. "Focus groups are ultimately less about gathering hard data and more about pretending to have concrete justifications for a hugely expensive ad campaign. ‘The sad fact is, people tell you what you want to hear, not what they really think,' Reiman said. ‘Sometimes there's a focus-group bully, a loudmouth who's so insistent about his opinion that it influences everyone else. This is not a science; it's a circus.'" The article went on to say: "Advertising's main tool, of course, has been the focus group, a classic technique of social science. Marketers in the United States spent more than $1 billion last year on focus groups, the results of which guided about $120 billion in advertising. But focus groups are plagued by a basic flaw of human psychology: people often do not know their own minds."
Ask a person to speculate about what they would do in a particular circumstance and they'll tell you what they truly believe they would do. But when the actual circumstance comes upon them, they do something else entirely. My advice: Quit asking people what they think. Begin watching what they do. Ignore their words; study their actions.
Still not convinced that numbers are easily misinterpreted and misunderstood? In a recent Los Angeles Times article Peter Gosselin writes about economists who won the Nobel prize and then made poor personal investment decisions, sometimes even fumbling the Nobel prize money. He then took a look at the investment decisions of the faculty of Harvard University. His conclusion? The financial masterminds don't do any better than the average goober standing in line at the bowling alley.
Remember the days prior to the bursting of the dotcom bubble? Everyone was talking about "eyeballs" under the assumption that web traffic could easily be translated into dollars. "It's just a numbers game." The Internet was ruled by computer programmers and numbers have long been the language of Wall Street. But any time the flaws and foibles and inconsistencies of humanity are removed from the persuasion equation and the chant begins, "Numbers don't lie," engineers, programmers, researchers and investors will align themselves into a magnificent fool's parade. And then, when the bubble bursts because the fundamental assumption was wrong, they blame it on the introduction of "unforeseen forces."
My partners Jeff and Bryan Eisenberg tried to warn the dotcom world, but no one in those days listened. Jeff and Bryan's heretical notion was that online shoppers are human beings and should be treated as such. "Remove the humanity from the data and you're left with nothing but dangerous digits." Data worshippers pooh-poohed the warning. Today the Eisenbrothers are regarded as two of the preeminent consultants in the world of online marketing. In fact, if the sales numbers can be trusted, their new book, Call to Action should make the Wall Street Journal bestseller list this week and maybe even the New York Times as well.
Let's hope that numbers, this time, can be trusted.
Roy H. Williams
PS – If Call to Action makes the bestseller list, it will be the first book to have done so without the benefit of brick-and-mortar distribution. The entire Call to Action marketing effort was accomplished entirely online. But then again, doesn't it make sense that the bookstore-distribution barrier would finally be shattered by the kings of online marketing? We're keeping our fingers crossed.
PPS – I don't really expect to change anyone's mind today regarding the veracity of numbers. To those with an entrenched left-brain perspective, numbers are a religion and my comments will be summarily dismissed. Those who agree with what I wrote will merely be affirmed in a long-held belief. So no one really changes. Hey, I may wander into strange intellectual territory sometimes, but at the end of the day I'm a clear-eyed realist. See you next week. – RHW
Recent Comments