CMOs say it all the time. “The great thing about digital marketing is that you have hard data to know what’s working. Data doesn’t lie.”
Nonsense. Data does lie. Data lies all the time. Like a rug.
Spanish Inquisition Analytics means torturing the data until it says what you want it to say. Reminds me of my youth when I was learning to drive a stick-shift in hilly New England – grind it till it fits.
I’m not talking about spin. If you completed 100 projects through April, then added 5 more in May and 10 more in June, you’re more likely to say:
“we saw a 100% incremental gain in month-over-month project completions in June” (June minus May, divided by May)
than
“we increased the number of completed projects by 9.52% in June.” (10 in June divided by 105 total at the end of May)
Both statements are factual. The first makes the project manager look better on a PowerPoint slide, and that’s okay – it’s just spin. Marketers frikkin’ love spin – having invented it and all.
Spanish Inquisition analytics aren’t mere spin -they’re deliberately crafted to mislead. While true analytics prove or disprove a hypothesis, Spanish Inquisition analytics assumes a positive position, then changes or eliminates data that doesn’t support that proof. Here’s an example uncovered in July 2010 by MediaMatters.org:
Fox News showed this chart to support a story that job loss was still soaring. But since the data didn’t support their premise, they tortured it by cherry-picking the quarterly results that fit their premise, and eliminating those that did not. Notice the timing of the data points. The first data point is December 2007. The second is 9 months later, the third just 6 months later, and the fourth a whopping 15 months later. The data they used is technically correct – but it’s showing cumulative instead of incremental data on an improperly paced timeline. The story? US jobs are still being dropped faster than 8am Foundations of Western Civilization. The reality? Here’s how the data actually looked – a decidedly different story.
It’s like picking out the cheese, tomatoes and croutons out of a salad and passing it off as pizza.
Another trick is to skew the research itself to support your aims. Target a marketing campaign solely to customers who have bought every iteration of your software on their release dates, and you’re likely to report a 95% take rate. But you better stay away from reporting the ROI -because you just paid to get people to buy who would have bought anyway. Test a targeted web product only amongst your allies and immediate circle, and you’ll get data, alright. But use that data to project satisfaction with your site, and you’ll be blindsided at launch when your target audiences don’t find it true to their vibe.
You want to look good? Get some Botox. You want to get some real insight? Let your data people tell the real story. Nobody expects the Spanish Inquisition.
POSTSCRIPT – February 24, 2011
Hey kids, here’s a brand new way to torture analytics from – you guessed it – your friends at the Fox News Academy of Creative Analytics. If Gallup Poll results don’t support your argument, just invert them! 61% of Americans oppose taking away collective bargaining rights? Presto! 61% of Americans now favor it! Yup, this actually made it on the air. Here’s a link to the story.
Full disclosure: I lean left. Yet, my beef with Fox in these two cases are as a data professional rather than a liberal. Facts aren’t inherently truths. But facts changed to support predetermined conclusions are inherently lies – regardless of ideology.
