Bad Analytics Trends- Part 1: Correlation does not Equal Causation
Aug 3rd, 2007  by Pedro Sostre

Gregarious FeedFlare

With web analytics getting more mainstream recognition and companies looking to undertrained employees to report on analytics, I’ve seen several disturbing trends. Hence, I’m going to start a series on Bad Analytics Trends.

The first of these trends in the idea that correlation does not equal causation. If that sounds like Greek to you, let’s define what we’re talking about here.

cau·sa·tion

• the action of causing or producing.
• the relation of cause to effect; causality.
• anything that produces an effect; cause.

cor·re·la·tion

• mutual relation of two or more things, parts, etc.
• Statistics: the degree to which two or more attributes or measurements on the same group of elements show a tendency to vary together.

The idea here is that just because two metrics seem to go up and down together, doesn’t mean that those metrics are directly affecting each other. Let me give a practical example.

You are running a lead generation site for a service business. In looking through the data you find that users who visit your testimonials page are 50% more likely to submit a Request for Proposal. From this you could deduce that

1. Your testimonials page makes people more likely to convert
2. Testimonials alone make people more likely to convert
3. None of the above

While it’s tempting to believe points 1 or 2, you really shouldn’t. Maybe they are true, maybe they aren’t. Here’s why. Based on the single data point, there’s no way to tell whether seeing the testimonials page caused people to submit the RFP or whether people who had already decided to submit an RFP are more likely to make a quick stop at the testimonials page during their visit.

The real problem is when companies start making decisions based on these incorrect assumptions. Decisions like, “Let’s put a big ‘Ol link to the testimonials page on every page!” or “Let’s put testimonials all over the site so people see them everywhere!” are not founded. They may end up helping (coincidentally), but don’t pretend they are genuine data-driven decisions.

Anyone have any examples from personal experience to share?