The Phantom Pattern Problem: The Mirage Of Big Data (pdf)

Description

Patterns in data are often used as evidence, but how can you tell if that evidence is worth believing? We are hard-wired to notice patterns and to think that the patterns we notice are meaningful. Streaks, clusters, and correlations are the norm, not the exception. Our challenge is to overcome our inherited inclination to think that all patterns are significant, as in this age of Big Data patterns are inevitable and usually coincidental.Through countless examples, The Phantom Pattern Problem is an engaging read that helps us avoid being duped by data, tricked into worthless investing strategies, or scared out of getting vaccinations.Pattern-recognition prowess served our ancestors well, but today we are confronted by a deluge of data that is far more abstract, complicated, and difficult to interpret. The number of possible patterns that can be identified relative to the number that are genuinely useful has grown exponentially – which means that the chances that a discovered pattern is useful is rapidly approaching zero.