Indexers use the efficient market hypothesis to help explain generally why stock market averages are so hard to beat with consistency. But efficient market theory does a poor job of explaining bouts of speculation or panic - irrational investors.
One intriguing theoretical approach that incorporates irrationality well is the evolutionary biology model. It requires no particular level of rationality, efficiency or other assumptions that have plagued the classical efficient market theory. A wealth of statistical techniques support it, and it incorporates computerized analysis easily.
"In the biological world species that do well are rewarded by proliferating and those who don't die off," says J. Doyne Farmer, McKinsey Research Professor at the Santa Fe Institute, a complex systems think-tank. His seminal paper 'Frontiers of Finance: Evolution and Efficient Markets' sums up this approach.
As a twenty-four year old physics graduate student he was a member of the "Dynamical Systems Collective" of UC Santa Cruz, a group of pioneer nonlinear dynamics scholars featured in the bestseller "Chaos" by James Gleick. He is also a successful institutional trader of indexes for Wall St. through his firm Prediction Company.
"A similarity in the investment world involves managers who do well and increase their funds and managers who don't and go out of business. It's quite clear that there is some strong selection pressure in financial markets. Good investors are selected and poor investors are weeded out. That is very different than saying things are optimal. In biology there is little sense of perfection."
The commonsense notion of relative efficiency works fine in evolutionary competition. Species need not perform at optimal levels to be successful. Humans, for instance, have chronically bad backs but they still compete well as a species.
"Performance in markets is very much relative to players in the market," he said. "The system is co-evolving. It is dynamic. Efficiency doesn't happen instantaneously."
"It is clear from classic arguments that the market can't be fully efficient. If it were, everybody would go home. There has to be some kind of stasis where the market is pretty efficient but not perfectly efficient. What does efficiency mean? What are reasonable profits for taking risk that might involve skill? Once you say the market is not perfectly efficient, you need a way to talk about what that means."
What does this mean for the average investor? Inefficiencies of boom and bust cycles are likely to exist, but it is hard to say whether a successful investor is methodically exploiting them or just lucky.
"My gut feeling is that there are substantial inefficiencies," he said. "Irrationality makes a difference. It s not just that people don't have the information but that it's hard to make reasonable inferences. I myself am often quite confused."
"Often managers who do well are very lucky. On average skill is rewarded and those without skill die out. Track records are indicative but it's important to know how to read the fine print. Someone making a lot of bets probably has a more meaningful track record than someone adjusting their portfolio once a year."
And, yes, that is what he does on the side with Prediction Company, a technical analysis firm he founded 9 years ago. The company makes thousands of small trades each year for money managers Warburg Dillon Read.