In late October, as the stock market was floundering, brash emails started to appear in my inbox.
With the subject line “Stock Exchange predictions for tomorrow and next trading week,” the messages said they contained forecasts “generated by 7,445 adaptive machine intelligence models…that predict stock movements one day and one week in advance” at an average accuracy of 72.45%.
Human intelligence usually fails to predict where stocks are headed in the short term, so I decided to see whether artificial intelligence is better at it. I used the most basic guidelines I know, from the chapter “How to Talk Back to a Statistic” in Darrell Huff’s classic 1954 book, “How to Lie with Statistics.”
To help determine whether evidence is valid, Mr. Huff suggested asking five questions. Following along as I apply his method here may help you size up other investment approaches elsewhere.
Question One: Who says so?
Peter Simmons, founder of Self Aware Apps LLC of Aurora, Colo., and a 20-plus-year veteran of the software industry, operates AIEquityPredict.com, the website that has been sending me the emails I’ve been getting.
“Everyone gets the same email,” says Mr. Simmons, 60 years old. “I’m not monetizing this in any way.” His firm isn’t a brokerage or financial adviser and doesn’t have positions in any of the stocks it is analyzing, he says. Nor is he selling subscriptions; the emails are free.
Mr. Simmons says he is running the service largely as “a social experiment,” to see whether artificial intelligence can excel at picking stocks. He has no professional investing experience.
Question Two: How does he know?
AI Equity Predict uses open-source artificial-intelligence software —freely available programs that can train computers to identify predictive patterns in data—to try forecasting short-term stock returns, says Mr. Simmons.
His software draws entirely on public data, including short sales (bets against a rising stock price), daily trading volume, changes in volume, and new high or low prices. The software learns to rely more or less on different measures for each stock as their predictive power waxes and wanes, Mr. Simmons says.
Mr. Simmons says AI Equity Predict’s claim that its “average model accuracy is 72.45%” measures the five-day rolling average of its predictions for the 2,345 individual stocks it has modeled—not for any stock indexes.
Question Three: What’s missing?
If stocks move in the same direction as AI Equity Predict’s forecast, that counts as an accurate prediction. If stocks go up 0.01% after an “up” forecast, that call is correct in theory—but, in practice, you can’t make any money on it after trading costs and taxes.
I tracked AI Equity Predict’s daily forecasts for the S&P since its emails started arriving. Its “next day” predictions for Oct. 23 through Oct. 31—a total of seven trading days—were “down” every time. Yet, over the same period, every one of its “next week” predictions—covering the five successive trading days—was “up.”
“That’s a good question,” says Mr. Simmons when I ask him how that could be. “That’s something going wrong on the back end. That’s definitely challenged.”
Question Four: Did somebody change the subject?
AI Equity Predict says that the “S&P” is among the groups of stocks it forecasts. That turns out to be not the S&P 500 index, but only 30 of its stocks.
The largest is
at 1.6% of the total market value of the S&P 500. All told, the 30 stocks add up to a mere 6.1% of the total market value of the S&P 500.
So AI Equity Predict isn’t forecasting whether the S&P 500, dominated by such familiar giants as
will go up or down. It’s predicting whether a few scattered and mostly obscure stocks, accounting for less than one-sixteenth of the total index, will go up or down.
“Those are the only ones that the models can predict for so far,” says Mr. Simmons. Thus, his estimates of forecasting accuracy exclude all the stocks his software isn’t ready to forecast.
Question Five: Does it make sense?
Perhaps someone with no investing background, using publicly available data and free software, could discover the Holy Grail of predicting tomorrow’s stock returns today. Stranger things have happened, but that doesn’t make it likely.
After answering my questions, Mr. Simmons says he will stop distributing forecasts for the S&P 500 (but not other markets) until he has models for more of its stocks. “It could be right,” he says, “but I agree the sample’s not big enough.”
He adds, “As you probably perceived, I’m not a financial guy at all.”
—Elisa Cho contributed to this column.
Write to Jason Zweig at firstname.lastname@example.org