A Smarter Computer to Pick Stock

http://www.nytimes.com/2006/11/24/business/24trading.html?pagewanted=2&_r=1

November 24, 2006
A Smarter Computer to Pick Stock
By CHARLES DUHIGG
Ray Kurzweil, an inventor and new hedge fund manager, is describing the future of stock-picking, and it isn’t human.

“Artificial intelligence is becoming so deeply integrated into our economic ecostructure that some day computers will exceed human intelligence,” Mr. Kurzweil tells a room of investors who oversee enormous pools of capital. “Machines can observe billions of market transactions to see patterns we could never see.”

The listeners, attendees of a conference sponsored earlier this month by the Capital Group Companies, are slightly skeptical. Some have heard that Mr. Kurzweil, 58, who takes more than 150 vitamins and supplements a day, believes people will eventually live forever. Others know he has said that in 2045, man and machine will achieve “singularity,” and humans will hold their breath for hours thanks to nanomachines in our bloodstreams.

But some are aware that a former Microsoft executive and chairman of the Nasdaq stock market, Michael W. Brown, is an investor in Mr. Kurzweil’s new hedge fund, FatKat, and that Bill Gates once described him as “the best person I know at predicting the future of artificial intelligence.”

More important, many of them have seen Mr. Kurzweil’s ideas used by stock speculators. So, they want to learn more about his brave, new world.

“These ideas are the future,” said David Atkinson, a private investor who attended another lecture later that day by Mr. Kurzweil. “I’m not really sure I understand them, but they’re making some folks rich.”

Complicated stock picking methods are nothing new. For decades, Wall Street firms and hedge funds like D. E. Shaw have snapped up math and engineering Ph.D.s and assigned them to find hidden market patterns. When these analysts discover subtle relationships, like similarities in the price movements of Microsoft and I.B.M., investors seek profits by buying one stock and selling the other when their prices diverge, betting historical patterns will eventually push them back into synchronicity.

Today, such methods have achieved a widespread use unimaginable just five years ago. The Internet has put almost every data source within easy reach. New software programs, like the Apama Algorithmic Trading Platform, have made it possible for day traders to build complicated trading algorithms almost as easily as they drag an icon across a digital desktop.

“Five years ago it would have taken $500,000 and 12 people to do what today takes a few computers and co-workers,” said Louis Morgan, managing director of HG Trading, a three-person hedge fund in Wisconsin. “I’m executing 1,500 to 2,000 trades a day and monitoring 1,500 pairs of stocks. My software can automatically execute a trade within 20 milliseconds — five times faster than it would take for my finger to hit the buy button.”

Studies estimate that a third of all stock trades in the United States were driven by automatic algorithms last year, contributing to an explosion in stock market activity. Between 1995 and 2005, the average daily volume of shares traded on the New York Stock Exchange increased to 1.6 billion from 346 million.

But in recent years, as algorithms and traditional quantitative techniques have multiplied, their successes have slowed.

“Now it’s an arms race,” said Andrew Lo, director of the Massachusetts Institute of Technology’s Laboratory for Financial Engineering. “Everyone is building more sophisticated algorithms, and the more competition exists, the smaller the profits.”

So investment firms have increasingly begun exploring mathematics’ furthest edges and turning to people like Mr. Kurzweil, who became an expert in pattern recognition building a reading machine for the blind.

For years, computer scientists had tried to help machines perform mundane tasks like reading printed words or telling faces apart. With algorithms similar to those used by stock pickers, programmers created millions of rules designed to tell an “A” from an “a.” But no machine could read a page of text as well as the average child.

So Mr. Kurzweil and others took a different tack: instead of creating sequential rules to instruct a computer to read, they thought, why not create thousands of random rules and let the computer figure out what works?

The result was nonlinear decision making processes more akin to how a brain operates. So-called “neural networks” and “genetic algorithms” have become common in higher-level computer science. Neural networks permit computers to create new rules and automatically change underlying assumptions by experimenting with thousands of random sequences and processes. Genetic algorithms encourage software to “evolve” by letting different rules compete, and combining the most successful outcomes.

Wall Street has rushed to mimic the techniques. Because arbitrage opportunities disappear so quickly now, neural networks have emerged that can consider thousands of scenarios at once. It is unlikely, for instance, that Microsoft will begin selling ice-cream or I.B.M. will declare bankruptcy, but a nonlinear system can consider such possibilities, and thousands of others, without overtaxing computers that must be ready to react in milliseconds.

“Most software fails in pattern recognition because there aren’t enough sequential rules in the world to teach a computer to discern between two faces, or to find almost imperceptible relationships between stocks,” said Orhan Karaali, a computer scientist and director at Advanced Investment Partners, a $1.7 billion hedge fund. “But a machine that can generate complicated rules a person would never have thought of, and that can learn from past mistakes is a powerful tool.”

Last year, the funds using Mr. Karaali’s model returned in excess of 20 percent by using nonlinear techniques, according to his company. Whereas older methods of stock analysis rely on certain assumptions — for instance, that market volatility always reverts to the mean — Mr. Karaali’s model calculates probabilities and generates assumptions on the fly, and might predict that during a panic, investors will sell Microsoft but, for seemingly irrational reasons, hold onto I.B.M.

“Only an elite group of people are using these ideas, but a lot of people are thinking about them,” said Stacy Williams, director of quantitative strategies at HSBC Global Markets. HSBC is working with Cambridge University in using models based on how viruses spread to forecast foreign currency markets.

“The downside with these systems is their black box-ness,” Mr. Williams said. “Traders have intuitive senses of how the world works. But with these systems you pour in a bunch of numbers, and something comes out the other end, and it’s not always intuitive or clear why the black box latched onto certain data or relationships.”

Such qualms, however, have not stopped Wall Street from scouring university doctoral programs or listening to people like Mr. Kurzweil.

In the pursuit of previously undetectable patterns, hedge funds are racing to quantify things — like newspaper headlines — that were previously immune from number-crunching.

Both Dow Jones Newswires and Reuters have transformed decades of news archives into numerical data for use in designing and testing algorithmic systems. The companies are beginning to structure news so it can be absorbed by quantitative models within milliseconds of release.

Moreover, companies like Progress Software are working with news agencies to create computer programs that instantly translate news — for example, a headline regarding Microsoft’s earnings — into data. M.I.T. is examining, among other things, evaluating companies by seeing how many positive versus negative words are used in a newspaper article.

Software in development could potentially respond automatically to almost anything; changes in weather forecasts on television news, shifting analyst sentiments or what a particular movie critic said about the new blockbuster.

“Right now, everyone basically has access to the same data,” said John Bates, a Progress Software executive. “To get an edge, we want to give investors the ability to immediately turn news into numbers. We want to automate what before required human analysis.”

But as these new techniques proliferate, some worry that promotion is outpacing reality. These techniques may be better for marketing than stock picking.

“Investment firms fall over themselves advertising their latest, most esoteric systems,” said Mr. Lo of M.I.T., who was asked by a $20 billion pension fund to design a neural network. He declined after discovering the investors had no real idea how such networks work.

“There are some pretty substantial misconceptions about what these things can and cannot do,” he said. “As with any black box, if you don’t know why it works, you won’t realize when it’s stopped working. Even a broken watch is right twice a day.”

______________

Thought this might lead to a good thread. Lets discuss.

 

dropped a few names, now tell us something we don't know...

 

Thanks.I'm interested in political bias by the BBC in favour of the Labour Party.I think this sort of technology will help to kill that sort of thing.

 

This article was geared towards non-technical people and geeks who like Star Trek but don't have engineering or technical degrees.

I was surprised the date was Nov 24th, because the information within is very outdated.

 

Just wait till quantum computers come along. Apparently the good old binary is in for a surprise where 1s are also 0s, or something. Too brainy for me !
Perhaps they could wire up what's left of Einstein's brain, plug it in and hey presto. LOL

 

I have often thought about what the future holds in respect to this, my best advice is to watch Terminator. A land goverened by Arnold it makes me shiver, hang on ...........

 

It's not clear whether this response is meant for Av or for the author of the article he posted. Either way, it seems rather pointless.

Av, how about this; inasmuch as fundamental market conditions will continue to determine the direction of these instruments at given times (stocks, commodities, whatever), there will always be room for individuals to trade the trend and compete with the most sophisticated machines.

Yes? No?

 

yes, but will they be or even have they been profitable. I can play golf against tiger woods but I know I will lose.

 

Right, you would lose, but let's say you were a -1 handicap, a very good player. Now, that's not good enough to play on the even the satellite pro tours (slope way harder for pro courses, your -1 is on your home course). However, let's say you could cherry pick the spots where you have to compete with Tiger; for instance, only on the par 3's or maybe only on fairways hit. Maybe you have a world class wedge game or you hit freakishly straight drives, 90% in the fairway. You could never beat Tiger playing every minute against him, but I am not trying to beat the machines playing their game. I can cherry pick the places where I want to get involved, and I pick only situations where I have a very good system.

If Tiger knew that was the game, wouldn't he then just start hitting the punch two-iron off the tee and beat your 90% accuracy? I'm not sure if the analogy is a good one, but I don't think the machines will be able to trade against me personally while trading the overall markets. In that sense, won't a market always be a market?

The only way this wouldn't work is if the machines were able to figure out how to collude.

 

The limiting case of machines colluding is smoothing.

I really like how the focus of big money is to go more and more macro to take smaller and smaller profits on many many small imbalances.

LO is cool and Sloan must loved him and his dialogue with the troops. Learning is important and I think it is fun to observe the smartest minds coming down the pipeline.

The inventor's videos on discovery are terrific and it is even better to anticipate the duel between his support of EMH and Lo's CMH. All reported out as said a while back.

So now we are in ET and looking at how the diferent market players will be affected: the pro's here and the amateurs here.

All of these guys in ET who are getting the job done would have been laughing at both Lo and the inventor. It is extremely well known that we need to have the big money doing their thing.

The basic reason is to be assured in a rock solid way that someone out there is going to be running a sales campaign to POOL more and more capital. NEWER and FASTER are watchwords of the big money to add to their pools.

I dig that they will be handling the 1:15EST daily "reconciliation" in bigger and faster ways. Guys I admire most who are pros are people that DO NOT have to deal with big money and are in offices with small teams that creme the market ALL the time.

Absolutely none of them are tuned to the article's palaver by a reporter who goes to meetings and writes articles. I should sit down and write up my notes on a session of four of the hottest small group pro guys I have ever sat with for two hours or so. They do in a week what is posted as an annual in the article.
I had some personal time with each just as a matter of passing time since we were in the same place.

Amateurs could care less about what big money and big computers are doing to solve the classic big money problems. Small groups of amateurs sitting in an office the size of the killer pros offices mentioned above is my choice of where to be. What is the contrasting draw of this space? It is activity that is the same as the killer pro's space.

What goes on during the day to make real money with capital is where the real action is.

Trading individual capital by either killer pro's or hot amateurs is not very different and it is so very different than the eeking done by the big money and new fast computers.

The hottest transaction cost group serves big money and it is sure not in NYC or London. But is just an eeker thing as well.

This thread is about the article and the names in the article. So be it; I said my piece.

Amateurs and small pro's need to have a forum on really getting the job done. Not here in this thread but maybe somewhere.

Using more and more to get less and less is the reporter's take on the future. Neat. It will smooth things out. The name of the real game is small people draining the huge infinite pool as fast as possible.

Killer small pro's and hot amateurs do that. And it is ALL out of the box thinking and IMMEDIATELY available!!!