Neural Network Prediction: a Narrative Explanation

maxdama · Jul 26, 2008

I wrote the following story to help beginners understand neural networks. I've noticed on ET that many people have a vague idea that NNs are very powerful but not a very good understanding of how they work. This story is about how the backpropogation training technique can be used to create a NN that makes accurate predictions. I will probably eventually write a second part to the story to illustrate a different cutting edge NN training system which has shown even better results than BP.

The Parable of Nyeralneht: Part One

BakhâprÃ¤p

The illustrious BakhâprÃ¤p is famous, even in legend today, as the wealthiest merchant in ancient Nyeralneht. He was known for having invented a system which could predict the number of shekels spice would be worth in the following week. When he knew the price would soon double, he stocked up and then sold once the price had risen.

His journal was only recently discovered, offered at an auction by an old hag whoâd found it in a jar. With the wonderful but short time I have been able to get my hands on the journal (those archaeologists are annoyingly protective) I have finally been able to piece together his trading technique.

As was previously well documented, he had a network of informants who reported to him the sentiments of other spice merchants, the changing tastes of the populace, the predictions of disruptive weather, the average daily market price of spice, innovation in the agricultural and nautical technologies related to the production and transportation of spice, and even the volume of spice that was being traded at the Grand Bazaar of Nyeralneht on a daily basis. This was quite a network to manage, but many merchants besides BakhâprÃ¤p had networks of a similar sizes and complexities. BakhâprÃ¤pâs ability to filter and manage these various inputs was his real secret.

-Â§-

From his disintegrating journal I have learned that he employed two layers of advisors. There were lowly scouts and spies, some were little children, some even pickpockets, some corrupt government officials, and all the way to beggars on the street outside of his competitorsâ distribution centers - his eyes and ears everywhere. The next layer, hidden to observers until my research, was a set of advisors that filtered and aggregated the noisy and seemingly intractable information coming from all the lowly informants. The advisors were trained by the insightful BakhâprÃ¤p to base their interpretations of informantsâ info on how much they trusted the lowly minion and on how useful they thought that particular type of data was to making their estimate.

The illustrious merchant could never be certain that an advisor was giving him an accurate estimate, whether because their sources were untrustworthy or because the advisors were untrustworthy. To BakhâprÃ¤p, it was the same. This placed the burden of ranking the lowly underlings on the advisors. When he averaged each of his advisorsâ estimates, he weighted them by his trust for the advisor- in the exact same way they had been trained to do.

-Â§-

He must have held absolute trust in his system and an utter lack of trust in human subjectivity. In BakhâprÃ¤pâs journal I found a table of numbers that appears to have grown over many months. At the top of each column of numbers was the name of an advisor. Each week, the merchant had assigned each advisor a number, either a little more or a little less than the previous week. These numbers quantified how much he trusted their estimates. Whenever the aggregated estimate for the week was inaccurate (causing the merchant to lose money on his trades!), he mostly decreased his trust in the ones he had trusted the most, and grew only slightly more distrustful of those he had not placed much faith in. The less he trusted someone, the less they were paid- so the advisors did the exact same thing as the merchant and placed less faith in the reports of those theyâd paid attention to the most the first time, and listened more closely to those theyâd dismissed before (paying accordingly).

While reading about his system I conjectured that it would spiral down to mediocrity if, by chance alone, an inaccurate estimate was generated. For then would not the deservingly trusted advisor be demoted and the conniving, worm-tongued vassal be promoted in the false upset? And then estimates would end up even further off, driving down the loyal counselor even further! But my reasoning was flawed. As long as over half of the estimates were accurate, then the trust in the top advisors would continue to rise to a certain equilibrium, where the available information reached its predictive limit. But if less than half were correct, then the ones BakhâprÃ¤p was paying the most attention to were leading him astray. So it did not matter that they had been the most trusted- their estimates were useless whether by inaccuracy of information gathered or by deceitful communication and they would continue to fall in favor.

The extensive entries in the journal narrate how difficult it was for the merchant to sometimes ignore the passionate recommendation of long-known advisors. It also shows how stressful it was for him to ignore the rumors which circulated the Bazaar and to only pay attention to his network of advisors. He had to merely trust that some lowly informant had reported the rumors and that it had been taken into account in the estimation. In a way, he was in the dark about the gears that drove his black-box system, but he had faith in the function by which it had been organized so he had faith in the whole.

BakhâprÃ¤pâs system is remarkable because it required no real thought once it was set up- so the merchant had plenty of time and money to indulge in a rajahâs lifestyle.
More...

This story was originally posted a few days ago on maxdama.com

Regards,
Max

dtrader98 · Jul 26, 2008

My .02 cents.

I like your writing style and the metaphorical analogies you made.

However, I think it would be extremely difficult for someone (who hasn't built a NN previously) to translate the narrative into a quantitative running model of a NN.

Reminds me a bit of the journals from the myst. You could have hidden diagrams and tables of the advisor's inputs and elaborate drawings of mechanism's behind how his methodology operated.

It would be nice if you also made an analogy of how he went about picking the original set of liars, scouts, and thieves from a universe of many, many potential liar, scouts and thieves (as many as there are grains of sand in the universe). Therein lied the true key to the wise Nyeralneht's success.

Good Job. I'd give you a hand clap emoticon if they had one.

maxdama · Jul 26, 2008

dtrader98,

Thanks, I certainly agree that it would be impossible to create a quantitative NN from scratch based on the story.

Input selection heuristics would be a good addition. While writing I also realized it would have been more complete to explain how the hidden layers [advisors] are necessary to give it non-linear prediction ability, and also to mention threshold functions- sigmoidal etc. There are quite a few fundamentally necessary concepts to cover with NNs which makes them challenging to explain without getting bogged down in the details.

Regards,
Max

dtrader98 · Jul 26, 2008

"While writing I also realized it would have been more complete to explain how the hidden layers [advisors] are necessary to give it non-linear prediction ability"

hmm... I would argue that one simple advisor ( a differential comparator with the error used to control feedback weight) would be sufficient to describe a non-linear prediction ability (albeit limited).
But I digress.

I like your blog as well. I always find regime changes fascinating, but extremely subjective (i.e. at what point does the regime change).

This is the problem with things like hurst analysis, the resulting value and corresponding market type (trading/trending) is only useful in retrospect. At least from my observations.

Cheers.

Log in or Sign up

Neural Network Prediction: a Narrative Explanation

I appreciate your feedback

Thanks- That made NNs easier to understand

Hmm- I didn't really get it

Um- The creative approach didn't ring my bell

Hey- Read my comment below

maxdama

dtrader98

maxdama

dtrader98