Why do I see "Trends" in Randomly Generated Data?

Discussion in 'Data Sets and Feeds' started by Rahula, Feb 21, 2008.

  1. Not really ;) As I said, it depends on the risk adjusted returns. I worked at a hedge fund recently. The risk adjusted returns were great, little bit better than what I wrote above. There were TOO MUCH money coming in from big institutions. The fund had to stop accepting money for a while ;)
     
    #391     Mar 13, 2008
  2. It does take a long time to complete the loop on why people with money (Those who wish others to manage the money making job.) cannot use it to make money, ordinarily.

    The largest sink (unlimited) for making money is sector rotation. The financial sector is very badly off right now and it is easily seen that people worked in and through that sector are now making tons of money by being on the right side of that market sector; others unfortunately are really taking the pipe. Analysts who rate stocks and sectors have lost the ability to understand some sectors (the finanical stocks, for example) and keep up to date at this point. Foxes guarding chickens, so to speak.

    In sector rotation as in any trading application, making the highest money velocity is done as the sector rotates towards the downside: financials, drugs and health care are doing this.

    Bear Sterns (a bad investor example) has just announced that they are leaving cuclics and going towards non cyclic staples. Buffett has made the word "durable" famous in terms of going further than "staples" for dealing with non cyclic sectors.

    Moving capital to where it can make money is not very possible for most financially oriented businesses where investment and money management is the theme. Assuming the shifting of capital is largely done to optimize money making money as a commodity has demonstrated results. The level attained is just about what it takes to stand still in true values or true wealth.

    For 2008, we get to see the federal full court press to salvage the IB's and HF's. Even the President loaded the gun today on "sovereign capital" by letting the Economy Club of New York know is was good money to be "taking" because it was our money after all. The three way bail out of BS was very humorous. Slipping a third party into the deal so BS could get a fed deal was really funny.

    Treasury Sec's speech (get a transcript) was a cool 101 on the four steps of securitzation, "squares" included, and tranch slicing, too. His saying we aren't going to do any of that anymore was terrific fodder for Jon Steward and Steven Colbert. How he explained the President's Working "club" was going to come up with the new transparent way was "pricelss".

    So the scene is set for bearing down on making money. The random and non random oriented threads in ET are pointing out just how well the Academic World gives money makers things (published papers and conferences presentations) to chew on.

    In 2008, we get to really make money every day (That's always been true) and the public is becoming very well aware of this. It is like a global pattern is having a Breakout that makes everyone aware that price change is how money is made.

    Look at one stellar thread where a guy uses a 2 point bracket and makes 2 points a day per contract. This is so he can turn 5,000 bucks (2 contracts) into an annual wage for a guy 3 years out of college. At 8,000 dollars capital he goes to 3 contracts that will make 2 points a day per contract. Others have figured out this is possible be cause they applied an Excel spread sheet to it.

    So will people form groups of thinkers here to respond to the academics and to the stellar threads of 2 points a day per contract compounded? The IEEE certainly isn't on that path. Neither is Trader's Monthly with their emails on "buy a plane" this week and God knows what next week.

    ET has had a tradition of threads that fell into a few forums and now it is at a contemporary operation of many forums and many short lived threads within the multitude of forums.

    Willmott does it's narrow thing. Wikkipedia does the broadbased thing. Why not parallel these kinds of things with a least one baseline substantive teamwork effort on making money by trading?

    I'm sure there are many reasons to not do this. Are there a few reasons to do it?
     
    #392     Mar 14, 2008
  3. Jerry030

    Jerry030


    Jack,

    On your comment about a teamwork effort, I've attached part of some discussion on another forum. In it we have some survey results on profitability in trading systems. In summary, there are few who will admit to making much profit. There is of course much talk about how to do it, but few will admit to having done it.

    The second par of the post I quote below proposes something call a Blind Binary Collaborative along with my experiences in various private investing teams and why they don't work.

    The response to the idea of a Blind Binary Collaborative was nil. The talkers have nothing to contribute and those making significant returns from advanced systems have no need to collaborate. This leaves very few people to build a team with.

    However, I'm still open to the idea for anyone using predictive models with the technical skill to handle automated exchange of predictions.

    Jerry

    ----- originally posted on Yahoo

    http://tech.groups.yahoo.com/group/Neural_Networks_and_Knowledge_Discovery/

    --------------------------------------

    <snip> The results I mentioned aren't confined to neural network or rule based systems.
    Even in general trading groups the results are similar. Here is one from Profitable Trading:

    Responses
    Choices Votes % 1 reply
    I’m Paper trading only no live trading 1 10
    I’m looking to develop a proprietary system 1 10
    I’m looking to develop a non- proprietary system 0 0
    Proprietary: Less than -50% return 0 0
    Proprietary: Between – 50% and 0% return 4 40
    Proprietary: Between 0% and +50% return 0 0
    Proprietary: Between +50% and +100% return 0 0
    Proprietary: Greater than +100% return 1 10
    Non-Proprietary: Less than -50% return 0 0
    Non-Proprietary Between – 50% and 0% return 2 20
    Non-Proprietary Between 0% and +50% return 0 0
    Non-Proprietary Between +50% and +100% return 0 0
    Non-Proprietary Greater than +100% return 1 10

    Your comments have sparked an idea, follow me on this if you will in my next post which will outline this in a new thread.

    Jerry

    Blind Binary Collaborative

    Your idea as a general concept: People cooperate on creating trading systems with improved profitability

    Comments: I've been in a few of these over the last 5 years. These ranged informal Internet cooperation like "here's may data and results, can you improve it?" to formal undertakings with proposed contracts, percentage splits of profits, web sites and hundreds of hours of group live chart with multiple parties world wide. In the end all came to naught for some fairly simple psycho-social dynamics.

    People who have only an Internet relationship have a very low level of bonding. One can spend lots of time on emails and analysis one week but when you see a better use of your time or have learned all you can from the interaction with the others in the group the motivation for additional cooperation quickly vanishes.

    Additionally many people in such situations present themselves with all kinds of skills and resources to gain admittance on the hope they can learn a lot before it's obvious that they can't carry their weight. Then those with skills conclude that they have a better use for their time than giving others a free education... and pop, the group vanishes in smoke.

    Since this kind of work is a bit like prospecting for gold on the surface there is little self interest in telling others the best place to look for riches.

    Summary of Dynamics:

    1) Human nature dynamic: Greed and self interest = get as much as possible while giving as little as possible.

    2) Human ego: What I've discovered is potentially really great, if I could just refine it a bit, why should I share my best stuff until I know I'm getting something back of equal or greater value. So I'll toss out a few tid bits and see what I get back.

    3) Fishing Expeditions: Folks will announce they have this great system, a string of investors and anyone who can help a bit with a couple of loose ends will soon be rich if they join forces. We have had some of these on this group. They usually vanish after one post.

    I could go on but if you are still with me on this, you might agree that these issues present a real obstacle to real cooperation.

    New Idea: A Blind Binary Collaborative

    What does such an odd string of words mean?

    BINARY

    A binary in many areas is something that takes two things that are kept separate in order to work. Some kinds of Epoxy adhesive require the mixing of two compounds in equal proportion to make the glue. Another example: the triggering sequence on nuclear weapons requires that two separate keys be pressed at the same time by two different people.

    BLIND

    Nobody know who or what is involved in the process. A blind taste test for a consumer product. If people knew what they were tasting the results might be skewed. A numbered Swiss bank account is blind in there is only a number and a password, the bank is blind to who you are in normal operations.

    COLLABORATIVE: Some kind of cooperation.


    How would a BBC work?

    The key thing is that it would

    1) Eliminate the problems I cited above

    AND

    2) Require ongoing contribution from all parties.

    Let me give and example of how it world work and let me know what you think.

    Assume you and I decide to BBC on the Russell Index Future.

    We both start with the same data.

    We both attempt to forecast the variance in average price 5 bars into the future.

    You do whatever you do to model that, say you use 27 carefully researched Dependant Variables and a combination genetically mutated NN architecture of your own design to forecast the Independent variable.
    You send me the result.

    I do whatever I do on the same original data set. Perhaps I use 6 different architectures BPM, SOM, RBM and a couple of GA modules on my secret 14 variables. I send you the results.

    Neither one of us share anything about our methods, architectures, designs or discoveries....ONLY the result.

    We each then combine the results of or best efforts by whatever means makes the most sense to us, to generate trade signals. We share the trade singals.

    Assuming the results are something better than what either of us could do on our own on the Russell we will likely both trade the Collaborative results. Since neither of us will know anything about how the other person did their half of the system, we are both strongly motivated to:

    1) put forth our best efforts and
    2) continue to cooperate and generate signals. If either person stops, the system stops working because it's both Blind and Binary. In that case we are both left with what we started with.

    This idea, while radical, eliminates the obstacles to cooperation, all that effort spent on cross education and trust building and focuses on hard results: is the Collaborative results better than what I had on my own?

    Jerry
     
    #393     Mar 14, 2008
  4. Summary of Dynamics:

    1) Human nature dynamic: Greed and self interest = get as much as possible while giving as little as possible.

    2) Human ego: What I've discovered is potentially really great, if I could just refine it a bit, why should I share my best stuff until I know I'm getting something back of equal or greater value. So I'll toss out a few tid bits and see what I get back.

    3) Fishing Expeditions: Folks will announce they have this great system, a string of investors and anyone who can help a bit with a couple of loose ends will soon be rich if they join forces. We have had some of these on this group. They usually vanish after one post.


    I can understand these points. Probably they are common amongst most people.

    In the two examples you gave, you characterized each method by describing what elements are at the heart of successful systems. People do not put those in the space for some reason as you say.

    I saw the Yahoo the posts and they do fit the description of the system dynamics.

    You steps for collaborating, I understand as well. The results you posted depict a spectrum of possibilities. Over time, I have met many people face to face who are leaders of their organizations. None of them fit within the performance spectrum you posted (they have better results) and their common problem is laying off capital to others who could perform in some sharing manner (They accept results less than theirs but they would not offer their system to these capable people). During real time trading seesions, these people are willing to share and exchange trading timing. It is not so much them saying something specific about how their system dictated the timing signal but it is more like they want to share that their timing and another's timing is in synch for a fiven market. An example, would be the one system trades inside another AND at critical times there is synchronicity with the slower system. During breaks ther is a lot to talk about and share. This may be a BBC thing going on informally.

    I think that your BBC may be something that gets around several of the system dynamics issues. Results and defining them in terms of trading signals, may take some doing with respect to the traging signals, especially.

    A soft way to handle this may be delaying the transmission of the trading signals by making them available just periodically and after the fact.

    The main focus of BBC is to have mutual aid for a type of people that are working to improve and refine they parsonal efforts. It is not to provide treal time trading information that can be acted upon; I understand that. I have had two types of experiences as a consequence of people treating my trades as "pre-news" or something approaching that.
     
    #394     Mar 15, 2008
  5. Jerry030

    Jerry030

    On your last point, it is really the reverse.

    The idea is not a mutual aid or educational project but a simple mechanical combination of each participant best prediction for the same market and the same time frame. What any one person does with them or how they combine them is their own business. One doesn't care what others are doing nor would one know anything about how they are created or anything else. The only thing that maters is if the when one combines them does one have something better than what one has alone?

    The project assumes that each participant has an automated mechanical trading system (rule based, quant or non-linear). Since these function automatically in that they acquire price, apply the system process to generate a result and then issue trading signals based on that result, the additional step to transmit the result to others is trivial. For those of us using neural network or other non-linear models, beach new bar generates a new predicted result. Of course only some of these generate trading signals.

    To illustrate with an example:

    Assume that each system predicts the S&P Index future
    with the rule of Enter MOO and Exit MOC. All positions are day trades. Each BBC system can say Long, Short or No Trade (NT).
    Assume 5 participants.

    BBC Traders 1 2 3 4 5 Conesus Signal
    Daily Signals 1: L NT L S NT Long
    Daily Signals 2: NT NT L NT S NT
    Daily Signals 3: S S S S NT Short
    Daily Signals 4: S L NT L L Long


    Profit Factor 2.7 2.5 3.4 3.1 3.0
    of Individual
    Systems

    The result of the combined system is likely to have a
    Profit Factor of greater then the stand alone PF of any individual system, based on some experiments with Ensemble models.

    This dynamic is what would keep people involved and create a naturally self organizing ecology. Signals that contribute to both the group, (as a net result), and to the individual, (it's better than what they had on their own) will keep the relationships stable. Those that don’t do both will be naturally pruned both individually and collectively.

    This approach eliminates tons of wasted time by restricting the relationship to results but not methods.

    Jerry030
     
    #395     Mar 15, 2008
  6. You make a nice statement of iterative refinement (through a type of information sharing) leadng to better performance individually and collectively.

    I am often reminded to focus on batting rather than ball parks. what you suggest does deal with improving batting averages and that is a very pragmatic and valid approach.

    For making more money velocity, getting hits all the time by swinging properly depends on few factors of a system. Timing is the focus and the context gives the timing focus. By shining a light on the present from the past and looking at the shadow in the very near term future and swinging based on the near term future shadow, I believe, optimizes the batting average.

    Considering the complex catcher-pitcher strategies to get a batter out is not the direct route to personally improving one's batting. What is important is more like considering how people converse. The two facets of each person that divide the conscious and the unconscious aspects of conversation.

    Conversations' unconscious characterisitics dictate the conscious.(LeDoux, 2002; Andreasen, 2005).

    Your example focuses on direction, if any, over timing; there is a correspondence in this regarding the unconscious and conscious re: traders and markets. You also put the batting average on the table (I know baseball is just crude and does not value hits qualitatively as yet).

    Three things come up for me which I feel are important for iterative refinement of systems (ball park) and trading performance (batting average):

    Autonomy of the parts (sub systems) and least connectedness (Alexander's Law).

    Heirarchy of functions within sub systems.

    Fineness of the trading required to extract what is offered (A system sufficiency requirement).

    BBC would need a way for the collaboration to turn from one refinement emphasis to another as the refinement ensues. Your example includes a series of four signals for a session. I would envision that the number would advance in about 6 stages of refinement to be able to approach taking what the market offers.

    One thing I square away with others who are excellent traders is our relative ratio of signals. I have always been the higher value in the ratio. 4:1 is common. If a day were considered and your example represented someone, the beginning point would be 10:1 and the PF would be very different as well under the BBC mutual sharing approach.
     
    #396     Mar 17, 2008
  7. Jerry030

    Jerry030

    Sorry if the example was not clear. It's not 4 signals for a session but 1 per session. The example would cover several trading sessions. The goal is to perdict the realtionship of the Open to the Close, prior to the Open. Call it a 1 day 1 bar model.

    One thing I square away with others who are excellent traders is our relative ratio of signals. I have always been the higher value in the ratio. 4:1 is common. If a day were considered and your example represented someone, the beginning point would be 10:1 and the PF would be very different as well under the BBC mutual sharing approach. [/QUOTE]

    Could you exapnd on this? Do you mean the ratio of signals to bars as in 1 trade on average every 4 bars?
     
    #397     Mar 17, 2008


  8. Could you exapnd on this? Do you mean the ratio of signals to bars as in 1 trade on average every 4 bars?
    [/QUOTE]

    what i failed to convey was a comparison of two traders.

    Here is an example from Las Vegas a few expos back. the CEO of Thinkorswim is trading the ES live in a session that may have been 2 hours long. as time progresses he is doing about 1 trade for every 4 that I do. So I call the ratio of my trades to his 4:1.

    Today, I was in a conference for about 1 3/4 hours. One of the conferees wanted to step up the pace on segments traded for the simple reason that the maoney was there he said. I am glad that it came up for several reasons. It is better to take what is offered if a person is able. It is very important to not do "freakout" trading on the otherhand. So for the next three days everyone is doing an assignment, getting together and exchanging info and then repeating the assignment to make another level of correlations then meeting again to compare notes. That makes them ready for Monday morning at 6:30 which is market time in Tucson.

    The consequence of this weekend transition is that from now on I will be mentoring (live trading is going on with each person) at about 5:1 compared with the trading up to this point.

    As has been said an assessment of the market takes in about 7 items for humans or with the use of scripts as described by gehko (who is on to how to automate trading except for a few things). To trade slowly with long profit segments and to trade five times as fast STILL REQUIRES THE SAME NUBER OF PERIODIC ASSESSMENTS except that the data sets are different kinds of data sets from time to time.

    The nature of making money is that an exit is an identity to a following entry and they both are simultaneous is data taking setwise. If a slow trader (like CEO of TOS) is doing profits, he does it differently than a fast trader simply because the criteria for segments is different.

    This am the 5:1 contrast would have been about 12 points to over 25 points per contract. There was a group decision to make the 25 from now on.

    In simple terms using indicators, most indicators offer about 15 signals. The slow trader uses but 5 of those and the fast trader uses all of them. A slow trader, those you see at most demos and trade shows do one trade at a time. A fast trader has 2 or three future trades presetty well set at all times. Right here you see what automated trading is all about. Automated trading has nothing to do with entries and exits or doing one trade or turn at a time.

    Using the topic of the thread "seeing trends.....", a slow trader is on a different fractal than a fast trader. They have the same data and probably the same display (thing in my case a clusterf@#% level of display or available data) but they operate off different signals.

    A few past ET participants demo'd doing slow and fast trading simultaneously; one streategy operated "inside" the the other. Both the single account and the mulitple account was demonstrated.

    You mention "soft program" to gehko after he gave his restrictive view of things. Which of you is going to be taking more out of the markets? Which of you is going to see the horizon as several trades out front of the present?

    Somewhere there is chit chat on esthetics og displays; these are conversations that are not about making money. People use the word "busy" to describe displays. This is not something that has any value for getting the relationship of the computer (whether manual or automatic) and making money in the markets.

    Why can't anyone get even a glimmer of what the markets offer? If they did they would be bearing down on things.

    Here is a comment on the last FOMC trading opportunity from a fast trader viewpoint.

    1. It is known that the before and after envelope of price is an identity. Let me translate that down to two levels above Tdog and Maestro. Before: Price is in a channel or to say it another way, there is a systematic and repeated cycling about the mean where regression to the mean is on and off about equally in terms of time and positive and less positive defined moves. This is what is going to be going on after the news effect of FOMC is over. It is just as if a period of time can be blacked out from just BEFORE FOMC to a point where there is no statistically significant difference with AFTER. so block it out ahead of time as a fast trader does because he has several future trades blocked out.

    2. It is known that a trader has to be prepared for any response to any FOMC announcement. So the first trade going into the announcement is fixed by having to take a trade in either direction. The values are dictated by the extreme dominant movement before the announcement and the boundary of the last internal pattern on the non dominant. It may take a little market knowledge to understand this and about no programming savvy to script it.

    3. Knowing 2. for the fast trader is determined, in terms of which way the BO of the bracket of 2. is going to occur anyway because the fast trader knows the number of cascades that will occur in an FOMC type or level of importance type announcement. This is simply based on the unconsious actions and rections of the herd that trades. Ask any mathematical psychologist. (that is humor). there a three cascades and they are odd harmonic cascades.

    4. What follows in cascading markets is a return to the former market operation as the TC's (time constants) of the news wears off.

    NB. This means for the fast trader there is a BEFORE and and AFTER an event. There, in this example are four intervening market actions between BEFORE and AFTER.

    5. The price gap between BEFORE and AFTER is determined both in magnitude and direction and real time separation. In the example it is up, a lot of points, and three cascades plus an asymptotic recover long in time. For a fast trader, these are known and precalced roughly.

    So we know the BEFORE and AFTER trends are long. We have an up gap to deal with. That means the first cascade is short ending on a spike down and the second cascade is long ending on a spike up and the third cascade is down and transitioning into a long oriented damped return asymptotic to the former long channel where a dominant traverse (long) comes first followed by a nondominant traverse (short) to the RTL giving points 1, 2, and 3 of the resumed trend.

    The money velocity is quite high for the fast trader. a bracket in short, reverse long, reverse short, reverse long and then traverses of the resumed channel.

    writing it in the log ahead of time is normal. Leaving some rows for comment on the rate of cascading, noticing how the extent of blow ups is occuring (all on the T&S) and watching the cascade on the DOM by volume per tick. Since the pertinent rows are filled in , Then only price has to be entered on the reversals, time can be picked off the print during debriefing.

    Traders size up other traders in several ways. The ratio is one of them. How parasitic traders are is important too. I am totally parasitic becaue I am risk adverse and I do not compete with others; I simply front run them. I rarely meet anyone in live trading demos who front runs me and several of them become very aware that I am front running them. It is unverving, they say afterwards.

    Automation is easier since it is done at leisure and not during RTH. Practice trading using display camtasias is also on a ratio: 6 1/2 hours can be done in about 40 minutes: about a 10:1 ratio.

    For any fast trader, it is a good idea to stay at least 5 to 1 minutes out in terms of having marks (time, price) corrdinants already layed down. This has a lot to do with harmonic analysis (Why a flute sounds different on the same note).

    Oh, on the FOMC example, a person can also over lay all of that by taking advantage of the money that is on the table as defined by bar overlap. This is like trading "inside" the above strategy. the defining thing is cascade interruptions.

    At this time of the markets, there are always going to be a lot of irrational news announcements as the assortment of players, fed and otherwise, play their cards. Everything is an opportunity and usually can be played a few trades into the future.
     
    #398     Mar 20, 2008
  9. Jerry030

    Jerry030

    Jack,

    Thanks for the reply, but I think we trade in different dimensions as I still don't follow what you are saying.

    It's not relevant to what I do so don't bother responding further.

    Jerry030
     
    #399     Mar 20, 2008
  10. I agree with what you say but damn Jack, is it possible to condense your thought process a bit?

    To condense the FOMC comment from my standpoint and experience.

    There was a group of us watching the ES on both slower fractal and faster fractal charts before the announcement. The slower fractal showed us that price was still going up with strength. The faster fractal showed us that price was about to retrace.

    Just prior to the announcement a short triggered on the faster fractal. It created a nice profit pool, stopped, reversed, triggered a long and went back up breaching the previous high creating another profit pool long.

    All the while the slower fractal showed that particular move/retracement as a simple minor oscillation in the continued move up. which confirmed a short term top itself later that day.
     
    #400     Mar 21, 2008