10 months fulltime and still no edge

CHVID you have a good point it does not take 12 years to write an automated system. The majority of my system was written in 18 months. But what took ten years to learn was some of the following:
- What style of system do I want to trade? Swings, Trends or Volatility.
- What time intervals are applicable? Daily? Intraday?
- Which trading rules work well for the style of trading in the time interval I trade in?
- What risk management rules will I use in the system?
- How do the risk management rules fit with my style of trading?
- How does the system compensate for changes in the markets and the security?

Then you add to this mix testing?
- What are the minimal expectations for testing results?
- What does in my trading plan expect as minimal when I walk forward the automated system?

I can go on and on. I tested for years before I answered questions like these and I’m still not done. Part of the problem is the horrible tools retail traders have to work with. This makes the process very slow. I spent 35 years as an IT professional before I retired. They would laugh if they saw the trash tools we have to use.

 

Rabbitone - you (seem) come to this from a daytrading background.

That is one way of getting into this.

If you have good grasp of math and statistics - that is another.

Though I will you this: I too did years of studying and testing before actually hitting production. And still I got wiser when I finally did ...

 

Good ideas. And trade more long-term. Don't be a daytrader - be a swingtrader.

 

How long did it take John Arnold to build about $2 billion in wealth by the time he was 34 yrs old? Within a couple of years after graduating from Vanderbilt, he was one of the more profitable energy traders in the market. Since his mid twenties, he's be averaging annual returns of 200%+ per year. The notion of about a decade of experience before you are successful is anecdotal "heresay" at best.

There are folks who've been trading for more than a decade who've yet to become consistently profitable. I'm sorry, but within 1 -2 years of intense training with good mentorship, if you still can't trade consistently profitably, then perhaps it was not meant to happen for you.

There are scores of young traders (i.e. John Arnold, Brian Hunter) who have made (and sometimes lost) hundreds or millions or billions. On the flipside, there are a myriad of "experienced" traders that have yet to earn a fulltime income with all those years of trading.

IMO... it comes down to risk mgmt, money mgmt, & common sense.

Walt

 

Chvid – in the 60s in college it was math and the then 35 years of IT including teaching college at night. My discretionary day trading was poor so I gave that up.

What got me interested in automated trading systems was when I was Manager of DB2 Database Administration (IBM’s big database product) for a large bank. I had to manage trust and trading databases. Part of this was reviewing the trading application to make sure it accessed the database correctly. I got to know many of the basic methods of trading big accounts (many parts were so proprietary I could not view them).

In the years before I retired I tried every trading piece of software I could get my hands on to try to build the systems I had seen making nice profits in the bank. It was in this process I realized I could not take the banks objectives and put them in my software.

Setting up my automated trading objectives was very tough. For example define a swing trade. Seems simple enough, until you hit the details:
1. Time Interval – Try weekly. No trends too much. Ok, Daily. But, if I use 60 minute intraday I can get a better entry.
2. Set up rules – What about 1-2-3 set ups. No wait reversal bars work better. What do I use for confirmation?
3. Price data characteristics – Do I trade only pullbacks in trends? No…Wait trade trading ranges offer better rewards?
4. Price Volatility – Swing Trade High volatility? No too many fast reversals…. Medium volatility is better for swings.
5. Initial Stop Loss – What works best for swing trades? ATR multiple?
6. Trailing Stops – When to use them for swings?
7. Profit Target – Does a profit target work best? Or should I let the trailing stop do the work.
….
26 Best Market conditions to trade in – Bull market. No I don’t trade in the trends?

What all this adds up to is the number variables require a larger number of combinations that the average trader has time to decide. The solution to this problem is to build a small number of models. Each model fixes the solution to a large number of these questions and the rest of the questions are allowed to vary. When you work through the models the trader will discover who they are and how they intend to trade. This is what can take an extended amount of time.

How could this process be speeded up - by simply by use batch processing that we don’t have. That is how the bank did it. They wrote the models. The defined programs to batch test them against 8000 stocks every night. This went on for several months. Then the viable models immerged.

How do we test swing trading? One model, one stock and one optimization, forward test and back to the drawing boards. This process takes forever. Though I do understand batch testing is starting to occur.

Chivid. I do enjoy your comments. It gets my old brain cells working full time. Thanks

 

Thanks Rabbitone for your comments. Regarding the sector/etf model your are right it is basically a scanner with some entry logic. The reason I want to try it out is because it is easy to code and you can go long and short with hundreds of symbols at the same time. However the gap trading as you also states will include more logic / common questions / price patterns

Your detail list is so much true which is also why I wanted to build some more generel models. Each time I have started to construct a new system my parameters has increased and increased and increased. And note by parameters I don't mean changing the parameters in technical indicators but the overall trading setup. I would be happy if you would comment a litle more on your solution to this: builden a small number of models.

Regarding testing I use two setups. I use Matlab for batch testing not just one symbol at the time but closer to 100 symbols (not 8000 stocks I used to have 200 stocks in my database which matlab fetched data from). Ninjatrader for testing 1 symbol at the time and smaller setups.

 

Jones247 – You made and excellent point about the expectation of developing trading automation. Let us continue that premise that you set up. If a trader is consistently profitable they should have no trouble building automation to keep it that way in a short period of time – say one or two years.

Since I was in my 20s, about 40 years ago, I have had a small trading account. This account grew from studying cycle mathematics in grad school and calculating cycles manually. My trading grew in fits and spurts over the years and is now about 36 times what I started with. This includes going bust 5 times from not understanding the markets first. The strategy I trade today is part of what I traded 12 years ago but is now automated. I was not automated 12 years ago.

During the 12 years it took to automate I still made money trading my manual set of rules. Now with Jones247’s premise, if I’m like John Arnold, I should have been able to automate my rules in a year or two.

In 9 of those development years as an IT manager time was tight and I devoted a cumulative 6 month’s added up to the project on nights and week ends. In the last 3 years I have been 100 percent active in developing the cycle and cycloid automation. So when I add up my project time I get 3 years and 6 months to get an active automated process. And when I consider my age and consider I have lost some of my mental capacity. Then the Jones247 premise of developing automation in a couple of years seems reasonable.

The only intangible in this process is John Arnold had years of time in grad school using monster computers to test his theories. When I went to school it was done with a slide rule and math tables. It wasn’t until the early 1980s when I owned a computer store that processors could even begin to do the cycle and cycloid calculations I had entered in my college notebooks. The first application I wrote on the Apple 2 Plus was hand entered data, stock by stock each day. Calculating the cycles and cycloids for a year’s data for 10 stocks took 29 and half hours. With today’s computing we have come a long way from that.

So my conclusion is yes an automated system can be built in a year or two, IF. The IF means the trader must be not a novice but a fully versed intermediate or advanced trader who can handle the 4 or 6 straight losses and large drawdowns that an automated trading system will deliver. To state it another way if you spend 8 to 10 years becoming an accomplished trader than in one or two years you should be able to build an automated system that will perform the same task.

This leads to the next question. What are the odds that a novice or beginner trader with minimal trading exposure will build successful trading automation before they learn how to trade? In other words how much trading experience does a trader need under there belt in order to know the correct trading practices to put in their automation?

 

Estim – Excellent comments about batch testing. This is one area than can tell you loads about. I have run more than 5000 tests in the 12 years of testing.

First, 5000 tests where most single file until I got Multicharts. This may seen like a lot but I had too many models to test until narrowed the process down. Then I added a number of optional input parameters that took years of testing. To non IT people this seems crazy. But in 35 years of IT work I probably over saw ½ a million tests. That is part and parcel of IT work. Especially in programming IT where it can be 80% of your job.

OK back to batch testing. This first design issue with an automated system is how your automated code will handle the variations in directional and volatile price movement. For example here are some common questions that may be part of your system
• How do you set you initial stop using volatility or placement with price bars.
• How do you add ADX to your code to handle directional trades.
• Are your trailing stops using ATR to maintain trailing stops.
• When ADX is non directional do you shut off entries or take exits.
• When volatility reverts to lower levels how does it affect entries and exits.
• Now do you handle price noise events (gaps, spikes in price, etc …).
• Will price targets be affected by volatility events.

When you automate your batch testing some of the considerations for batch testing are:
• Are the market conditions typical for the automation I am trading.
• Do the securities in the batch test consist of homogeneous price data with respect to price direction, volatility and price noise.
• Do the input parameters allow me select the type of volatility that exist in the current market.

You can actually build a long list of batch testing questions. However one of the questions stands out and that is homogenous price data for testing. If you were to select say APPL, IBM, PFE, HD, MRK and RIMM for your testing you might see it fail in batch because of Pfizer (PFE) and Merck (MRK) do not have the same price data characteristics as the other securities with respect to direction and volatility. Now imagine testing with a 100 symbols. This may dilute the results of the test.

What I did to over come this dilution problem was test with a limited number of similar securities from a sector with the same price data . Once this test was past I moved to other sectors with different directional and volatility constraints. This overcame the dilution issue with a heterogeneous price data mix.

Estim I would greatly appreciate your thoughts on this topic since I have just begin to test more in batch.

 

Rabbitone,

I think you are familiar with the expression 'curse of dimensionality' from all your testing. Curse of dimensinality is the fact that when increasing the numbers of parameters in your model you must exponential increase the data needed to optimise the parameters in order to reduce overfitting. This suggest to keep it simple with few parameters.

Generel what differs from trading models compared to mathematical models is the error function. By this I mean, the overall setup parameters as you stated in your list is normally optimised using a framework that searches all possible combinations. And this off course leads to overfitting the data to one unique solution. So walk forward processes normally performs bad.

What I do to come around this is to used a different version of the typical cross-validation technique. Which I'll explain here considering only one symbol.

Split your data set into eg. 4 equal size. Optimise your model 3 times on the first 3 data sets and save all the parameters for each model. To select the optimal parameters you chose those that fit best to all 3 datasets. Finally you test in on data set 4.

This procedure should be applied as walk forward meaning you will have to introduce a new parameter namely the size of each data set. This parameter is optimised as well and is sometimes called the learning curve, that is, how much data do you need to train your model on before it produces a stable result.

Optimising using a framework like this should be valid when optimising only one symbol. If you consider more symbols, I agree with you, that first they should be classified in different price characteristics distributions. However when this is done, you can reduce the size of data you are optimising on and then optimise the parameters to fit on all the symbols.

Hope this make sense.

 

These securities are too efficient...
To trade profitably using infrastructure...
That is as simplistic as yours.

Your competition has:

(1) Sub one millisecond latency.

(2) Inside information.

So you have no chance.

Poker Pros always look for the softest game...
But it never seems to occur to people here...
You can only beat the least efficient markets out there.