Just to clear up, how are you backtesting then? Are you waiting for last traded price to go through your limits? Are you using L2 and T&S data to simulate the order queue? Because as you should know, these are the main issues when backtesting at this timeframe.
I would rather not say. But I will say I have accounted for the best case, a worse case, and brutal scenarios. I know I am most likely not doing anything groundbreaking and just about everything has been thought of in an infinite universe but I still like to keep my IP to myself. After this conversation I am going to take a guess you have had these ideas or concepts not work for you in a similar manner in some form? Like I said, I have been there as well, but 15 hours of work a day for 2+ years literally non stop has gotten me out of that rut. If you are still there I recommend you just get real creative. For me it was all trial and error. Are you 100% automated or just in it for the conversation?
Then discard that best case immediately because your place in the queue will be very bad if you cancel, resubmit or modify your orders every minute. Yes, I am 100% automated but my trades are from 5 minutes to 5 hours.
Yeah, the best case scenario is "too hot for TV" if you know what I mean. I knew that would be inflated. I would still be very much more than pleased if it was 1 step above brutal. I also have very quick access and ultra low latency at my disposal which I guess could do nothing more but help overall I guess. Luckily I have others that are taking care of all the loose ends so to speak which is really nice because it leaves me and others to solely concentrate on what we ideally do best. In a perfect world that is. I have just been fortunate enough over the past couple years to be put in this position though. There would have been absolutely no way I could be where I am now if I had a full time job or was still in school. I don't sleep a lot of nights, so I have put some thought into this as well as trust in others I guess. In these smaller 5 min timeframes, are you still running the same strats as you were previous years? 99% of all my work is long only btw.
I'm still running the same strats but I don't trade smaller than 5min. My average time in a trade is about 2hrs. It's certainly easier that way. I know for fact it requires exponentially more time and money to trade the smaller timeframes. From your posts, it seems you are using 1min ES data and cancel/resubmit your orders every bar. That's not good enough. You need lower resolution (1tick = T&S), more info (L2 data) and custom software (queue simulation) to properly backtest ES trades lasting less than a minute. The reason is your average trade in terms of $ will be very low, probably less than one or two ES ticks. So you can't trade this with MIT orders instead of limits. You will depend on good limit fills to make this work. Or else you'll lose money. If you don't believe me, just live trade your stuff for a few months with the lowest possible size. You trade very often so one or two months will give you enough sample size to evaluate slippage.
Yes, it is quite the work in progress. Fortunately for me I am only responsible for so much. I do not doubt anything you are saying but I do have enough information *IMO* to plan for a wide enough variance for now that will yield well. It is now down to execution within these resolutions and some other things of course. No detail being of any more importance than the other as they are all important. I have had other thoughts too about throwing up a "web" of orders with multiple length limits / cancellation times. This just added to the frustration that I was experiencing and I guess I found it easier to try to build out these models using brutal - > ideal scenarios and I guess basically figuring out how much shit will indeed be sticking to the wall. Truth be told I am certainly not the brightest when it comes to mathematics / statistics etc. The only thing I got going for me besides a receeding hairline is determination, some half-bunk rolls and some white on white paper that has yet to be determined. I think though after all this research and all the bullshit that accompanies it I really do not see how great math skills above even high school level will get you to another level over any other retail competition that is. I think if you put 2 artists in a room and maybe some guy coming off a prolonged Dead Tour still in recovery they would put up better results than 2 kids fresh out of school with advanced degrees in mathematics. That is again of course dealing with retail tools. Diving into the tools of major HFT boutiques and their network of information is most likely another story. Maybe not though, but that is 1 full step ahead of where I am. Making this thing happen though would be a big step. Maybe I could even launch myself into the big leagues of full out drug abuse that HF Managers and Politicians only are lucky enough to partake in. But on a more serious note, how long have you been trading these same programs? If you didn't read all the thread I basically came to the determination shit started to really turn around in these smaller timeframes right around the start of 2008.
We do not agree on anything but the very important requirement for accuracy. My prior comments dealt with item 1. My comments below are from a presonal context that does not have these issues because these issues were processed before a long term context, meaning during a long period of time. I ranked five things you are failing to deal with, or more specifically are dealt with unfavorably. As we know, others have pointed these out to you already. They have also implied the basis for thier views. 1. Failure to appraise properly, 2. Omission of market variables, 3. Failure to have a neutral bias throughout, 4. Self defeating risk protection position, and 5. Market timeframe context. As you see there is a great range of successful trading systems. You are going quite well as you say. This is even with the downside atributes mentioned above. What wolud be the marginal improvement if you handled each of these to get into the ballpark? These items also happen to be synergistic.
1. You could apply this to any scenario dealing with anything at anytime. I am unsure yet if this is even true because my job is to model. Think about it this way. Microsoft pushes out Windows 7. Is it perfect? Fuck no, you need to update that shit especially if you have x86 systems etc. I think people are much too focused on this and when this happens you start to dwell on minute things that bring you away from the initial purpose of your whole strat. in the first place. I am sure we have all been there if you have been involved in these things as I have. 2. Dude, I have MANY variables - but only a few that will make a glaring difference. I think the downside to bringing in too many variables is again you get caught in a situation where you are optimizing these variables to death and ruining the "fluidity" of your original thought and creation. 3. No, IMO you should have a bullish bias in this market. All my research from 2008 shows this. No reason to even want to take short positions IMO. Again, I am buy side bias because I am not a fan of shorting markets. So I may be blind to one glaring thing that is staring me in the face. I will never discount this. So yes, I will give you that. 4. My variable exit strategies are the best parts of my program. Averaging about .1% MAE. Remember, that is open position. I have had these methods for a long time and trust me, these are by far my best ideas. I think I could probably improve on entering the marketplace, but at the same time would again hate to start to marginalize things where I am again losing the "fluidity" and original thought. 5.Yes, this will be expanded upon in time. It's not like this point hasn't been sticking out in my mind but from all my experience, again, it is much better to create an initial model and this has a large enough sample size to be statistically significant. Once you have this "model" then you can go in and really figure out what shit will indeed stick to the wall... IMO it is best to use a very broad paintbrush in the beginning, then once you have your outline then it is time to buckle down and really start to analyze strengths and weaknesses. I mean here is pretty much my theory. I haven't read 1 book on strat creation, I don't have a degree from any college, and I certainly couldn't tell you how to solve for the sum of squares. I just have a creative mind, I started to trade with my dad at 16, and I have been developing these things with enough frequency and really it is the only thing I have been doing the last 2 plus years. I don't really have any more friends after shutting them all out and I certainly do not look as good as I did before this venture to have any nagging woman to waste my time. It wouldn't matter anyway because I really don't even have any money haha. But I know what I know, and I am most likely above average for some reason at pattern recognition and I guess after enough trial and error over this whole period of time I have gotten pretty good at this. I think these are the main thing between me and people that are struggling. 1) I have had almost unlimited time without major interruption for over 600+ days. Working everyday and I am not kidding. IMO there is no fucking way possible you can develop these things with either kids, 40 hour work wk bullshit job, or the classic wife that makes you even question your existence anymore. Maybe the once a week Saturday night sex would be worth it, but I am guessing it probably isn't when it would be just as fun to beat it and throw it in her face. 2) I want it more than 99.5% of everyone else. I am not fucking kidding, I have given up everything for this. So it is either this or just realize my true fate, go pick up a Z of kind buds and head on down to Citgo and fill out the application where I will become "one of them". But it is all good because at least I will be well fed on Tacquitos and fountain pop. Maybe even I will get lucky and they will have a grenadine option on the coke dispenser. I could only be so lucky. So again, I probably want it more than everyone else that has bothered to read these opinions of mine. Because when you look like me at 26, turning 27 in a few days, the proverbial window is just about fucking shut.
It appears like you've spent a lot of time and effort in pursuing this endeavour, that said, you've mentioned some of your background and it may be causing you to not see the forest through the trees. Lets step back a bit and take a 10,000 foot view of things. (1) One good strategy does not a profitable operation make. (2) Strategy diversification and portfolio risk management are what pay my bills (as well as of many whom I know). (3) Don't sell yourself short on your math skills (and if they are not up to par, get educated). (4) While math skills do make some difference in analyzing system performance and can help you not step on landmines, by and large it is creativity that always wins out in the end. I'll take creativity over analytical prowess anyday. (5) More trades != better system. (6) Unless you're in a HFT firm, limits and cancels are a headache in of themselves in real time. There is lower hanging fruit. I trade a portfolio of futures 100% automated and 100% intraday. Hold times are 30 mins to 5 hours on average. All market orders, all simple concepts. I've been doing this a while I know some very adept automated traders, here's a rule of thumb: For the ES specifically, $7k PnL on average per year per contract is a good number to hit. This is with 1 tick slippage and 5 RT commissions included. That number is actually tough to hit if one goes back to 1998 and averages out all the PnL yearly... Regardless, the approach you're taking with 1 minute bar trading, limits, cancels, etc etc is a difficult one, unless you've got access to serious technology, clean tick data and a serious staff of programmers. There is much easier fruit to be had out there, and, there is much more money to be made through instrument and strategy diversification and longer holding times. In general, one of the biggest mistakes I see in newbie thinking (this is not directed towards you, you're not a newbie), is the idea that one single instrument strategy that has some ridiculous profit factor and teeny DD's is the ultimate... IMO that's far from the truth. Its the other way around. Lots of mediocre, well managed, diversified strategies working together on a very large diverse portfolio will make one a very cushy paycheck... Just my 2 cents. Mike