lol . . . I agree. Yes, they are software engineers but some of the ones I seem to deal with think they are masters of their own universes. I'm pretty eccentric but these guys make me look normal.
Very good . . . that is why, individually, we need to strive to secure more perfect tools. Money management is always crucial but the better the tools the less crucial the management needs to be babyset.
greaterreturn, I'm glad you mentioned simulated data. I've been thinking about that one for awhile. Do you know of a source for simulated tick data sets? Do you develop your own at the tick-bar level? I think I could create a very good tick-bar data set by rearranging my existing algs but they use up a lot of computing power and I don't have the resources to take on this sort of project. -bluelou
Well, I built my own system for this. I haven't found anywhere that has market simulation data. That means entire data with bid/ask, DOM, and trades. Well, at least not anything approaching affordable for an individual trader. So I collect my own for 12 pairs of Forex, unfortunately, it only goes back several weeks. The 5 year data I downloaded has only bid prices. I find that I test out basic ideas on the 5 year data and then run it on the real market simulation data. There's a number of cool trading rules based on the spread width, DOM size and relationship, volume, etc, that you can use to filter when you allow trades to drastically reduce risk and increase profitability. I will just have to patient as my data grows. If someone gets TickZOOM (at tickzoom.org when it's released next Sunday) and records a lot of data, I bet they could sell it later if their data provider allows it. There'll always be people wanting good simulation data. Wayne
greaterreturn, I'm not following you on the "real market simulation data"? Where does the simulated data come from? How is it produced and at what frequencies is it available? -Lou
Oh. MB trading provides a feed with quotes, trades, and Level II data. Quotes, you know, right? Bid/ask? And trades are actual buy/sell, price, and size of trade. Level II data actually notifies you of EVERY change to limit orders in the market. They're most often used by "market makers". TickZOOM collects that keeps it updated and generate DOM (Depth of Market) data. DOM tells what size of orders are standing to limit by or sell and different prices. The levels of DOM refer to how far away the prices are from the current price. First Level of DOM is actually synonymous with the bid/ask spread. The Second level and so are are just more pairs of bid/ask at further away from the current price. At various occasions the market makers will bail out of the market, that's an advance signal of something big happening. Hey, good time for us little guys to get out too, right? And you can monitor the volume and DOM size to decide when it's safe to trade. It's amazing when you chart all this stuff how the market "comes alive". It's because you can actually visualize the "people" involved. When traders talk about the "screen time" you need, it usually refers to a wholistic learning of Price/Volume/T&S/and DOM. That gives you a complete picture of what's happening and you can start to "see" the people involved and their emotions. I could talk for hours about this. And I'm not even an expert yet by ANY stretch. If you understand what i'm saying it's because I'm not an expert yet. It seems the real experts are almost incomprehensible. Wayne
Wayne, I think there's been a misunderstanding. I was referring to a simulated data set for a given contract such as the ES contract, not running a simulation/backtest. By a simulated data set I meant a data set that replicates the statistical properties of the ES contract. With simulated data we could run backtests on a variety of random realizations of a facsimile of the ES contract without falling into the path dependencies that can occur when optimizing a trading model to a single data set. I'm assuming that you do not have this kind of data? Is this correct? -Lou
There is an additional point greaterreturn didn't mention if I may chime in. I have a system similar to what he is building and he is right on track. He is steadfast about recording the tick data and backtesting using the same "data pipe" used in real trading. It is essential that you "record" the data you would otherwise be using for realtime trading and have the ability to "playback" the same data while backtesting. If you depend on anyone else's data it will almost certainly be different than the data your system would "see" realtime. I go so far as to put markers in my data stream to tell where my local Internet connection momentarily goes down (or my router hiccuped or whatever) so I playback the data exactly as it happened realtime. If you run his system just using the ability to collect data then in just a few days you will have enough to begin testing those kinds of trading days. I can almost guarantee you will be busy full time with each new day that comes in and deal with each new surprise it brings. Wayne says he will have the ability to create any kind of bar data from tick data so that should not be a problem - maybe he can verify that.
hey stop posting your resume..... pure and simple.... the day of the quants is, well over leveraged and detrending into idosyncratic noise....
Absolutely right! Way to go. And cool idea about the "markers". It would be nice to see that on the screen as the reason for a gap, I usually use range bars so those gaps don't really show. Sometimes, I do wonder if a 15 second gap is due to the feed, my server, the network or what. But the ones that occur right when the FED makes an announcement or otherwise are usually due to everyone collectively catching their breath before their next trade. At first, I disagreed with this but upon re-reading. I see you're DEAD ON. I thought you to mean a few days was enough to build a strategy. But what you mean is that it's enough to spend a lot of time play back and toying with it. That's why I built the playback tool to set any acceleration multiplier. I actually would like to make play back more sophisticated so that I can mark areas on the screen to speed through and then go slooooowly though a news announcement, etc. It's amazing how the market can be in doldrums, going nowhere and then EXPLODE in a frenzy of activity. It's VERY educational to study those changes on a play back tool where you can witness the changes in the buy/sell sides of the DOM the trades coming in, the volume expansion/contraction, and price movement. Unbelievable stuff. I'm glad you understand the feeling of epiphany. Sincerely, Wayne