oops, logged into an old account.. well here is what I have so far, it is only 2 levels deep, I'll add the other 8 levels on Monday, have a look, see what you think, it's in CSV format http://manageproperly.com/level2.csv
My only suggestion would be to use a different symbol to analyze. A stock with less volume where patterns would be easier to identify. A single stock that is in play. FAS is highly efficient and order flow doesn't dictate its movement. The movement of its underlying assets drives price movement. Co-located HFT servers run that stock. Speed and latency are required to trade FAS. Bids and Offers flash and back off thousands of times a second in FAS. Cant trust the order flow. What ever patterns you do find I doubt you would be able trade off them due to latency. A stock that is less efficient with less than 2-3 million avg daily volume is my suggestion. A stock that is in-play that day. Where you can see large orders that can move the stock and you can trust the orders. If the stock is not in play. meaning no news or catalyst to drive it, than it will simply be HFT volume. Trading based on correlation to its sector and or market. Waste of time and untradable . Take data from several stocks in play. First hour of trading 9:30 - 10:30 when the real volume trades. Record the trading tape as well. See if patterns exist across those symbols. Meaning trading patterns in stock xyz also are seen in stock abc. Then apply those trade ideas to potential future stocks in play. There is no need to analyze 10 price levels deep from the L2. Only the top five if that. Bids and offers deep in the level 2 below 4-5 price levels will not have any factor in price movement. This may help crunch the data faster. Good luck.
Nice job getting the data! mastertrader456 has a few good points. I'd also be interested in seeing the data. If you ever get a zip file link you can share, PM or e-mail me. Btw, in that CSV i didn't see anything resembling "market maker names" like you said in your original post. What do you mean by that? On some symbols the levels 5+ can indicate the MMs are pulling out if they expect the price to not hit those levels, I think. I may be wrong, but shouldn't this be valid in some futures markets with the limitations on signalling they impose?
Disclaimer** This is very preliminary and pretty basic tests I'm running on a VERY small data sample, wish I had more... If what I'm trying is "useless", "stupid", or "naive" please keep it to yourself or start your own damn thread! Premise of this test assumes: - buying at the ask - selling at the highest bid within 100 seconds of purchase - I realize that you will never be able to sell at the highest bid consistently, the reason for testing in this fashion is to test if buying under certain, specific conditions "could" possibly lead to a higher profit by what the best case scenarios are under different conditions - The data for this test is from the etf FAS from 9:30 - 12:00 (forget which day, sorry...) - The purchase price will always be the current best ask price. - The selling price will always be the highest bid price within the time frame of 1 second after buying up until 100 seconds after buying. - I realize that there are flaws in this type of test but I am looking for obvious differences in the results. - I ran 3 tests that bought at random and sold at the best bid within 100 seconds - The results of those 3 tests were - .10, .10, .09 - So I am assuming that if you were to buy at a random time and sell at the best possible bid within 100 seconds, your average profit would be 10 cents. Results of this test so far: Out of about 10 different scenarios I've tried so far, the average best profit made was 10 freaking cents... right in line with the buying at random tests, what a shocker... Now I really wish I had about 30 days worth of data, but there has been some hopeful results on the last test I ran, which brought the average best profit up by 50% to 15 cents, seems significant, of course it doesn't tell you when to sell, only when to buy and concludes that if you buy at that time for the indicated price, then you have a better chance than random to make a profit, even if you sell early and only make a couple of pennies. Anyways, here are the parameters that produced the significantly better than random results... take it for what you will, open to suggestions, I know this is a very basic test and much more could be done with this amount of data, but just got back from out of town and did this in about an hour.. so... if this upsets your sensibilities, sorry for ruining your day and upsetting your dinner, etc.. etc... In PHP if ($lastprice == $askprice1 && $spread <= .01 && $asksizetwo >= 800 && $bidsizetwo <= 400){ echo $futureprofit; } In Plain English and explaining the variables If the last price is equal to the best ask price, the spread is equal to or less than a penny, the sum of the ask size in the first position and the second position is greater than or equal to 800, and finally, the sum of the bid size in 1st and 2nd position is less than or equal to 400 then show what the difference of the best possible bid (within 100 seconds) and the current ask price is, whew...
"Best," determined afterward, at the end of 100 seconds? How would you know this in real time? Maybe I'm missing something.