Sorry, 50 bytes per tick is horrible. Using my LZMA based market data codec, 11.000.000 ticks will consume 25 MByte (~2.3 bytes per tick, incl. 2 timestamps -exch and local RTC-, volume and exch special code). Data organization for backup purposes is a major issue - I have 5 years of tick data for 6000 instruments and can tell you.
Sounds very interesting, I would love to hear more about the LZMA codec? Could you share some knowledge?
You can also use z-lib (http://www.zlib.net/) which is free and gives you also very good results compressing ticks. I use it in my app for some months now and I am happy with it. Daniel
Hey, there's a new thread called "TickZOOM Details" which has some similar questions about the 11 million under 40 seconds stats. It's running on a Dell Inspiron 530 which is Quad Core or Pentiums but only a single Core (single CPU) was used to run this benchmark. So, of course, it would go faster on a Xeon. What we're aiming to do with TickZOOM is remove the software as the bottleneck so you can always throw better and faster hardware at it. Wayne
Every form of database has been tried. They're all way too slow. The ticks are written in binary format and read almost directly into memory. I always say: "Do the math. When you multiple 11,000,000 times anything it becomes a very long time." Databases are "fast" but even the fastest can take at least a few milliseconds to return a row. Well 11,000,000 times 1 millisecond is 11,000 seconds. So the I/O must take only nanoseconds per tick. Wayne
That's a great estimate. But it's that large at the moment only because every tick currently stores bid/ask/last/size/timestamp/symbol/4 levels of bid/asks The new format has eliminated the symbol since it's redundant and now eliminated DOM if don't have that data.
Thanks Danial, for the tip. I researched and tried all the fastest algorithms for compression. The fastest ones, of course, are still generic and lack the advantage of knowing the nature of the data in advance. Anyway, TZ has it's own compression borrowing ideas from the best ones. It takes only a few more seconds in addition to the time for the 11000000 ticks and compresses almost as good as the best. Wayne
This is an update to correct earlier information in this thread. Multi-Symbol Support As of right now, TickZOOM supports multiple symbols as well as portfolio trading. That has been well tested and regression tested. Marketing Consultant Progress Also, a business development and marketing expert has been help with the marketing and communication plan. That's progressing well but still not perfected yet. Real Reason for the "Selection Process" The #1 item which came from the marketing consultant as needing correction from this thread is the nature of the "selection process". Apparently, I previously came across as "judging" potential members as to their worthiness. My apologies. The reality is that we're judging our own ability to provide excellent service. More specifically, the rate of adding new members must be carefully controlled to avoid getting overloaded with support to the point that nobody gets good service which would harm everyone. New Member Openings Now So, for example, right now we're able to take on 3 new members. Still, everyone who joins the list will probably eventually get accepted. Other advantages of the Waiting List And getting on the waiting list (link below) will get you connected to receive news about updates to the software, releases, and most important: updates on the next opportunity to join. Waiting List In my signature you can see the waiting list to signup if you want a FREE trial which lasts 2 weeks. Thanks for your patience if you need to wait till the next opening. We endeavor to follow a first come, first served policy for the most part. Sincerely, Wayne
Let's discuss any further aspects on the thread, "TickZOOM Details". Here's the link: http://www.elitetrader.com/vb/showthread.php?s=&threadid=167891 Sincerely, Wayne
Hey, sorry to bring this thread up again. But I am forced to prove how blatantly false this is about calling me a "shill". If you search all my old posts when using the chaostheory account, you will find that many of them (7) I signed with my own name, Wayne, as this search link shows: http://www.elitetrader.com/vb/searc...=3066018&sortby=lastpost&sortorder=descending Plus, one person asked who built the tickzoom DB and I told them it was myself at this link: http://www.elitetrader.com/vb/showthread.php?s=&postid=2279273#post2279273 The facts are that I switched from using greaterreturn account to chaostheory and then to tickzoom. Here's the proof that is true. The last post for greater return was on 01-29-09 as this search link shows. http://www.elitetrader.com/vb/searc...=3066010&sortby=lastpost&sortorder=descending And you can see that I began using chaostheory on 01-27-09: http://www.elitetrader.com/vb/searc...by=lastpost&sortorder=descending&pagenumber=3 So there was a couple days of overlap only because I accidentally logged in as greaterreturn and posted without realizing it. Similarly, my last post with chaostheory was on 5-22-09 as this search shows: http://www.elitetrader.com/vb/searc...=3066012&sortby=lastpost&sortorder=descending But my first post as tickzoom was on 5-22-09 when I switched to using tickzoom: Under no circumstances was I ever pretending to be other than myself when posting as chaostheory. I recognize now that my switching between 3 accounts could confuse people and will avoid that in the future. But that's hardly acting as a shill which is a serious and very dishonest crime. On the contrary. Everything in all my posts is true. Like, I am a user of tickzoom myself as well as the author. It really is very fast with tick data. Unfortunately, this thread with the lie about being a shill gets to the top of google results when anyone searches on the tickzoom name. So, I've taken the following steps: 1. Contacted bluelou by phone and email to give him a chance to correct the original post to remove the reference to shill. Shill is a crime that has neither been proven nor have I been convicted of that. As such, it consitutes libelous defamation to write that about anyone. I understand that wasn't his intent, of course. So has one week to fix it and said he'll consult legal advice tomorrow. 2. I've contacted by email and PMs to elitetrador this evening and will call tomorrow by phone about having the post edited. 3. If you, the reader, reply to this post, I plan to continue to re post this same message after every reply so it will always be the last post. That way if someone visits from google they hopefully will jump to the last post and get the facts. Sincerely, Wayne