Hi, posted similar question in another thread. I am setting up a server to log ticks for a minor number of instruments. (30) Anyone out there have a solution to share? Are there ready-made packages, database-applications designed just for this? Or do I have to hire a programmer to tie apps together and make it run smoothly? Also, how do you handle large files of data? Stats-software have rather conservative limitations. I am looking at calculating spreadsheets with +1'000'000 rows. Thanks
FYI: Spreadsheets are not databases ! They were never designed nor intended to deal with GB of data. The proper way to do this is to use the right tools for the right job: that is let a specialized database server - or if necessary a custom software component working in conjunction with a specialized database server - do the job. The kind of thing you are asking to do is done everyday: you use the spreadsheet/office application as a front end to a faster, more optimized server: the database or other server handles the calculation and simply feeds the results back to the frontend for display or further ssimple analysis on a small returned data set.
Cal: Thanks for taking the time to enlighten me, as you'd guess I'm more accustomed to spreadsheets than 'proper' database handling. Yes, I really want to be able to run queries and display the results. But with daily log-files of transactions, can it be easily combined into a database or do difficulties with integration tend to be great? Anyhow I will have a talk with a programmer I know, and see what he recommends. Since you already got the database set up , this is what I am thinking to do with it, once I get it organized: - Scan for changes in order flow, e.g. Large versus small lots. Research shows that there are discernable changes occuring for example before takeovers; larger amounts of medium sized orders than at other times (the rationale beeing to move large quantities unnoticed). - Plot ratios retail/institutional byuing/selling. Like a commitments of traders index but on tick data, to see if the gap in trading behaviour between small speculators and commercials is wide or narrow. - Look for (historical) price-levels where high-volume players are more active, in order to anticipate spots of large liquidity. Any info of this kind you are extracting from your database? Finding it useful or a waste of time? joe ======== Thanks Cash for good link. Seems a good programme.
excellent! thanks for the link. a good start! we need free/open data software and sources to kick the ar*sses of greedy commercial data providers.