For what the OP is trying to do, sma crossovers. Then yeah Tradestation will do just fine. I'm not trading professionally, but am making money while keeping my day job. Last few months been earning more from trading then my job I think for the sort of question asked in this thread, it is directly assumed that the OP is probably retail like me. I don't trade anything right now that generate non base currency pl, and most retail traders don't. I don't do options much, except to hedge. Currently execute it manually and use optionvue to model greeks. For any retail affordable datafeed there is no accurate tick based quoting. Well, theres the tick chart option in multicharts. anything fancier gotta export the whole thing to R or mathlab and do your own visualization. theres the extended backtesting mode with bar magnifier, that allows me to do tick by tick backtesting with bid ask spread. It lets me model fills more accurately especially on markets where spreads are volatile and see which tick my entry landed on. Yes, besides MC .net that is sometimes linked to my external R or mathlab module I have 16 HP blade 460 servers running 24/7 crunching out alphas. Thats pretty decent setup for a retail guy like me don't you think ? Anyways most of the post in this forum is asking technical stuff that is achievable and doable by retail anyway. So mister HFT Punter, just how big is your AUM anyway ?
* Why would you need a day job if you were remotely profitable on the trading side? Maybe it is worth listening to those who make a career and living out of what they do best. That applies to all sort of skills and industries. * Lol, even trading 2-3 currency pairs would expose you to non-base currency (or at least divergent non-base currency) p&l. Just because you do not trade such and hence do not care about it does not mean that the platform you mentioned are sufficient in handling non-base currencies. Most everyone who trades for a career most likely has had p&l exposure in different currencies. * So your platforms fail on the options side as well. Hence the point I made that your mentioned platform suit the absolutely most basic needs of retail punters, those that you describe you are yourself. Again, I wished you well for what you do but it remains basic and non-professional. More like a hobby than a serious endeavor. * No tick based data feeds that are relatively affordable? Sure if you are hackling and having sleepless nights over whether to spend 30 usd over a close-to-free charting platform then you most likely will have issues to invest a few hundred dollars per month in a quality data feed. * You wanna export tick based data to R and crunch numbers? Why dont you do that with a simple set of 30 or so securities over a year worth of data and around 100 million data points and report back when you are done. R is about the worst tool to do such. Please talk about things you truly know about rather than making slippery assumptions. * You mean random ticks between bars? Lol. Have fun with that and your assumption a random number generator replicates reality. If not random then why would you not just run your backtest over tick based data but called it some fancy term "bar magnifier"??? You have "16hp blade 460 servers crunching out alphas"??? Lol not sure what it is and what you are trying to say but using MultiCharts and talking about server hardware makes me chuckle. It just does not sound right. Listen, I am not trying to be confrontational but your talk sounds very BS-y. Sure, what OP looks to achieve can be done on very simple hardware and software but he also asked about architectural questions and I gave the simple advice not to start with SQL which is precisely when you went onto your retail-punter-rant and attacked me how I would dare to say SQL is not sufficient. SQL is NOT GREAT for time series analysis, whether you deal with 10 rows or 10 billion rows, period. You and also I do not have any idea what OP wants to do next. Can your platforms for example dynamically run screeners and you submit a script dynamically (without stopping your platform) which the screener runs over and applies to existing or streaming data? No, Multicharts cannot nor can Tradestation, so for heaven's sake, give me a little credit for knowing what I am talking about. You can hate me and the way I deliver the message but the message nonetheless is a message based on a lot of expertise in this area.
@volpunter will the SQL column store index not decrease the number of joins in the same way that a full on column oriented DB like monetDB or kdb+ would? My understanding of the main issue with using SQL (the kx website has a few articles on this subject) was that the relational model suffers mainly because of the multiple joins, so when I read this https://msdn.microsoft.com/en-us/library/gg492088.aspx, it seemed to imply that issue has been largely mitigated by this feature. I totally agree that a column oriented approach is the way to go here just from an algorithmic standpoint, so could you briefly explain the difference between this column index and the result of using a column oriented db? And yeah, I see what you're saying about teafiles. The only real advantage a teafile confers over binary is that the structure of the file will be provided by the file, similar to HDF5. So, for our intents and purposes, the format of each asset's file will likely be identical, so having the file tell us it's struct before we read it isn't going to be necessary in this case. I still think its interesting though, might play around with it a bit when I have time. I think it's a simple and clean idea that could be really useful. Some of the people in industry on other forums seem to have taken an interest in it as well.
I do not know enough about how SQL (particularly MSSQL) implements its columnstore indexes to comment with insight. But I do know that even a 10x speedup over a well indexed rdbms will not even come close to the performance of pure columnar based databases. So I do not bother learning more about the SQL approach. If you know which tool is best in this case why don't you then just use such tool? Nobody asks you to learn Q and handle kdb. You can just pick up HDM5 or other approaches and get a columnar db setup in less than 2 hours. The schema/header of a teafile is actually very similar to how I have implemented my own binary data store. I keep information in the header such as symbology, what type of data is contained, how large each datapackage is (which is essential knowledge for a binary reader), how many data points are included (not necessary as you can deduce from the size of the binary file and the size of each data point after you subtract the size of the header), and other information. Everything else is basically elementary operations on binary data. The basic idea and underlying schema of a teafile is good imho but how they then implemented the storage and retrieval of such data is not optimized at all.
WOW, I started with Super Charts which was first attempt to do backtesting before Tradestation came out, then many versions of Tradestation where you first bought the software before they came out with month subscriptions. I did all my testing back then until certain limitations were stopping me from testing more than one contract of tick data. MultiCharts certainly a very good software to have and get started on and go on to more advanced tools or making their own platforms. One stays with very long trading off weeklies, I think advanced software just a waste most of the time depending on timeframe. But as I get older I see K.I.S.S. to actually better with good skills on price patterns than complex systems.
See the attached File for more in depth explanation on mssql column index implementatoin I make 200 grand a year on my day job and can grow silicon right next door with access to all the toys I could ever want. Why would I leave when they allow me such flexible work time ? Most people use R and Mathlab for the handy Matrix functions and other calculus aproximation, just take a look at the library, and call it directly from your own C program. problem solved. It is the actual tick data, most signal are based on m1> OHLC data, but for precise entry and slippage info the bid/ask tick by tick info is provided in the bar magnifier function. I've developed my own pattern finding tool in java and some C called with JNI. its a GA and uses SVM + NN as the final filter condition. I can set the treedepth, tournament size, kernel type and node number via parameter. I unleash them on m1> OHLC data, then if they find any intresting patterns implement that in MC .net to verify the results and get a more visual feel and more detailed performance report. MC 9 does, its called portofolio trader. Hm I suspect you do work in the industry but in the IT support side, as in the guy that the traders calls asking why can't they open redtube during lunchbreaks or why the office printer is jamming again.
What they mean to say is that if you can't infer how to do it from the provided documentation then its probably futile to try and accomplish that in an acceptable timeframe better pay us to do it instead.
* I looked at the file and that is where I took the stated speedup from (10x). It is by far not enough to even spend another second thinking to use SQL for columnar data. Let alone the fact that you have to pay hundreds if not thousands of dollars for a full MSSQL license if you want to store the amount of data a full-fledged time series data base requires. MySql and other SQL versions do not really offer such columnar indexing features (I think Oracle does but not 100% sure) and are a real pain to setup in a .Net environment (which I target, and no I won't go into any debate of pros and cons of this choice). Above all, writing a simplistic binary data store would take a proficient programmer not more than a few days. * Great if you make 200 grand a year then you should honestly stick to what you do best and not talk about things you understand unarguably very little about. (fyi, 200/yr is the salary a senior associate-junior VP makes in ibanking, not to talk down your salary but by any measures it is nothing special at all in the financial industry and certainly does not impress me at all). * Sure they use R for matrix computations, but R and Matlab stink big time when it comes to running code that is path dependent which most backtests are. It does not matter how you access R it simply is way too slow for code iteration (which is required for backtests). * Sorry but regarding your "BarMagnifier" example it is nonsense: Here is why: If you ran a system over a full tick-based data feed then you do just that, period. Why do you need a "bar magnifier" or other toys? You can build whatever algorithms please you but it does not change the fact that a platform that is capable of handling tick based data runs over every singe tick, allows submission of orders at every tick and fills your order at whatever tick your simulated orderbook fills your order. * And for your exercise you need some hardware "blades"? Why? Sounds more like you try to show off. If you have efficient code then you can run most screeners on commodity hardware. I can stream several million ticks a second on a commodity quad core machine, which includes query and loading of binary database data, deserialization to ticks/quotes, sorting by timestamp in case you request multiple time series, streaming of those to the strategy core engine, running strategy algos, submitting orders when each strategy generates orders, filling them in a simulated order book, generating statistics and reports as well as visualizations. What do you need a server or blade for? ;-) * You confuse things here. "Portfolio trader" simply means that it runs backtests over multiple strategies and generates performance statistics across the whole book of strategies rather than running each strategy one after another. It has nothing to do with dynamic script execution. What I mean is that you have a live data analytics engine or live trading engine and inject a new analytics or strategy script and the engine will execute and run the same without interruption. * You are wrong about your last assumption as well. Someone who takes care of jamming printers would not school you about even basic issues such as we discussed. We have not even gone in depth yet. We have argued before and equally you have refused to learn from anyone. You got all agitated just because their/my delivery of message did not sit right with you. If you had ever worked on the trading side in banks or funds you would have grown a thicker skin and had a keener attitude to learning rather than trying to shoot the messenger just because he did not kiss your ego. I do not intend to spend more time debating issues with you after you displayed a very limited insight into the topic under discussion (remember, you stated "SQL is GRRREAATT for time series data storage and retrieval"). Not saying I cannot learn from you but I do not see much I can learn from you in this particular area. Good luck with your endeavor but please be careful what you recommend to newbies because you mislead others with your erroneous statements.
Hmm, unless you've developed a new OS and filesystem I've never heard about then this I find hard to believe. Anyone using the standard linux kernel fread() will not accomplish this. Guys Listening to volpunter makes you think that retail automated trading is hopeless, well it's not, I'm proof of it
WRONG http://www.multicharts.com/trading-software/index.php/Portfolio_Trader See, it even have currency conversion. Heheheheheh Ready made tools available for retail are starting to make serious gains. It'll be a shame not to take full advantage of them.