thank you for your interesting reply.....well I have 50 currency pairs to analyse daily - 4 time frames each - each time frame shoudl be analyzed using a few different large excel files (5 or so) - around 300 mbs on average....meaning on average the 300 mbs excel file would need to be recalculated 5*4*50=1000 times including getting the data into it and outputting to Matlab that would need to be recalculated 1000 times as well per day.... do you think this process can be speeded up if I break each of the 4 time frames into different virtual machines and place the analysis pipeline into them?.....you think a 4 core E3 1275 V2 might be enough for it???
Why not go virtual and host it somewhere? Someone else will give you the 99.999% up time and you won't have to worry about it.
The i7-3930k is a great choice. My data mining machine has a 3930k overclocked to 4.1 GHZ paired with 32 GB of high performance ram, and a solid state drive. Since purchasing this machine 4 months ago, it has run at 90-100% CPU non-stop data mining for strategies. I have not had to reboot it a single time. I do not have any concerns about the long term stability of this processor vs the xeon. Go cheaper on the 3930 vs the xeon and spend your money on high performance RAM and a SSD drive is a must. Especially if your like me going through hundreds of gigs of historical data. The read time on the SSD makes a big difference.
@dima777 If you would like to consider using Amazon EC2 for batch processing let me know. With Amazon EC2 you can fire up multiple machines for pennies on the hour and have the results pushed back to you.
thank you very much for the detailed reply.......can you please give me the specs of the high performance ram you mentioned as well as the sdd disk you are using...I am really balancing between the xeon and i7....) I am thinkign of runnign 4 seperate vurtual machine each crunching its own subset of data.....what do you thin of this setup? do you think XEON E3 1275 V2 can be acceptabel for this purpose? ti has similar benchmark as i73930 even if 4 cores...
thanks...yes I have been thinking of Amazon clouds as well...I did not fully understand their operation....do I create a virtual image of the pipeline windows and upload it to them for running? how about the security of the EXCEL model files inside the image?
I could build you a custom Excel processor. As far as security goes it can be as secure as you want it to be. Meaning if you desire, anything you consider proprietary or private would never be written to disk, only transferred to the VM when its needed and stored in memory.
Sounds like you could do the entire process from within matlab (excel not necessary) and dramatically speed things up with the parallel computing toolbox from within matlab (enabling you to use parfor loops etc). having your data on a SQL server (on a local SSD drive or array of SSDs) instead of from within excel would be a big improvement for you as well. And matlab is great for querying SQL... [you could do SQL express 2012 free edition even] I'd worry more about disk/etc than processor here....
Yes I'll also add that if you consider moving away from Excel that you might look into R and RStudio. They're my preferred tool for analyzing trading results saved in csv format. RStudio has a web interface option will allows you to easily run it from a Linux server.
I have 2600K, 2700K and 3930K CPUs, all running 24/7 for months. For stability you need a good motherboard and PSU, as well as memory. So far I've found that in my case the 2700K is outperforming the 3930K. This is because of the kinds of instructions that the compiled software is using are more suitable for the 2700K. There are differenes in what compiler was used and what kind of processing it, how parallel etc, so it's not such a straight answer as to what is the best. Also, if you use 32bit applications on a 64bit system and the software is using many Windows API calls, the 64bit system will be slower than if you installed 32bit Windows on it. As far as stability is concerned, you can create a 99.9999% stable top notch rig with a 2700K for ~$1200 if you know where to shop and you build your own PC. Don't worry about stability of the CPU, you should worry about how reliable your software is to automate the usage of Excel which imo will be your biggest problem. It is slow, and you can probably get 20 times the performance with some custom bit of C# software. Just my 2c.