 |
stevegee58
Registered: Feb 2004
Posts: 2402 |
08-25-12 06:12 PM
I import 1 minute data into my own data files, converting it to all the timeframes that interest me. The 1 minute data goes back to 2001.
I toyed with databases I finally realized it added more complexity than it was worth in benefit. My binary data file implementation is simple and very fast.
|
| |
|
Edit/Delete • Quote • Complain |
NetTecture
Registered: Mar 2009
Posts: 1010 |
08-25-12 06:17 PM
My advice: Clear text files.
Why not a database? That is a GREAT storage, but for backtests you want something FAST that is REPEATABLE. We test here always against a specific data set - then some others, then another one, then forward test. but reputability is critical, plus the ability to get the data fast and move it to some external systems (external to the storage system) in order to run tests on an external backtest machine ;)
Which, btw., seems not to have Xeons - not price efficient, I go for high end high overclocked desktop machines ;)
|
| |
|
Edit/Delete • Quote • Complain |
vicirek
Registered: Dec 2011
Posts: 364 |
08-25-12 06:37 PM
Quote from stevegee58:
I import 1 minute data into my own data files, converting it to all the timeframes that interest me. The 1 minute data goes back to 2001.
I toyed with databases I finally realized it added more complexity than it was worth in benefit. My binary data file implementation is simple and very fast.
I do the same. Own dedicated binary file data structure is faster because of simplicity.
|
| |
|
Edit/Delete • Quote • Complain |
calhawk01
Registered: Jun 2009
Posts: 75 |
08-26-12 07:41 PM
Thanks everyone for replies.
To add some color to this request.. i'm obviously new to this so I would really appreciate commends that include references so that I can do this all on my own. I don't have the resources to test out different methods with programmers. I am asking this question on this forum because I want to get the most optimized, cost effective structure down the first time around and then ask a programmers help (btw if anyone is interested in this project PM me).
What methods do these backtesting softwares use during their backtesting? You say MYSQL will be able to recompile my 1 min data to different timeframes on the "fly." Really? So If I uploaded 10 years of 1 min data on MySQL; and then wanted to backtest 5 min time frame, and wanted to test a semi-complex formula.. MYSQL will be able to do this within a few minutes??
I mean another thing I was thinking was.. maybe I should create a library of historical values for all of my indicators and store them on the MYSQL file as well. For example:
date, time, price, indicator 1, indicator 2
So if my buy was when indicator 1 > X and exit is when indicator 2 > X; the backtesting script will easily be able to ''fetch'' the data.
I'm not sure if this would be the most effective/quickest way to do this.. because I have not tested this out for all the data that I have (this can be extremely time consuming because I would have to manually create all the indicators and values in excel for all the stocks data that I have).
All thoughts are appreciated!
|
| |
|
Edit/Delete • Quote • Complain |
PocketChange
Registered: Jul 2008
Posts: 2036 |
08-27-12 03:50 PM
Read up on sql views.
Quote from calhawk01:
Thanks everyone for replies.
To add some color to this request.. i'm obviously new to this so I would really appreciate commends that include references so that I can do this all on my own. I don't have the resources to test out different methods with programmers. I am asking this question on this forum because I want to get the most optimized, cost effective structure down the first time around and then ask a programmers help (btw if anyone is interested in this project PM me).
What methods do these backtesting softwares use during their backtesting? You say MYSQL will be able to recompile my 1 min data to different timeframes on the "fly." Really? So If I uploaded 10 years of 1 min data on MySQL; and then wanted to backtest 5 min time frame, and wanted to test a semi-complex formula.. MYSQL will be able to do this within a few minutes??
I mean another thing I was thinking was.. maybe I should create a library of historical values for all of my indicators and store them on the MYSQL file as well. For example:
date, time, price, indicator 1, indicator 2
So if my buy was when indicator 1 > X and exit is when indicator 2 > X; the backtesting script will easily be able to ''fetch'' the data.
I'm not sure if this would be the most effective/quickest way to do this.. because I have not tested this out for all the data that I have (this can be extremely time consuming because I would have to manually create all the indicators and values in excel for all the stocks data that I have).
All thoughts are appreciated!
|
| |
|
Edit/Delete • Quote • Complain |
| Receive
an email whenever a new post is added to this thread by subscribing
to it. |
|
|
|
|
