I don't know about 1 in 1000. To implement a neural network or genetic algorithm one needs a reasonable level of understanding (unless it's precanned in some software package). I think feature engineering is obviously very important to effectively implement machine learning algorithms. I like messing around with AI stuff a lot--even though im just learning--and I've been coming up with what I think are some pretty cool results.
I'm not going to let this turn ugly on me, but let me just say politely that I'd appreciate if you at least acknowledged that I may know what I'm doing. You don't know who I am or what my aptitude is just like I don't know you or yours, so let's focus on the initial question instead of immediately belittling results you haven't actually seen.
I said that because you said "even though im just learning". It took me 10 days to answer you because I was busy trading. It seems you have no trading experience. Yet you get emotional when someone tries to warn you about possible mistakes. Good luck, you will need it.
Have you gotten anywhere with building this thing? Being a software developer, the best advice I can give is to start small and iterate quickly. Prepare to throw the first iteration (or 20) away. I got interested in algorithmic trading last week, so I whipped up a quick system this weekend. I used C# with SQL Server since I already have them installed, and I just wanted to test out a simple idea. Turned out the idea was really horrible, but at least I found out with just a few hours of coding. I implemented my algorithm slightly differently. Instead of eventing on everything that happened, I chose to go on a schedule because that's the idea I had. I threw all my data in SQL, and just asked the algorithm every "second" what it wanted to do. The algorithm would then go grab what it needed to make a decision from SQL. To speed this up, I implemented some caching so that I am not repeating the same queries millions of times.
I'm not sure if I'm understanding your approach. Are you testing a discrete time system where a new second constitutes an event, or are you pulling ticks from SQL each second? Both of them are valid approaches, but it depends on your objective, I suppose. Your definitely have a solid approach if your algorithm operates on a discrete time interval such as 1 second. I've been using C# for a lot more stuff lately since my lack of knowledge of it probably got me put in a company's the 'never' pile in an interview recently haha. C# is perfect for this task, and its integration with SQL server makes your solution a great set up for this task. C# seems to be a lot better suited to the task of event driven backtesting than a MATLAB or R because of the necessity of a loop instead of a vector, but my C++ approaches werent much better because I just really don't have the time or expertise yet to manage a large C++ project. Not that I 'couldn't' so much as there's no reason too really and once I start prototyping actual trading strategies, I would prefer to do so in C# to C++. I'm poor and a masochist, so I can't afford SQL server and have instead been testing a few different databases. I think theres a C# oriented open source Backtesting tool called TradeLink available, so that might be a good place to look at how they've set theirs up. I don't have professional development experience at all, so I'm not sure how much help I can offer you here, but if you have any specific questions, im interested.
I'm doing a mix of both. I have ticks in the database, and my events are seconds. From what I've seen so far, other trading frameworks give you a stream of data, and it is the responsibility of your algorithm to store the data however you like (plus a few charting features like bars). I'm flipping that around a little bit. The framework throws all the ticks in the database. The contents of the events are just the current time. The algorithm now has the option of pulling all the historical data it needs from the database if it likes without having to do the work of storing it. I mainly went this route since I didn't want to create a bunch of windowing buffers to keep all of my indicators up to date. I can write a SQL query much faster than I can create a sliding window. I'm fairly sure that this is a very slow since I am incurring a huge overhead just talking to SQL. Plus, SQL is giving me a hard time with some of the queries I want to write. I am interested in kdb+, especially q. Any idea how much it costs for the 64-bit version? SQL Server comes in a developer edition, which is relatively cheap at $60 (http://www.microsoftstore.com/store...er-2014-Developer-Edition/productID.298540400). MySQL or Postgres may be good, too. I wanted to use Postgres for its nice JSON support, but didn't like how queries are single threaded. That really hurts my analysis queries. I need help in learning the trading domain. I've just been reading through investopedia. Any other places I can look for more information about what to look for in the data?
im using kdb+\q as well. I built a similar database in MySQL a while ago and the performance was pretty much unacceptable, although I did nothing to really tune it. kdb+ has been really fast for me so far, but the development is a lot slower just because q is so different from SQL, it's a vector oriented language and it's pretty interesting and well suited to my tasks for it, but for someone just learning like myself it's hard to use things that don't have big online communities, they do have a Google group which is pretty good though. It's 25k for the cheapest 64 bit version I believe, I just use the 32. I realize youre a professional, but obviously if you're having problems with your queries which you are reusing, you should build a class to create queries for you...you already knew that though. I'm also testing another column oriented dbms, monetDB at the suggestion of some folks on here. It's my understanding that column oriented is the way to go for tick data. Many people also recommend just writing and read binary files. There are tons of sources for trading ideas and statistical techniques. Finding stuff that actually works is obviously the challenge. If you're brand new to this you probably know only a little less about trading than I do, but one thing that's really popular in quantitative trading is looking at relationships between timeseries instead of just single timeseries. Stationarity is the basis for a lot of the econometric techniques that apply to trading, so that's good to understand. There's all sorts of different stuff out there though, you can get a lot more in depth than investopedia just surfing random blogs, forums, research papers, interviews, free PDFs etc etc. that's where pretty much the entirety of my trading knowledge came from.
By the way @John Tseng , 'make one small one, then iterate 20+ times til I have a decent one' is the best piece of advice I've received thus far on this thread. It echos what I've heard from several other professionals.
This start small and iterate idea has become quite a large movement (Agile method). I think it plays well with "premature optimization is the root of all evil". I like your idea on TradeLink, though. It would be good to learn from what others have already done.