nowhere did the OP express his desire or interest in systematic trading. You may have jumped the gun here. There are tons of trading related jobs where programming comes in handy and many do not even tangent systematic trading. ...and I feel you sightly overemphasize SQL/RDBMS. Most all trading systems can do away without. In fact the most optimal way would be to just store the data as a blob and then use Linq-type technology (every Java coder's dream but hope is on the way...) to filter,search,sort -> faster and a lot less hassle than setting up an SQL db. I highly oppose anything SQL like and have never ever needed to use it to solve even demanding and complex problems. It is generally for those who are not allowed to test and use new technology or those who are afraid of trying new things.
Not sure I agree with that. Seriously. Yes, the trading system itself does not use SQL - but any analysis on larger data areas (accounting data, backtests) would be seriously challenged without a proper database system. My trading front end (currently rewritten) will use SQL locally to track all trades and executions, so I can easily grab them in a standard way with standard tools. I have yet to find a non-joke or non-game scenario (i.e. a serious application that is not a game - and yes, games are serious applications, if you look at the size of the teams etc.) that does not use a database below. Once you enter enterprise space / business space, pretty much every app does databases. Especially if you do not want to do trading, databases are a 100% must. And ieven in trading - man, if I would not use a RDBMS for analysing the stuff we do it would cost me man years to come up with the same functionality.
using RDBMS for long term data storage...ok fair. But you say you work on optimizing the speed of your strategy profiling and testing platform and then you want to feed all your result data through this big huge bottleneck called SQL. If it works for you, great. But most quant teams at hedge funds or investment banks do not, I can tell you that first-hand. They store such data in flat files and/or in memoryDBs in order to further profile such result data in platforms like Matlab or R. Of course you can wait an eternity to load and properly parse your RDBMS data tables into R. Again, if it works for you great, but most professional quant teams seem to disagree.
Seriously? I Have 20 computers hitting that SQL database and dumping backtest results into it and so far it is not breaking a sweat. The trick to that is not being an idiot and reading the documentation of the database in question, as well as choosing one that helps. Could be not idiotic programming combined with not a cheap small VM as SQL Server that in combination makes sure it not only keeps up, our analysis are super easy to look at (thanks to a web portal) and in general available within seconds. For example when a backtest end dumps the - sometimes tens of thousands of trades - it does not use SQL, as our database has a special API for bulk data uploads It also helps that my backbone is buld using 10g networks - while the individual machines (still) are on 1g, the database server as well as the market data archive are on 10g (which makes sense - my paltry hexacore workstation pulls around 600mbit during a backtest). And contrary to Matlab our backtests involve not only the same codepath as real trading later, but actually run always in a full market replay mode Not sure where you get the "slow" part from. Most professioanl quant teams, btw., are very weak on the programming side. Many come from a math background and would never get hired as developer for their development skills (outside quant).
I increasingly get the impression that you love to throw expensive hardware at problems to compensate for an either unwillingness or lack of ability to implement smart software solutions. I never understood your here on ET well-communicated solution of preferring to run 10 or 20 servers with some crazy cooling requirements but at the same time you run your code 19xx style synchronous, single-threaded. Lets think of a simple example. Your crazy machine factory pumps out strategy results. Now, are they immediately available for analysis? Hardly at all. You described yourself that the data is processed in batches and hence takes time to be available on the db server. So, you consider it efficient to work on a strategy, to potentially having to optimize parameters but then having to wait each time for results before being able to get back to the strategy and re-run? I beg to differ. Next up, lets say you wanna quickly calculate some key stats, average spreads, number trades, avg pnl per trade, whatever for date xx-xx-xxxx between 9am and 12noon. How are you gonna do that? Of course you also want a visual representation. How do you get this data into R within seconds, how do you chart it? It sounds like your solution takes rather minutes than seconds just to get the R datasets set up. Dude, we should stop talking if now the problems lies with "poor programming" at professional quant teams. Many banks have highly compensated and highly skilled programmers who are dedicated to implement solutions for quant trading teams. Why everyone who disagrees with you or runs other solutions is automatically disqualified? You run your SQL and you are happy, great, something I said a lot earlier. I only shared my experience and knowledge that professional quant teams at hedge funds and banks do not peruse SQL DBs much aside for static data in their data analysis and strategy profiling quest. They must be doing something right because I know of quite a number teams that are long-term profitable so they must be doing something right. My whole point was that you overemphasized SQL and RDBMS, its hardly a requirement at all to get the feet wet on the quant trading or even general trading side. In fact I have never asked nor would I ever ask a programmer about his SQL knowledge. Its irrelevant.
The difference between me and you: I know what I do. It is generally accepted single threaded code is faster. Plus, backtests / Optimizations are well suited to isolate into individual steps. There is no need for each step to be multi threaded if I can run multiple steps - single threaded - parallel and get rid of all the expensive synchronization. The code was - written by someone else - once multi threaded. Spent seirous time syncing. Now it is single threaded and a LOT faster, and more plannable, and the machines just start multiple jobs in parallel. I throw hardware at the problem where appropirate. You seem to be a quant - throw around keyworkds, incapable of programming and arrogant as hell. Try learning some IT first. And waiting is nice - at the end, yes, the data is available the moment a backtest is finished. Which takes aaaaages. In computer time. Less than a minute for a full market replay. Now tell me in Mathlab you get the results before they are finished. Hm. That would be something. Maybe instead of pretending to be an IT guyy you should ask for the nobel price in physics, yo just invented the time machine. And see, here is the point. First, most of our analysis is standardized. Not standardized as in your case (is there some software to do it, please help, i am an idiot), but standardized as: if we ask the same question for any strat, then it goes onto our analysis portal. Otherwise some minor SQL is needed - very rarely. Then, where you must rerun a test in mathlab, we do not. We just go to the existing data of the old run. We keep our runs quite a long time if we assume they are worth it (obviously most are - for once they may simple be bad strategies, second they may be technical runs, to test the programming). I likely have more data available within a second from tests going back to the beginning of the syystem than you have created in the last year. Are you american? This level of ignorance points to it. American poiitician? Seriously, I do IT for 20 years now. THe WORST programming and infrastructure I ahve seen is in the investemnt side of banks and in the trading business. Flat out like that. Hacks throw together by people that are hired by traders for being smart - while in reality they would not bet a job outside the quant area as juniors. Tons of Mathlab hacks, tons of Excel hacks. I rarely have seen more incompetence than in the trading side. The worst case was a moron in my last job - he got some cofee on his fast in the meeting from me - that proposed to publish market data in XML Format in the trading network. The bandwidth requirement of 20 gigabite for every workstation was an implementation detail for him. Sadly - given that it is impossible to wire a whole building with infiniband - the implementation detail was reality. No, man, you need to maybe get a job outside of trading for a year, in a nice little IT company that values quality. Maybe aviation, maybe something around health data. Not something where you throw together some hodge podge code dudes like you think nice, then bakrupt your company due to lack of structure (as happened to the shining example of smart assses like you - Knight Capital - which obviously had all the money to hire smaaaaaaaart people, just forgot that ignorance and arrogance warely mix. You are the perfect example of a Dunner Kruger syndome - so incompetent you do not even see that.
1) I never said I want a professional trading position. I don't. 2) I never said two introductory courses makes me a proficient programmer. I know it doesn't. Obviously, it's just a start. If you guys would read what I write, you would not need to assume anything. I'm considering two different degrees, the one in fluid mechanics being the most math intensive and with most programming by far. If there's value there for me as a independent trader, it would make another case for choosing that degree instead of the other one. Thanks.
You are an incredibly arrogant (that itself is not a problem) AND thick guy: 1) Almost every profiling AND execution platform peruses multi-threaded paradigms. Every Finite State Machine implementation runs multi-threads, every Complex Event Processing implementation runs multi-threaded. How can you claim single threaded code is faster? It does not even make the slightest sense what you are saying: Unless you fan out very limited data to multi-threaded workers, on a large-enough data scale the process speedup is almost on the order of numberOfMultithreadedWorkers. Every first year CIS major understands that. 2) Analysis is NEVER standardized. Let's see your time series under investigation turns out to be co-integrated. Now what? You prepared for that as well? What if you want to run a different statistical analysis and need to query the data in a way you have not prepared for. A single linq-one-line-rewrite takes care of everything. In your case you need to make changes on the server side, you need to potentially write new stored procs, you need to adjust your ORM, LOL. How much more? If you have no idea what I am talking about here in terms of having to be flexible in terms of queries run then I am fully convinced that you yourself are the IT monkey whose jobs is to carry the server machines to his basement. I would then hardly believe you had the slightest understanding on how to develop tradable and profitable strategies (in fact I already believe that you know very little to nothing about how to profile time series data, you seem to be a full-blooded IT person). 3) You also need to re-run strategies if you want to adjust parameter settings or adjust strategy code. I am not talking about how to archive result data, that I mentioned in my first post you can do in whatever way you want, including SQL RDBMS. The point of mine is rapid prototyping and profiling of strategies. To your own admission you only run strategies over 2-3 dozen assets: Needing the computational power you run around showing off only tells me that you severely lack algorithm programming skills. With your machine farm you can easily run 100x more complex algorithms and strategy tests IF the underlying software was coded up in the right way. 4) Oh American? All Americans are arrogant. My friend, you have a huge mouth but display time and again that you cannot let other opinions stand as long as they make sense. Running single threaded and in turn having to buy half of HP's inventory of servers only tells me that you seem to lack advanced programming skills. Of course everyone who uses new, but proven technologies must be an arrogant American. Nice testimony to your intellect. 5) You are right about general IT departments at banks. However, you can possibly not judge special IT teams that are purely dedicated to successful prop groups and that have almost unlimited budgets (as long as it supports the PnL side). Why I say you cannot judge is because you have never set foot into such groups and teams. They are physically divided from the general IT teams and often on completely different floors, together with the quants and traders of the prop group. They run on completely segregated hardware, software implementation, feed handlers. Its literally a hedge fund within investment banks (and yes, some still exist and they still make a lot of money). So, I respect your 20 years on the IT side (which in my opinion rather is a disadvantage because you are too married to outdated technologies as you publicly display here with your totally BIZARRE fear of everything multithreaded (no its not slower!!!)) but PLEASE do not talk about things you CLEARLY have no knowledge of, which is on what IT budgets and what skill set such prop groups operate. 6) You are seriously a MORON. You are deriding the industry where IT personnel is probably paid the highest salaries (and I want to zero in only on the absolute hotshots and brightest, not every IT boy). The reason they are paid 500k-1m per year (yes those are salaries for top programmers at GS and Co. , and even Google or FB does not pay that much to its best programmers) is because the requirements are extremely high to get things right that despite the fact that every now and then a bug sneaks in. The highest paid guy for Interactive Broker each single year is in effect the head of technology, who spearheaded the IT platform. Everyone who has ever used the platform or its APIs knows that not everything is rosy in the land of TWS and IB but yet the platform improves each year and have most recently even switched to No-SQL solutions on their global IT systems (thats what I heard from a guy who works at IB in Hong Kong). Cannot all be that bad right? Dude, even the worst coder of the Knight core team most likely has more programming capabilities than you and me combined. Shit happens and yes sometimes that shit costs a lot, up to a bankrupt company (though a company of such stature that goes bankrupt because of a 200million USD error already stood on shaky legs). AGAIN: Trading or Quant trading ? NO SQL REQUIRED, NO RDBMS required!!! Simple as that. Run your funny server farm the way you want, all single threaded, all synchronous, your choice. But not everyone who does it another way is less capable than you. I feel sorry for your employees especially those who work on trading strategies. I had a boss similar to you before, he knew very little about algorithmic trading yet felt he had to micro manage those who proved for a long time they know what they are doing. Such boss IS TO BE AVOIDED!!!
so you chose those degrees to be an independent trader? I am even more confused now. A very steep price to pay for such endeavor ;-) Or trading is not really related to your choice of degree? Then why are you asking on this forum? You stop making sense...
If you put your mind to it and make an effort, could you be willing to imagine a scenario where someone has a full-time job in for example engineering and trades his own account on the side? Could it be possible that the math and programming taught in the engineering course could be transferrable to market knowledge and execution of trades? Could it be possible that someone does not want a career in trading, but would still be interested in trading his own account? The collective stupidity and arrogance consistently expressed on this forum never ceases to amaze me.