Thank you @d08 for the suggestion of the Panda's library. My software however, is coded in Java. So I had to create some code myself to take care of these things, for as far as they are relevant to my trading setup. I'm not aware of a Java library similar to the library you suggest.
While not ideal, you might build a type of bridge to communicate between the two. I've found trading dates to be quite complicated and building from scratch is a lot of work. The Japanese system is even more complex and with many more holidays. I've created profiles for each market, so when I tell it to use "US_STK", it already knows to follow ET as timezone, USD as currency and the NYSE market calendar. It simplifies quite a bit as this info is repeatedly queried.
Hi all. Quick update. Nothing too noteworthy has happened in the past few weeks because I've been mainly hardening my code. Anyone who works in software will know that it's easy to knock up a proof of concept, not so straightforward to get it through the environments and meet all the requirements. Not that I have those environments or a passive-aggressive product owner with a troubled home life, but I do want my stuff to run on time and not crash in a hail of java errors, so it's been mostly debugging. And putting in new indicators. As mentioned, a strategy in my system is basically a set of parameters saying which indicators you want to use and what their parameters are - so it's basically a row of 30 or so integers. With my system (which I call BLW - a system needs a name) I can test that strategy on the 20 years of S&P, FTSE100 and FTSE250 data I have frigged (survivorship-aware) and I can unleash it as a live strategy. I have just done that with the first one I am reasonably confident with. It's some kind of trend-following strategy running against the FTSE100. I don't want to say it's a world beater but it has turned out just under 9% annualised in backtest (which sounds quite paltry, but the average for the FTSE100 in that time was 0.4%, so it'll do for me as a starter). I say "some kind of" trend-following strategy because it isn't the purer trend-follower I first envisaged. Backtests have led the parameters too far away from that original premise for me to call it anything but "some kind of", but it works and I understand why. To be clear, at this stage I am talking about a system that generates buy and sell signals every morning and sends them to me, and I must manually execute and tell the system that I have executed. Automating the execution seems quite straightforward but I am not touching that until I am confident. I'm unleashing this first proper strategy on Monday. Wish me luck! I have said from the start that this journey is about using my software skills (I am an data architect) as a way to learn about trading. I am a terrible discretionary trader - I am impulsive, prone to panic and I am sometimes convinced that I am getting divine trading signals from cloud formations or the reg plate of a car that goes by. BUT I understand rules, and if I can codify them I can beat my crazy weaknesses.
Update time. So this system is live now in its MVP incarnation, meaning I get the signals and told what to do but I have to do it - no automated ordering yet. I have two strategies running, both trend-following stocks - one on the FTSE100, other on the AIM100. TBH I discovered the AIM100 only recently and it works much better than the FTSE100 because all all the volatlity and more potential for momentum. I don't have as much historical AIM100 data as my FTSE dataset (which is reasonably survivorship-aware) but the backtest results were way better. Now to the reality of actually running this system. Each morning I get an email saying what to buy, how much of it, at what price (target and max) and a relative stoploss. Signals are ranked by liquidity, although I am thinking of flicking a switch and using a ranking system based more on performance. It's up to me to then place the trades. I am using a pretty basic trading account - only market orders are available and no stoploss. I am now seeing the value in these things so I'll be stumping up a bit extra to get those features. I'm kind of wary about using the stoploss because I've been warned how it can work against you, but I am so bad at doing it myself. I just can be relied upon to be decisive enough (more on that coming). Also the whole point of this is so I can do a few clicks first thing in the morning and then get on with my day, not check the prices every hour or so. So I started getting a few trades in. Some tech stocks like AVV.L and AVST.L. Others like BRBY.L, SGE.L and PSON.L. All of those have been stopped out - only Sage with a (tiny) profit. The others tool more of a beating than they should have - because I just dither on the SELL button and watch the price, giving it chance after chance to recover! Luckily the FTSE100 strategy also presented Pearson a week ago, which I just sold yesterday for a 15% profit. (If you dig into the Gamestop story you will see Pearson mentioned as one of the Euro equivalents. I had no idea of this when I bought or sold, it just got stopped out, and for once I was happy to click SELL!) Anyway, I'm happy that my short term trend-following strategy picked up that short term phenomenon. Not so good was that getting another PSON.L signal this morning and going back in. It looked good for about 10 minutes, then turned. Looks like that party is over. So right now I'm happy that my system is doing what I ask it to, and that I am asking it to do the right things, just not so cool that the market is shit. I do have regime filters but clearly they are not kicking in yet. Which reminds me - I need to get a better AIM100 regime filter. Currently I am using the FTSE100 index MA for that, which is very crude because those two indexes behave differently, but I can find an ETF that tracks AIM100 that I can use as a proxy index. I could do it myself, aggregating the individual AIM100 stocks after refresh each morning, but that would be ignoring the individual stock weightings. Next steps, other than the AIM100 regime filter I want to get some kind of GUI for this so I don't rely on emails (and going under the hood with Mysql workbench). Someone suggested a slackbot - will look into it. Also I need to start working out which broker to use for the next level of automation, IBKR seems the obvious choice but there's also IG and a couple of other with APIs (I am in the UK). Happy to take suggestions from anyone. Like I keep saying, I can do the tech but I am a trading newbie! Also happy to answer questions.
Nice update; quite some progress here. This is a well-known psychological effect. And one where automation can protect you from yourself.
Quick update: the system has been running for a couple of weeks and it is slightly profitable. And I am enjoying it! This is the key question for me - do I like this? Do I like researching, designing and building this system? And do I like the fact that I am channelling actual funds into this system? Yes on all counts. I am a fairly obsessive person - once I get an interest I try to follow it to the ends of the earth. In the past this has led me to many dead ends, but also life-changing things in a positive way. I'm hoping this will be the latter. So yes, I am enjoying it. I build the system, I unleash it on the market and I execute the things it tells me to. But part of this pleasure makes me uncomfortable. Many years ago I went through a phase of betting on horses. This was before the advent of online gambling, when you had to walk into a seedy bookmakers to place your bet. And I was really too young to be doing it a lot of the time. But I loved placing those bets! And I recognise some of the same pleasure when I manually enter one of these trades. You select your ticker (horse name), place your bet and then eyes on the screen, rubbing your hands together - come on you beauty! WARNING WARNING WARNING! YOU ARE USING TRADING AS A SEMI-RESPECTABLE WAY OF GAMBLING! I want to get away from this. There are a lot of other aspects that I am loving, and I want to focus on those... and preferably avoid being the old guy outside the bookies, begging for 20p for a cup of tea (but secretly accumulating enough 20p's to place a punt on Old Rosie in the 2:45 at Newmarket). And I think the answer it the automated trading part. This was my goal when I set out, and I'm feeling the need to accelerate to that point. It seemed inconcievable a couple of months ago that I could be considering this - letting a robot manage my money - but I really am. Yes, I will continue to gamb... I mean trade manually for the time being (resisting the urge to drink whisky out of the bottle while I do), but I need to start exploring brokers, APIs and order execution. ORDER EXECUTION. This is one area that I have not been able to get comfortable with, vis a vis automated trading. Or manual trading even! If for some weird reason you are still reading, do you know of any threads here (or anywhere) where this is discussed? I am talking limit orders, bid ask spreads, timing, angling for the dips etc - all from an algorithmic perspective. There is plenty of material about strategies and signals and backtesting, not so much on this aspect of the machinery. Maybe because it is considered mundane? Not to me. To me it is like haggling with a Turkish market trader over a fake Ronaldo soccer shirt for your kid (who is watching judgementally) - if you offer market price, the Turkish trader will disdain you. But you don't want to offend him by going too low! Oh god, this is the kind of dysfunctional anxiety the English are cursed with! And this is the reason I want to automate everything - so I can walk in the woods with my dogs and avoid all interaction with the world at large. Apart from Elite Trader, obviously.
There is tons of stuff written out there about order execution. Especially because it is such an important aspect. You just need to be able to find it. I won't bother you with a long list. Start by reading this https://qoppac.blogspot.com/2014/10/the-worlds-simplest-execution-algorithim.html and see what follow-up questions this raises.
Thanks for that Hobby - TBH that is exactly what I was looking for, and puts into words and cleverness the vague ideas and concerns I had. Matter of fact I was looking at Robert Carver's blog a couple of days ago about something else, and didn't think to search for that.
He has several more blog entries about his trade execution system. But those entries are all focused on the software implementation. I knew that he had this blog entry about the "reasoning behind the system", but had to dig a bit before I found it again. Over six years old meanwhile...
Here's an update - although I have been developing my automated system I have also taken time away from it to do discretionary trading. Premise being that I don't want to be the quant guy who approaches trading solely through algorithms - I'd rather understand the strategies and rules that a good discretionary trader does. And what I have learned is that I am very, VERY far from knowing the first thing about being a good discretionary trader! Couple of cases in point - 1. Yesterday I pulled the trigger on a lovely breakout setup I found. Only I didn't wait for the actual breakout. Despite telling myself this is what I was going to do, I went ahead and bought the thing in anticipation of it. It looked so good! And it seemed to be slightly breaking out kind of, for a second there. But nothing happened. It sagged a bit for a day. Luckily for me it broke out today, but the point is still true - I told myself to wait for the breakout, and I didn't. I got excited and failed to control my emotions. 2. Today I bought NIO. For fuck's fucking sake! You know that time when you were in the snowy mountains and you innocently whispered something to your friend and it triggered an avalanche and everyone died? That seems to have happened here. All I did was a boring old "buy the dip". It was only on about a 1% dip for the day at the time. You have probably seen the rest. Aaargh! Lesson? This was an indisciplined trade. I had no right executing it, because I hadn't found the setup via any of my usual means. I had just heard a bit about NIO and knew it was a trending kind of stock and wanted some. What harm can come from this?? The market has extracted some expensive tuition fees today. Onto the automated system - I have been upgrading it to be capable of: Looking for more nuanced setups. E.g. long term momentum but shorter term consolidation - both via exponential regression. Daily industry momentum scans - reporting to me the top tens by my rules (again exponential regression... or just MAs if I so feel) and also filtering scans through that if I want. Ability to quantify annual and quarterly EPS and rank/filter by that. A pretty basic way of factoring in fundamentals but it's something. What else? Oh yeah, figuring out how to get daily refreshes of all my market data - for free! All this data is out there and you can tap it for zilch if you work out how. And I have done DQ checks and the quality is fine. I realise this makes me a cheapskate, but once I work out a proper edge for myself I will be less mean. Promise. Reporting. Piping my trade data into some dimensions and facts allowing me to slice and dice it all in Power BI. I would show you a screenshot of my headline dashboard breaking down performance by strategy... but I'd be too embarrassed. But mainly just learning. E.g. that it's all about what the market wants to give, not being clever enough to get what I want from it come what may. And that I am sloppy and indisciplined as a discretionary trader. And that I know so, so little about the real market forces. And that I'm loving it!