be aware: back trading in ninjatrader is corrupt https://ninjatrader.com/support/helpGuides/nt8/?discrepancies_real-time_vs_bac.htm
it's a backtest so it doesn't need to be in realtime. dump the data into https://questdb.io/ and write a query. should be simple
Awesome - so much feedback! @earth imperator: yes, exactly. I will check out ORATS. Thank you! @angrytrader: Honestly, I didn't expect it to be that difficult as many trading softwares show you scans like "biggest % winners" and so on. It took me a while to understand that the real time scan for 1 minute bars would be so much more difficult to realize. @ZBZB: interesting - I will check out firstratedata and ORATS! Thank you! @SunTrader: Correct. I simply didn't think I would have to restrict myself @ZBZB: trade ideas was one of the first ones I looked at. It can actually scan in real time the one minute timeframe. The problem with TI is that the backtest is restricted to 64 days so I would mainly use it for the real time trading in case the backtest yields positive results. I might be trading it a little to get some experience but realistically it is rather a strategy for algo trading. Always being ready and fast enough to scalp when the alert goes of in the 1 min timeframe would/will not be very practical. @rb7: Maybe I don't understand the offer correctly. I contacted them to clear this but haven't heard back yet. On the homepage they say: "500 simultaneous symbols can be watched with basic subscription (up to 2500 at a time available for an additional fee" @Mason: do you mean backtesting in general has weaknesses? That I know. If your link was regarding ninjatrader: I did take at look at it but didn't consider it - see below screenshot of software research. @2rosy: Thanks - I'll look into that. @all: in case anybody else is looking for the same kind of software: I have attached my findings to share with everybody. Currently I'm trading with IB. The result of my research so far was that multicharts or tickblaze might be adequate. Seems like the data provider is rather the bottleneck for the backtest as well as the real time scans.
IMO that is exactly what you should want to do. Scanning the whole stock universe (Nasdaq anyway) sounds good but think of it as if you were wanting to find the ideal place on earth to relocate to and set out traveling to every place in the whole wide world. Not very practical at all.
You don't need the whole market as about half of NASDAQ stocks have anemic liquidity, effectively untradable on a 1 minute scale.
You can totally do this. 1-min bars on all exchange traded stocks is not a lot of data. I've tested algos on that for years using a very moderate Dell server running Linux. You can also do 1-sec bars and tic data. It's a bunch of data but entirely doable for a server you will spend less than $5,000 on.
There's one bullet point to stop at. Backtest vs Realtime. Your search for a data feed in realtime will be useful for live trading, but since you want to run a backtest first, you can do it without any problem. You don't need a data feed for a backtest. You can store datasets locally and run your backtest on them, like FreeGoldRush said. Another point that I am also thinking about is the processing time of those 2500 symbols in a minute. Since your time frame to act is one minute, you will need to be sure that your code can search for patterns within those stocks in that minute. As it would happen in realtime. That to me would be a lot more critical than to look for a data feed in the first place. Processing those many stocks can take longer than a minute, easily.
May I ask what timeframe you chose in order to not crash your engine? Backtest the entire market on 1 min timeframe for a period of 1 month or 1 year? How long did it take your software/hardware to produce results?
Your are right. If I can not backtest or trade it in real time it won't work. Both things need to be feasable. Regarding the storing of the dataset: What I learned so far is that it is way more expensive to download the data in order to "own it" and work with. But maybe I just haven't found the right data provider. Anyways as you say it would definitely speed up the process to have the data locally stored. Regarding the speed of the scan: I agree. As earth_imperator said the easiest thing would be to get a snapshot directly from the data provider so my software doesn't have to do the scan. My first indicator is a simple "x% gainers in 1 minute" and that will allready filter out a lot of stocks and reduce the amount of data needed for that 1 min timeframe. In order to solve I have contacted IQFeed but haven't heard back. My thought was that their custom scan service might deliver such a snapshot. Additionally I will have to ask Polygon directly. They seem to be a sponsor here but it looks like they don't partake in the discussion.