Hello, ---- I started a new thread under "Trading Software", but found this to be more appropriate as there is discussion around development. I don't know how to remove the other discussion. ---- I have started looking into trade and time & sale data for equities with mixed results. My problems seem technical in nature and hence sharing what I have and get feedback here. My setup: 8 core CPU, decent GPU, 16GB RAM, tons of SSD storage. Using python to digest incoming stream - using Tradier for data. This basically requires me to open up multiple streams while passing the symbols I am interested in the request. I have a multithreaded system retrieving this data and puts them in influxdb (making sure data with same timestamp also gets written - incrementing timestamp by 1ns). Now, have a Grafana plugin to digest the data and build relationship with data. My questions: 1) Is there a framework to allow me to do this so I can focus on actual data and number crunching? 2) If not, what are the best tools to digest the data on a local machine - I am looking at about 500K to 1.5M lines of records to digest based on my filter and if I scan the entire US equity market
Generally speaking when dealing with large quantities of data you create views of the data in the database and then access those views locally to perform additional manipulation. You may want to look into using grafana to do this.