Hello, I have started looking into trade and time & sale data for equities with mixed results. My problems seem technical in nature and hence sharing what I have and get feedback here. My setup: 8 core CPU, decent GPU, 16GB RAM, tons of SSD storage. Using python to digest incoming stream - using Tradier for data. This basically requires me to open up multiple streams while passing the symbols I am interested in the request. I have a multithreaded system retrieving this data and puts them in influxdb (making sure data with same timestamp also gets written - incrementing timestamp by 1ns). Now, have a Grafana plugin to digest the data and build relationship with data. My questions: 1) Is there a framework to allow me to do this so I can focus on actual data and number crunching? 2) If not, what are the best tools to digest the data on a local machine - I am looking at about 500K to 1.5M lines of records to digest based on my filter and if I scan the entire US equity market
One gzipped csv file for each day. Potentially hd5 not gzip if IO matters more. Then load into pandas, one core for one day of data. Scientific pipeline it.