Need to handle websocket data for 10,000 symbols.

Hi,
here I have add some points which I need to solve for move ahead in development. I am using Polygon.io for get data feed.

Point 1 :
I have 10,000 symbols of my selection. Now I need to get live records of those symbols only. When I am connecting to websocket using channel.* then it's providing me all symbols data without any issue. But when I make query string for fetch live data for 10,000 symbols then it's giving me message for connection close. Also when I make query string for 4500 symbols then it's working fine and websocket run smoothly.
So, how can I fetch live data for 10,000 symbols in websocket ? in a single query string.

Point 2 :
When I call any api for get historical data then it's taking 3 to 5 seconds for response. Which is very high for development. I am working for strategies of live stock market and for that I need to call historical data API for 10,000 symbols on regular basis and if single API call takes 5 seconds then for 10,000 symbols it will take very long time.
So is there any way to make api call response very fast ? my expectation is nearly 500 milliseconds.

so please check this points and if possible then guide me with proper solution so it can help me in development. I am using c# for development.

 

2. Why do you need to make historical data calls fast? Call it once and cache it.

 

I am not using them but generally speaking:
A) Don't request all at once ... Send requests 1000 at a time with 1 second delay in between.
B) if they allow, use multiple sockets where each is responsible for a range (A-L, M-Z)
All initialization needs to be done prior to market open.

 

Sounds like there is a limit.
I would just make an array or vector whatever and compare if the symbol is in that array of 10,000. If it is not then ignore the incoming msg.

 

Please can you guide me that how can I use multiple websocket at a time ?
Right now I am using connection like this.

Code:

 
private void websocket_Opened(object sender, EventArgs e)
{
       createSymbolsQueryString(); // using for get symbols list
       Console.WriteLine("Connected!");
       this.websocket.Send("{\"action\":\"auth\",\"params\":\"123456789123456789\"}");
       this.websocket.Send("{\"action\":\"subscribe\",\"params\":\"" + queryString + "\"}");
       this.websocket.Send("{\"action\":\"subscribe\",\"params\":\"" + queryString1 + "\"}");
       this.websocket.Send("{\"action\":\"subscribe\",\"params\":\"" + queryString2 + "\"}");
}

 

Yes, I have tried for this solution but every time it's checking our list of 10000 symbols for those symbols which we don't need to handle so it's not solving our issue. I have tried with it. Polygon providing more then 25000 symbols so for 15000 symbols are not in list and loop will work for those also.

 

are the 10000 symbols on the same exchange?

 

No, it's from different exchanges like NASDAQ, NYSE,etc.

 

I use Polygon.io, but for realtime quotes, not historical data. During my development I noticed that after about 1000 symbols were being monitored, the websocket would close. So I switched to the NATS interface instead of websocket, and all socket problems disappeared. I do not know if there is a NATS interface for historical data.

 

Are you on Windows or Linux?