In Alpaca i used to have an API call to get the Nasdaq tickers with IB API do not have them and if I call a ticker from the NASDAQ site that is made for options/Warrant or is unlisted i get an error that block all the batch i am loading, which is really frustrating because need to implement resume strategies, parsing the dataframes in python and manually get the daily list. Do you have better recommendation to automate the process so that can easily automate the delta. https://www.elitetrader.com/et/threads/best-nasdaq-nyse-ticker-list-that-is-ib-api-friendly.377540/
To help automate symbol retrieval, you could try something like: Code: curl --show-error --insecure --silent --location --max-redirs 10 --max-time 30 'ftp://ftp.nasdaqtrader.com/symboldirectory/nasdaqtraded.txt' > nasdaqtraded.csv You would need to parse out undesired fields and symbols. For example symbols for warrants have more than 4 characters and end in W or .W See also https://www.nasdaqtrader.com/trader.aspx?id=symboldirdefs#nasdaq https://www.nasdaqtrader.com/conten...taproducts/nasdaqfifthcharactersuffixlist.pdf https://curl.se/docs/manpage.html
If you mean Nasdaq 100, S&P 500, DOW that should be easy to scrape from wikipedia with python. Chatgpt can write that kind of script usually pretty good too: https://chat.openai.com/share/dafb2fcd-0d06-425b-9c3c-746717274a24
thank you I already implemented in the past this solution scrape+yahoo following a tutorial from youtube(i guess) but wanted something more complex
I made it, pigs gonna get slaughtered! Did not know about the ftp, thank you. Will look next days in future for delta updates, although maybe are present also AMEX data(which i could easily subscribe) Also I did not think tickers had between 1 and 4 characters LOL, would have been much easier, i ended up implementing business logic checking with regular expressions warrant, Warrant, Warrants in the name/description field. Boosted a lot my confidence being able to do it, still I think python is ugly, but implemented my custom conventions to architecture some clean code. Most challenging task has been checking for consistency when I stopped the connection and make retry afterwards to not incur in data limitations. Worked on concurrency, batches and nice workarounds. I stopped candles in *.CSV, but i see a postgree SQL DB with some satellite star strategy as the way to move further to craft my custom indicators, should be better than NOSQL for my plans Looking forward to work my ideas,strategies, filters, backtests, and add options and fundamentals data and possibly update my m1 mac pro to compete with fellow Jim and Ken.
Anybody know a list that shows all the currently available options expiration dates, incl. LEAPS, for all listed US tickers w/ options? Something like: Code: tickerABC,expDate1 tickerABC,expDate2 tickerABC,expDate3... ... tickerXYZ,expDate1 tickerXYZ,expDate2 tickerXYZ,expDate3... Update: found it! One can extract it from this csv file (delimiter is the pipe sign '|'): ftp://ftp.nasdaqtrader.com/symboldirectory/options.txt I hope it contains all listed US tickers, this not tested yet.
I have scraped a dataframe before of the Russell 3000 tickers and then had a column of true/false if that ticker is part of a major index. Then it is trivial to create some functions to return whatever index you want. From doing this though I realize I don't really care about the bottom 2000 tickers and just look at IWM for that. Then I do care about the Russell 1000 and what ticker is part of what major index. More interesting is what is part of the Russell 1000 but not in the S&P 500. I just don't see the point of SQL here. A more robust solution to a problem you don't really have. I just think it is hard to beat a query on dataframes with a datetime index if the analysis is going to end up in a dataframe anyway. I haven't worked with IB API though. Anything I have used has been trivial to get all symbols but I could see all symbols on IB being a bit much to say the least. Such a huge percentage of that though is going to be illiquid instruments that you will never bother with. Based on the thread though I am still not sure what you are looking for exactly.
good moment to take care for IWV, a boolean or a 1-0 equivalent is the way to go for databases or whatever is your upstream source of data I do not know how much you are familiar with algorithms and data structures, every use case depends, arrays, dataframes, list, push/pop, graphs, SQL,no SQL, LinkedList. Is all about big O a dataframe vs a database are two different kind of beasts. Let's say you have ohlcv and then you want 30 indicators, and fundamental data, and you need to operate so called joins to merge two or more tables and want also to store performance, would you still more comfortable with datasets? I would not be so comfortable if you need to establish relationships, if you need proven ways for querying and analysis on data aggregations, scalability if you have REALLY large volumes of data, optimization mechanisms, consistency of the data, atomicity, integrity. If you need simple operations, and one is not able or do not see profitable to optimize SQL queries then storing CSV files can be a better solution, especially in terms of storage space(as you will avoid metadata) There is also the streaming option, but for a retail use case, no HFT, does not make any sense imo.