This website https://www.histdata.com/download-free-forex-data/ allows access to tick/1sec/1min decent quality data for many forex pairs, probably known to some of you. It is possible to download the data month by month and unzip it, yet this is far from effective and takes hours if you need 10 years of data for a few instruments So I have made a little bash script that creates a one time token (needed to have access to data) and uses it to download data and extract csv files, this allows downloading files without manually clicking. THought it might be useful Code: #!/bin/bash for i in gbpchf chfjpy nzdusd nzdjpy usdhkd bcousd jpxusd udxusd xaugbp; do for year in 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020; do for month in 1 2 3 4 5 6 7 8 9 10 11 12; do TK=$(curl https://www.histdata.com/download-free-forex-historical-data/\?/ninjatrader/tick-last-quotes/$i/$year/$month | grep tk | awk '{print $5}' | awk -F '"' '{print $2}' | head -1) && sleep 1; PAIR=$(echo $i | tr a-z A-Z) \ && sleep 1 \ && if [ ${#month} -lt 2 ]; then month_long=0$month ; else month_long=$month; fi\ && curl 'https://www.histdata.com/get.php' \ -H 'Connection: keep-alive' \ -H 'Cache-Control: max-age=0' \ -H 'Upgrade-Insecure-Requests: 1' \ -H 'Origin: https://www.histdata.com' \ -H 'Content-Type: application/x-www-form-urlencoded' \ -H 'User-Agent: ### COPY FROM YOUR BROWSER ###' \ -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \ -H 'Sec-GPC: 1' \ -H 'Sec-Fetch-Site: same-origin' \ -H 'Sec-Fetch-Mode: navigate' \ -H 'Sec-Fetch-User: ?1' \ -H 'Sec-Fetch-Dest: iframe' \ -H 'Referer: https://www.histdata.com/download-free-forex-historical-data/?/ninjatrader/tick-last-quotes/'$i'/'$year'/'$month'' \ -H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \ -H 'Cookie: cookielawinfo-checkbox-non-necessary=yes; cookielawinfo-checkbox-necessary=yes' \ --data-raw 'tk='$TK'&date='$year'&datemonth='$year''$month_long'&platform=NT&timeframe=T_LAST&fxpair='$PAIR'' \ --compressed --output ${i}_${year}_${month_long}.zip \ && unzip ${i}_${year}_${month_long}.zip; done; done; done; rm *.zip && rm *.txt To get User-Agent download any file manually and inspect the website (see screenshot) This is an example for [gbpchf chfjpy nzdusd nzdjpy usdhkd bcousd jpxusd udxusd xaugbp] pairs, last 10 years This URL in Referer is for 1-second quotes, you can change it for any other - just copy-paste from the browser leaving the '$i'/'$year'/'$month'' part Code: I'm not affiliated with hostdata.com in any way, also I suggest paying for access to this data (despite the fact it is unsecured and publically available) This data is also "dirty", but can be usefull
I have no need for this data, but I feel the fact you've done this as a bash script deserves some recognition. GAT