FYI: from now on I'll post follow-ups to this thread only in my new journal at https://www.elitetrader.com/et/threads/myjournal.363333/ See ya there!
If you want to speed up, you may run the correlation computations in Kubernetes with Docker containers. Divide up the work data in suitable blocks/segments and run it simultaneously, how many in parallel that is needed to get meet the time constraint you. Either on-premise at home or on some public cloud vendor. That is how I would do it.
Hi @globalarbtrader I wanted ask your opinion. I currently trade FX and look at the correlation between the absolute value of pair prices. I was thinking that it might be better to measure the correlation of a change in PnL for 1 lot of each pair; A and B. And since you are looking for future relationships, you could take the correlation based on change in PnL and do a simulation to find the likely future range of those correlation values within a given future time, using the worst case value in that range say at sigma 1 or sigma 2, to help decide how much risk to take on a new position.