Just in case you're asking for your own son's benefit, there is no age at which teaching him this stuff is inappropriate. My son still remembers the calculus lesson I gave him at age 3 (subdivide a square infinitely). Sons want to be like their fathers and any age you sit him down in your lap and explain to him what you're doing with passion, he will latch on to it. I have been teaching both my kids stuff I do for many years, they loved it at every age. Except maybe the girl when she became a teenager. But after a while, she was back on the wagon. Anyway, no matter how hard I try to dissuade mine from being a nerd, he is a super duper nerd. So I lean into it. We recently did an experiment on momentum while driving, for example, which made him excited about being able to take physics. The main problem, if any, would be that mothers sometimes interfere because they don't understand that a son loving what dad does is part of loving mom, since mom loves dad. Women are weird like that. Be patient if she interferes. And in case you were just asking for curiosity, no harm done, I hope!
Compressing information I currently track tick data for about 100 instruments at any given time. I find this information extremely valuable, and use it in my trading. The problem is that the SQLite database is getting pretty large now. I have been using 50 tick bars for some instruments and some multiple of 50 tick bars for other instruments, so I thought of forming bars out of each set of 50 ticks and archiving the tick data as opposed to outright deleting it. This lets my database remain light, but I can retain the information. To be clear, I don't "just" use 50 ticks/bar, I've got some magic on top of it to discard noise. Is this a sign that I should be moving to some cloud thingy? I don't think so, but I can understand if others might see it that way. Haven't decided on an approach yet, but without a doubt, I need to have a process for archiving the tick data. I would be willing to dump this on some cloud thingy, so long as I could use it in Jupyter seamlessly for research.
Is the database getting so large that it slows down your software? If not, then it is only a question of whether you want more storage on your own computer, or that you want to use storage somewhere else (e.g. cloud). Adding an extra hard disk (or SSD) to your computer is easy but requires a one-time investment. Offloading data to storage is also easy, but might result in a monthly bill. Also the upload and download delay times could hamper your software.
Good questions! No, the database size is not impacting performance at all. Queries are instantaneous. The "problem", if you can call it that is as follows: I actually run the system in two different places, on my local desktop and the cloud. I actually don't use the cloud (lol). However, the cloud is much more reliable and occasionally, I have data issues locally that I need to resolve by downloading the data from the cloud instance. I think I need to think this through more carefully. And ask myself some intelligent questions: Why am I not using the cloud system? Answer 1: it is way more efficient for me to make modifications locally, test them for a week, then push them to the cloud, which I never use anyway. Answer 2: deploying to the cloud is manual and painful. Answer 3: the cloud machine is slower than my desktop machine even though I pay $150/month for it. Is data collection the only reason you care about the cloud system? Yes. Otherwise I don't care. When is the last time you needed the data from the cloud system? Actually, it's been at least a month. The only reason I had data issues was that I used to reboot during the day if I wanted to play Fortnite, but since I found TeamFortress and Enemy Territory on Linux, I'm satisfied with my violence quota. So, should I be retiring the cloud system? I feel like the answer here is yes. Hmm...
I see that you use the word "cloud" slightly different than I do. You seem to have a computer instance in the cloud with computing power and storage. Such that you can run software from it (e.g. automated trading system). I only use "cloud storage", Dropbox in my case, to replicate files over multiple computers and to have access to settings files and log files while I'm not at home. I use a computer at home to run my trading system. The difference: I don't have to pay $150/month, as you seem to do, but had to buy an extra computer. The data I have at Dropbox is less than 2 GB, so I use a free account.
I like the simplicity of your setup. I currently have about 100 GB of data w/ Dropbox. I'm not too worried about $150 on the remote machine, but whether it is solving a problem for me. The question still remains whether the desktop setup is stable enough. In the past, I've had problems with machines dying when they've been on 24/7 which is really the main reason I chose to use someone else's infrastructure as they can avoid world ending hardware failures better than I can. However, the local machines that died weren't really high quality parts. I have very high quality parts in the machines I build today. Yes, I think we get rid of the remote machine assuming no serious problems by end of this year. Thanks for the chat buddy
The pleasure is all mine. I learn a lot from your posts. By the way: the computer I use to run my trading automated is not your regular PC or laptop. It is actually an Intel NUC. It uses very little power so hooking it up to a small size UPS makes it run for several hours in case of the occasional electricity cut. It is connected to my home network so I can connect to it from my desktop computer. Some time ago I found an app which enables me to also connect to it from my iPad in case I am away. That is a bit cumbersome though, so I only use that when absolutely necessary.
Typically quant data is suited to a columnar layout not SQL rows. Also, these days you can get extreme storage along the columns. Some good options for storing quant data are https://github.com/man-group/arctic or use pyarrow parquet files. I use pyarrow parquet files and roll my own flat file database - very very fast!