Nooby McNoobs lame-ass code: High performance, highly tuned Arctic Compression on-disk performance reported earlier was maintained. Please, find me something SQLite can't do.
And here is the new performance profile for the same test as earlier. As an aside: this is why I hate software. Most people using Arctic will set up the following: 1. Message queue 2. Broker feed going to message queue 3. Arctic pulling stuff off message queue to put into mongodb Default reliability around 90% Me: SQLite and cron. Default reliability near 100% This is why I get paid well when someone recognizes the skill, but also it's hard to compete against the incessant "high performance" bullshit. I watched the Arctic video and I lol'ed because just by default, my SQLite query performance beat his. My ONLY issue was SQLite <-> Pandas was slow. But muh multiple clients: https://github.com/canonical/dqlite But muh columnar database: see above. I am the software equivalent of Richard Proenneke.
Sorry if I sounded like a dick. I have worked with enough of these bullshit artists. Write thousands of lines of code to do something that can be accomplished in 20.
I'm a big believer in understanding yourself to see how to best reach your goals. Some of us are lucky enough to be well rounded people with decently stable upbringings. For whatever reason, I am not well rounded so in the spirit of Dalio, who made me take my first personality test which opened my eyes, I decided to take another one by the Petersen fellow. According to this, I am basically a big dumb jerk/genius visionary. Interestingly enough, a lot of these align with the Dalio personality test.
Funny, it feels like we're moving in opposite directions. My current system is like yours, sqllite and cron. Now in the new system I'm using Arctic/Mongo, and I was thinking vaguely about using zeroMQ although I keep flip flopping on whether to bother or not (fair to say your post has pushed me back away from the idea). GAT
https://understandmyself.com/ - DM the results if you do it! You should do what you feel comfortable with. The problem with the "SQLite and cron" approach is that they give you too much freedom to do what you want which may or may not be desirable. If you feel like you need structure that is provided externally, so you can think about what you feel is important, then you should feel free to do it. For example, I couldn't justify switching to Arctic if I didn't want to use MongoDB. That to me is an invalid decision-making process. Fortunately, SQLite is the best database in the world and supports the MongoDB use case quite handily with the JSON extension. Edit: so I had no choice but to copy the Arctic model and my system is better for it. That being said, I do use ZeroMQ! As the glue between the rest of my system and the broker. This was the only part of the system I did not like initially because it seemed like things would devolve into spaghetti but over time, it's proven useful. The key "insight" to avoid spaghetti was that any message through 0mq would use a serialized Python object which would implement "executeOnServer" and "executeOnClient" or something like that. This let me wrap IB Gateway behind my own interface which turned out to be useful for simulation. I haven't really had to touch it much since I initially coded it. Here is the "place order" message which could probably be made neater (why would I manually copy all the fields... am I some sort of cretin???) to give you an idea. BTW it is pretty common to flip flop when you feel like someone else is doing a good job on something you felt you didn't do as well. But as you saw here sometimes I steal ideas from others and bring it into the domain in which I am already a master.
Here are the interesting Python libraries, just so I don't forget anything like 0mq next time: sqlalchemy/alembic numpy/pandas ib_insync (at commit aca9ab6c239920dcb70653177d464f36ee60ee9f)/IB Python API pyzmq python-twitter statsmodels/scipy/sklearn - mostly for research though