Developing "Spartan"

Discussion in 'Journals' started by nooby_mcnoob, Feb 3, 2019.

  1. I've been experimenting with running some intraday models to complement me and one of the problems I consistently have is my tweaking often leaves me stuck with no idea what I changed from one model to the next. This is actually a form of robustness because I delete everything and start again quite regularly when my results go crazy.

    Anyway, I currently have a "series" table which can hold date/observation pairs associated with a series. Considering (ab)using this table to store model history as well. Would allow me to revert, compare and test trained models. However, models are only one part of it. The other part of it is the training dataset (preprocessing, normalization, etc). So perhaps each instrument model would have one or more associated characteristic:

    * Instrument id
    * Model
    * Training data
    * Test data
    * Metrics

    Creating tables isn't expensive or annoying for me, so I suppose I should simply have a Model table. The problem was likely to occur when looking at storing training data. But I just checked that and it looks like it's about 5MB per model according to pd.DataFrame.memory_usage() so I could compress it as I did before with tick data and make it nearly cost-free.

    Not ready to commit to this yet, but just collecting my thoughts.
     
    #351     Feb 22, 2020
  2. Whether you store a logbook of your activities and studies in electronic form or on paper does not really matter much (to me at least). It is the discipline to log each activity which is important. After a while the discipline tends to become less and the log becomes incomplete.
     
    #352     Feb 23, 2020
  3. d08

    d08

    Why not use pyqtgraph for this? It's much much faster and very versatile. I began trying to implement QtChart but it's very cumbersome to use and I ran into silly problems like plotting the datetime axis properly - QtChart will plot all periods, that means weekends and holidays included. I switched to pyqtchart and after a few hurdles, I could achieve exactly what I wanted or very close to it. On top of that it's very fast looks great. It should be the first choice in my opinion.
    Matplotlib is great for static charts but not for anything dynamic or with a lot of data points. I still use it to plot equity and drawdown analysis but that's about it.
     
    #353     Feb 23, 2020
    Axon likes this.
  4. I don't like to rely on discipline, but process. So something like:

    * develop
    * develop
    * develop
    * Hmm I like this, save with comments
    * continue

    Sort of like with source control which is the model I'd want to follow.
     
    #354     Feb 23, 2020
  5. Hmm, I will give this a shot though I use jupyter as the interface. https://stackoverflow.com/questions...aph-plot-to-work-in-ipython-repl-or-jupyter-c
     
    #355     Feb 23, 2020
    d08 likes this.
  6. SteveH

    SteveH

    #356     Feb 23, 2020
  7. Axon

    Axon

    Short answer is I have not found this to be the case. Slightly longer answer is Matplotlib is a tricky library that has been described by some to be almost "charting via side effect" haha. After using it for many years I've figured out all the quirks to make it very fast (I use it for charts that update every 100 milliseconds using less than 10% of CPU though actual real time is certainly possible) and flexible, i.e., there's nothing I ever want to do that I can't accomplish with it. I've tried a few different platforms like d3 and pyqtgraph but never found them to be a compelling alternative beyond being more noob friendly. However, I'm always open to something better so if you could give me some scenarios where pyqtgraph is appreciably faster or more flexible I'm open to it. Also if there's something you've tried to do with matplotlib that had you stymied fill me in and I'll tell you what I think.
     
    Last edited: Feb 23, 2020
    #357     Feb 23, 2020
  8. I would suggest you give plotly a try. It is way more limited than matplotlib but it is also interactive by default.

    I use the Jupyter integration, like this:

    Computationally intensive primarily because I'm too lazy to make it efficient, but results in interactive charts (translate/scale/hover) like this:

    upload_2020-3-13_22-35-38.png
     
    #358     Mar 13, 2020
    Axon likes this.
  9. Love this comment.

    "Why offer just one confusing API, when we can offer two and make it twice as confusing?"

    GAT
     
    #359     Mar 14, 2020
    d08 likes this.
  10. There's a method to the madness... Convenience vs power. Used to bother me too.
     
    #360     Mar 14, 2020