I would like to create an automated trading system that is applicable to every liquid exchange traded product with listed options. To be honest with you, I am sick and tired of trying to grind it out every day on just one commodity or product, mostly via point-and-click trading. It's like you become married to the instrument you are watching and are a slave to it. A universal trading strategy is the goal, and I believe is feasible with the right amount and correct method of data mining.
"Having trouble predicting and timing the direction of volatility and the shape and slope of the skew intraday as well as long term." This is a problem for most but not all. There is a paper - "The Volatility and Efficiency Effects of the Emergence of Futures and Options Markets on Equity Indexes". It was revealed to a very small audience in 1989. Somehow the author established contact with Hull Trading Company - an options market maker in Chicago that knew that bots were the wave of the future. Hull Trading Company was bought out by Goldman Sachs in 1999. The store goes that there is a back door to the methodology and it has be exploited several times with discretion. It is like being able to look at the GS book.
I don't think that's what the OP was asking for but even if he did I would also seriously doubt that any model from the 80s still makes sense in the modern volatility markets. I am not a subscriber to the "great theories of everything" but let's assume you found it. Is the final goal to be a highly profitable OMM or play in a deeper pool somehow? Just from personal observations, each asset has its own volatility dynamics that are driven by the specific flows and asset-specific quirks. For example, you can't expect rates, oil and equity index volatilities act ever remotely similar. Even at a more granular level, volatility of specific single name equities is very much driven by the specific little things - from larger details like it's index membership to smaller things like its popularity among the call overwriters.
Of course each instrument has its own nuances and market behavior. The goal is for the trading program to discover and recognize each individual commodity or stock options implied volatility tendencies over a long period of time. The more around-the-clock data you can accumulate, the more accurate and higher probability position trading strategies you can implement. I may have misspoken by stating a universal strategy, but rather a universal methodology via a proprietary trading model that can be applied to the whole spectrum of exchange traded products. The program is ultimately predicting paper/order flow or anticipated order flow by the primary option market-makers of each instrument. Like any other asset class order flow usually determines the direction of prices and/or implied volatility of options. The key is to find and properly weigh in all the factors that cause order flow to behave the way it does, and then decipher if they are predictable enough to warrant a trading strategy that will exploit it.
Not sure if Hull Trading or Timber Hill (now Interactive Brokers) executed with obvious great success with what I am trying to do, but they both made hundreds of millions trading a massive options book with a "big picture" approach. I used to trade with both entities all the time during my time on the floor, as they would often sell/lift the "hard" bid/offer from traders like myself, who always wondered "how the hell did they make so much money?" Obviously looking back, our bid/asks must have been too high/low relative to other option markets they were monitoring in their portfolio.
I will never say anymore about this except - sitting on a lot of updated code is some very simplistic origins. Think - War Games. Not only the stock market but in many, many, different places.
I don't know about "different places", but I would be very, very surprised if any of the currently operating OMM shops have any production code that's older than 10 years. Alpha and infrastructure in that space (both OMM and AMM in general) decay very quickly and you have to keep up with the Joneses from across the isle in your colo center.
In likelihood they were using you to manage their unbalanced risk. Their book earned on the uncorrelated uninformed retail flow by competing on tens of thousands of 10 lots. Citadel supposively had two hundred people working in its EMM group (this is from 15 years ago).