A situation where vol is up but the market is flattish will hurt you a lot and we have seen plenty of these in the past few years. Also, if you are trading them in a fairly long dated form, you can get hurt by a reset in volatility anchor. E.g. day 1 you are long a -7% put and short a 20/25 call spread. In the next 4 weeks, market rallies 8% however, at some point VIX will stop going any lower (say spot VIX of 14). Next thing you know, there is a new worry in town, SPX sells off 3% in a day and the VIX curve rallies. You go home crying since you are naked short a VIX call spread and your put is now far OTM.
I hear you saying that the strike risk in the spx changes as the vix does not at low levels.... then after a period of time the put is relatively farther otm compared to your vix spread... the relative changing strike risk... is the risk...