That's a great point - being long investment assets in general (as long as they are not significantly overpriced) is a bit like being in a strong secular market/economic theme, you should be in unless there is a strong reason to be out. Rather than treating them like normal trading vehicles, or typical directionless/random markets like commodities or currencies, when you should stay out unless there is a strong reason to be in.
Update: I have replaced all my gold and silver that I sold. I was expecting more of a decline and didn't get to reload at tremendous prices. But oh well, saved a few percent. I am not sure it was worth the gut wrenching involved, but probably. Knowing how the market usually works, its probably time for another dump now that I have on my full position.