From a cryptographic standpoint they're equivalent and if either the secure card or smartphone are stolen your account is still theoretically safe since they don't know your username/password. Unless of course you've written your username/password on a piece of paper stored with the device which nobody does (LOL).
How many folks have had their life's savings stolen by unauthorized trades in the past decade? Vs how many folks have died in a car crash, while crossing the street, or literally by being hit by lightening? It's one of those low probability events like a plane crash that the human brain dwells on far beyond what is statistically reasonable. Which isn't to say it's bad to dwell on it, we probably can't help it to some extent, it's just to say it's probably worth trying to overcome with a rational line of thought before you sink a bunch of time/money/inconvenience/missed opportunities into it.