I also use hand vac for my cage. I always remove my piggies from their cage and move them to a new room, however. To me it's not a matter of whether or not the sound scares them, it's that their ears are much smaller than ours and I am positive the sound of vacuum is painful to their ears. I do the same thing while they're in the car in the back seat, I adjust the sound settings so the bass is down really low and only the front speakers are playing music.
We often forget how much more sensitive our piggies ears are than ours!
But back to vacuums, I use a hand vac but will cease to do so when I get coroplast. Currently, their fleece is under their grids, which holds it down from the suction and is (usually) pretty easy. It's annoying to accidentally suck up a corner of fleece and have poopies go flying, haha.
When I get fleece and attach my cage liner to it, I will resort from twice daily vacuum patrol to twice daily fleece shake-out patrol. That protects the filter from hay better, anyways.
Hope I helped! Please don't forget about your piggy's super sensitive hearing when vacuuming!