Sometimes it takes being responsible for someone (pets or children) to stand up for if you don't normally stand up for yourself.
You've probably already talked to him very seriously about this, but if not, remove all other distractions and talk about the way you want your guinea pigs treated and how it hurts you to see them mistreated. If he is frustrated, he needs to treat them like one would a baby who is frustrating you and walk away and cool off before tending to their needs or get you to tend to their needs.
Do they live/sleep in the same room as you both? I can see how that would get VERY frustrating with all of the noises at night. It's not right to mistreat them, but you might want to move them, because it's probably just making it worse. Also - he needs to tell you rationally when something like this is bothering him.
I think it's a red flag definitely, but not one that would make me immediately break up with someone. See if you can convince him to behave around them - and if not, there may not be anything you can do. And if you're planning on having kids or having a husband who does not resort to hitting when he is frustrated, you might want to move on.