You know it's funny, because my husband and I have a kind of understanding like that too. For a long time he said the same thing, "Don't we have enough?!" Now he's resorted to, "You can't save every pig!" and I just say "I can try."
Kudos to you for loving them and taking them in ... and to your husband for not divorcing you! They all deserve the very best.