I don't think they are dangerous, but I agree that you should look for the cause. Are you leaving the veggies out for longer than you need to? Is the cage cleaned regularly enough? Is there a window being left open? I recommend getting some sticky fly-paper to catch them quickly and cheaply if it becomes a problem.
I'm having the same problem. They seem to like my house and they started coming in when I got my great dane, but now the hang out in the piggy room. I clean the cage atleast twice a week and make sure all veggies are picked up but they still seem to come in. I've also put up sticky paper and it seems more are coming in, but it did catch quite a few of them. The rest we got with a fly swat.