No, being in the other cage does not make him impatient and aggressive. His hormones do that, and would do so even if he were in with the other pigs. It will help if you move his cage to another room.
Please keep him separated and do not put him with any females. Pregnancy and delivery are VERY hard on guinea pig sows and pups, and the death rate is high. Also, guinea pigs have several genetic conditions that can cause blindness, deafness, wonky or missing teeth, intestinal deformities, and painful, untreatable bone conditions. We NEVER recommend that guinea pigs be allowed to breed. There are enough accidental pregnancies to keep them going without every deliberately breeding them.
You will also have to separate any male pups from the mom and any sisters at three weeks of age, or they can impregnate their mom and sisters. That just increases the chances of a genetic condition.