No worries about feeding sweet potato.....I doubt that they will eat more than a few bites. My cavies will only eat baked sweet potato and I give them a small chunk and let them decide if they prefer flesh or peel. Of course, as with any vegetable, scrub the potato before cooking.
My boy has what we call a "sweet tooth"......meaning that he likes many things that are sweet. However, he apparently just wants to taste it because it smells good. He rarely will eat more than a couple of bites. You don't have to be rigid and never give your piggy some items. Just use good judgment and don't feed them very often.
For instance, Chester loves dried cranberries.....which are loaded with fruit sugar and which neither of us needs to be consuming on a regular basis. But, he won't eat more than two of them, and often just one. He's a smart boy. lol