Unless he's pooping and peeing in the pellets, you can just leave them in the cage. It won't hurt them -- there's no need to throw them away every day.
And if you'll look around for a feed store or a farmer that has grass hay, you can get it MUCH cheaper than buying it in the small bags at the pet store. Small bags is absolutely, hands-down, the most expensive way to get hay. You can order it online much cheaper than you can get it at a pet store. But if you can find a bale, it should cost you less than $10. Even if you don't have any way to store all of, you can still throw half of it away and still be way ahead of the game. I keep mine extra in a box in the garage.
He does need clean water, but I doubt that has anything to do with the calcium in the urine. What the yellow color could indicate, though, is that you have hard water. In that case, you need to filter it. Some refrigerators have filters on their water dispensers, and you could use that, but you'd need to be sure the filter gets changed regularly. Or, you could buy something like a Brita or a Pur pitcher with a filter on it.