The cheapest place to buy hay is from a local farmer, but you're not in a great area of the country to be able to do that.
You want a long strand grass hay (timothy, meadow, orchard, blue, brome). Timothy, which most people with GPs use, is a cold-weather crop, so it doesn't grow in the south.
You might be able to find timothy hay that has been trucked in from somewhere if you look at a local feed store. If there's one that's close to horse farms, they're likely to have a good grass hay.
The cost in the middle between local farmer and local pet store is to order online. KMS Hayloft, Sweet Meadow Farms, Small Pet Select, and American Pet Diner all have good hay that you can buy in 40-50 pound bags. Oxbow will also ship a 40 or 50 pound box (can't remember which) to a local pet store for you to pick up. Which one is the cheapest will depend on how close you are to their shipping site and whether or not you've got a coupon for a discount.
You can check with their websites and see exactly how much shipping will cost you, and how much their hay is going for. The only way to compare is to figure out the cost per pound, including the shipping, and compare those costs. The bigger the order, the more money you'll save.
Even though online shipping is much more than local costs, it's still WAY cheaper than buying by the small bag in a local store.