So you're picking him up quite a bit every day, it seems. My pigs get disgruntled if I pick them up more than two or three times every day, it's just too much jostling and stress for them. I picked Fred up three or four times one day out of necessity, and by the end he got sick of it and bit me. I gave him a day or so to cool down, and then picked him up no more than twice a day (floor time, lap time/nail clipping) since, and I haven't been bitten again.
Ani, my other pig, doesn't "taste" people like Fred does when looking for food, she bites down and drags. So if I'm silly enough to put my hand out in front of her like I'm offering food, I might get bit.
Try wrapping your pig in a towel to hold it. Sometimes they feel more secure that way and they'll tolerate being held longer. Try to go for longer at a time holding your pig -- maybe half an hour in one sitting. If he doesn't like being held, then he doesn't like being held. Lap time is for you, not him, so handling him less might give him a chance to relax.
Consider trying to be more active in the cage instead. Fred HATES lap time so instead I mess with the hay and the food and the water a few times a day, not to add more, but just to change things up and be present. When they get floor time I like to sit in the middle sometimes, and if they want to come say hello, they get to say hello. If not, no stress for anyone.
My bottom line is, not all pigs like being held, and if yours doesn't then you're exacerbating the problem by picking him up three or more times every day. Try to be more active in the cage instead, and hold him for more time in each sitting. That will relax him and make him more receptive to you, because he won't have to worry about being jostled around and floated feet above the ground every time he sees your hands near the cage. Make your hands more than just something to fear -- bring little veggie treats to hand feed, and don't pick him up every time you go in the cage. I always take the pigs out to feed treats like dandelions or fruit, because they get them so rarely, and on the whole this makes them more tolerant of lap time and being held -- they get something good out of it.
Hope something in there helps!