Also how long have you had, him and does he live with a friend?
Guinea pigs do better with a same species companion, and this will often make them feel braver and has the added bonus of you being able to see them interact in a natural way with their companion.
It is similar to you suddenly being put in a room full of people speaking a foreign language. If you suddenly discover someone who speaks the same language as you, your confidence will automatically increase as you have a friend who really understands you.
But ultimately some piggies never really like being handled and picked up, even after years in a safe and loving home.
It isn't a reflection on you - it is just how some piggies are.