You just need to give it a really hard tug (ooo errr)
Get a watchmakers screwdriver and spray some WD40 up inside to help release it. Worth only really doing if you are replacing for a new one which pops on with a firm slap using the palm of your hand.