The metal sleeve will push out of the rubber bushing you just need alot of elbow grease. Having a vice mounted to a solid table will definetly help, and I usually use a big C clamp, large deep socket, and something to push the bushing out (old bolt works well) just to get it started, then finish with a hammer and punch.
Cover the whole thing with wd40 before starting :cheers: