How hard is camshaft removal on a 3rz-fe?

I made a mistake on my valve job, I removed the shims and rotated the cam shaft and chipped the edge shim bucket in the process. While there isn't a clear answer where that leaves me, the two answers I got both had me removing the cam shaft and shim bucket and either filing down sharp corners or replacing it (the shim bucket) altogether.

I am most concerned about getting this engine back together and in time. What does removing the camshaft involve and is it difficult to get it back together once removed?