I would plug it with the same material that you would use for an end grain bushing in a peg hole. Boxwood or maple would be fine. (I like to use yellow heart for my bushing stock.) Even though you need to move it 4 mmm, by the time you are done not very much of the bushing will be visible so it doesn't matter that much.