Ive got a dent in swingarm from where the exhaust has given it a good whack, but its not rubbed away at it like yours has. If it were me i would change it, something like that would always be in the back of my mind.
Its not a patcularly hard job to do, put a day aside to work on it and you can take your time and get it right.