If you are restoring the car and want it to be as "correct" as possible, then technically it should be restored to the way it left the factory. However, I can see the exterior color 'mistake' being a controversial item. Do you plan on having the car judged at a national level? If so, you will have to have some pretty serious documentation to prove that it was a factory mistake. If this is the case, you may want to contact someone like Jerry MacNeish to discuss the documentation you have and paint color impact on judging and/or value.
With all that said, I say if you really wanted a black car and the trim tag and POP shows black, why not restore the car to the way you wanted it in the first place.