That is true, we cannot know what the mastering engineer heard. The task of an audio engineer though is not to second-guess the mastering engineer, but to reproduce as faithfully as possible what is on the recorded medium, which is always going to serve as the reference against which all subsequent steps are measured. In other words, the only known fact is what is contained in the recording, and the engineering buck should stop there.
Your assertion though that we cannot measure imaging is wrong. Imaging is a temporal delay in signals arriving at your ears, either because of unequal distances between two sources, or due to reflections in a domestic environment. Its effects can be precisely measured in an anechoic chamber, so we can easily characterise the imaging information inherent in a recording.
Though I have not been in every recording control room on earth, in the ones a have been in (I helped build one) the alignment process did not address imaging.
Some of the loudspeakers in this discussion have very different imaging characteristics from more conventional designs and some consider their sound stage to be exaggerated. Still, they are highly regarded. Are they reference quality?
In my experience audiophiles place much more emphasis on faithful reproduction than audio engineers do in the original recording. It doesn't make the pursuit of reference quality production meaningless but does say that there is a point of diminishing return when discussing "accuracy."