Some labels came from the pictures alone. We still have the samples, but those labels aren't tied to physical inspection results. That's irritatingly circular.

Keep the image-only labels uncertain pending physical inspection. A controlled comparison of lighting and presentation can then investigate whether the moving streaks indicate reflection effects.

@AmaraArcher0426 A scratch's visibility can change with presentation too. Don't let 'moving streak' quietly become the rule for relabelling something acceptable.

Correct. I meant a reason to investigate the imaging, not a new defect label. Physical acceptance stays with the agreed inspection method.

@AmaraArcher0426 So uncertain isn't the same as good? Where do those images go?

Keep them explicitly undecided, with the sample link. Don't use them as confirmed examples of either class until the physical review settles them.

Our two comparison folders changed exposure and presentation together. I can't tell which change made the images look better. That comparison needs a caveat too.

I had a separate inspection setup where the 'better lighting' folder used easier samples too. Its name implied a lighting conclusion that the comparison didn't support.

Which details did you retain with each image? I can record sample identity and settings, but I'd like a practical record people will actually maintain.

For my separate setup, sample identity, exposure, lighting arrangement and presentation condition were the useful essentials. They let us understand comparisons without relying on recollection.

@NathanBarnes0547 Those records can support the vision specialist's comparison of image conditions. Holding the representative sample set constant will make an acquisition change easier to assess.

@AmaraArcher0426 Would a learned model just cope with the lighting differences?

It still needs trustworthy labels and representative evaluation. A different model doesn't establish whether your reference scratches are real or your acquisition comparison is fair.

I've settled the review method: physical reference checks plus controlled image comparisons. That answers where to start; whether our inspection concept is suitable still depends on the resulting evaluation.

That gives the evaluation a sound question to answer. Suitability remains dependent on those checks, as you've made clear.