Some labels came from the pictures alone. We still have the samples, but those labels aren't tied to physical inspection results. That's irritatingly circular.

Keep the image-only labels uncertain pending physical inspection. A controlled comparison of lighting and presentation can then investigate whether the moving streaks indicate reflection effects.

That clue needs a limit: presentation can also change how a real scratch appears. Movement in the image isn't enough to reclassify the sample as acceptable.

Correct. I meant a reason to investigate the imaging, not a new defect label. Physical acceptance stays with the agreed inspection method.

So uncertain isn't the same as good? Where do those images go?

Retain their sample identities and undecided status. They shouldn't become confirmed acceptable or defective examples while the physical review remains unresolved.

Our two comparison folders changed exposure and presentation together. I can't tell which change made the images look better. That comparison needs a caveat too.

@LouisBrown0920 I had a separate inspection setup where the 'better lighting' folder used easier samples too. Its name implied a lighting conclusion that the comparison didn't support.

What did you keep beside each image? I can manage sample ID and settings, but I don't want a form nobody fills in.

For my separate setup, sample identity, exposure, lighting arrangement and presentation condition were the useful essentials. They let us understand comparisons without relying on recollection.

@LeoBaker0449 Use those details to ask your vision specialist for controlled comparisons on representative surfaces. Keep the sample set fixed when assessing an acquisition change.

Would a learned model just cope with the lighting differences?

It still needs trustworthy labels and representative evaluation. A different model doesn't establish whether your reference scratches are real or your acquisition comparison is fair.

@HenryAli0177 Keep physical sample identity in mind when forming evaluation sets. Different images of a sample used for tuning don't provide an independent sample-level evaluation.

I'll keep all images of one sample together when we split tuning and evaluation sets. Our file names alone wouldn't have caught that overlap.

The reference problem is clearer, but I still can't judge inspection performance from our example images. The physical labels and image comparisons need more work.

That follows from the reference gap. Explaining it more clearly helps, but the images still can't carry a performance claim.