I've found that some labels were assigned from the images themselves. The samples are still available, but those labels have no corresponding physical inspection results, making the reference circular.

@SaraBrown0896 Mark those labels uncertain and have the samples inspected properly. Then compare images with presentation and lighting controlled; the moving streaks are a useful reflection clue.

@EllaBell0642 A scratch's visibility can change with presentation too. Don't let 'moving streak' quietly become the rule for relabelling something acceptable.

Correct. I meant a reason to investigate the imaging, not a new defect label. Physical acceptance stays with the agreed inspection method.

So uncertain isn't the same as good? Where do those images go?

Keep them explicitly undecided, with the sample link. Don't use them as confirmed examples of either class until the physical review settles them.

@EllaBell0642 Our two comparison folders changed exposure and presentation together. I can't tell which change made the images look better. That comparison needs a caveat too.

I had a separate inspection setup where the 'better lighting' folder used easier samples too. Its name implied a lighting conclusion that the comparison didn't support.

@MinaAli0250 Which details did you retain with each image? I can record sample identity and settings, but I'd like a practical record people will actually maintain.

@SaraBrown0896 For my separate setup, sample identity, exposure, lighting arrangement and presentation condition were the useful essentials. They let us understand comparisons without relying on recollection.

Use those details to ask your vision specialist for controlled comparisons on representative surfaces. Keep the sample set fixed when assessing an acquisition change.

@EllaBell0642 Would a learned model just cope with the lighting differences?

@OmarChan1100 It still needs trustworthy labels and representative evaluation. A different model doesn't establish whether your reference scratches are real or your acquisition comparison is fair.

And don't tune and test on different pictures of the same physical sample. That isn't the independence the result would suggest.

I'll group images by physical sample before dividing the tuning and evaluation sets. The current filenames wouldn't reliably expose the same sample appearing in both.

Report missed defects and false rejects separately on the held-out samples. One agreement percentage can hide which mistake your method is making.

How should the evaluation report handle samples whose physical status is undecided? I want them visible without assigning an unsupported error label.

Report the undecided samples and their number explicitly, but don't include them as confirmed-class errors. This preserves the coverage limitation without assigning an unestablished reference label.

@EllaBell0642 My starting point is clear: establish physical references and compare image conditions in a controlled way. That resolves the planning question while leaving inspection suitability dependent on evaluation.