@FionaAllen0325 I've found that some labels were assigned from the images themselves. The samples are still available, but those labels have no corresponding physical inspection results, making the reference circular.

Keep the image-only labels uncertain pending physical inspection. A controlled comparison of lighting and presentation can then investigate whether the moving streaks indicate reflection effects.

That clue needs a limit: presentation can also change how a real scratch appears. Movement in the image isn't enough to reclassify the sample as acceptable.

@CallumChen1143 Correct. I meant a reason to investigate the imaging, not a new defect label. Physical acceptance stays with the agreed inspection method.

So uncertain isn't the same as good? Where do those images go?

Retain their sample identities and undecided status. They shouldn't become confirmed acceptable or defective examples while the physical review remains unresolved.

@FionaAllen0325 I've checked the image folders and found that exposure and presentation both changed between them. Their improved appearance can't be attributed to either change individually.

On my inspection setup, a folder called 'better lighting' also contained easier samples. The folder name was doing a lot of unearned work.

Which details did you retain with each image? I can record sample identity and settings, but I'd like a practical record people will actually maintain.

@LuisBrown0956 For my separate setup, sample identity, exposure, lighting arrangement and presentation condition were the useful essentials. They let us understand comparisons without relying on recollection.

@JuliaChan1109 Those records can support the vision specialist's comparison of image conditions. Holding the representative sample set constant will make an acquisition change easier to assess.

Would a learned model just cope with the lighting differences?

Any learned model still depends on reliable labels and representative evaluation. Changing the method doesn't resolve unconfirmed physical defects or an uncontrolled comparison of image conditions.

@FionaAllen0325 And don't tune and test on different pictures of the same physical sample. That isn't the independence the result would suggest.

I'll group images by physical sample before dividing the tuning and evaluation sets. The current filenames wouldn't reliably expose the same sample appearing in both.

Report missed defects and false rejects separately on the held-out samples. One agreement percentage can hide which mistake your method is making.

My starting point is clear: establish physical references and compare image conditions in a controlled way. That resolves the planning question while leaving inspection suitability dependent on evaluation.

You've settled the starting method, with inspection suitability still tied to the physical-reference and controlled-image evaluation.