I've found that some labels were assigned from the images themselves. The samples are still available, but those labels have no corresponding physical inspection results, making the reference circular.

Mark those labels uncertain and have the samples inspected properly. Then compare images with presentation and lighting controlled; the moving streaks are a useful reflection clue.

A scratch's visibility can change with presentation too. Don't let 'moving streak' quietly become the rule for relabelling something acceptable.

You're right to qualify that. The streak behaviour motivates an imaging investigation; it doesn't replace physical classification under the agreed inspection procedure.

So uncertain isn't the same as good? Where do those images go?

@BethAli0237 Retain their sample identities and undecided status. They shouldn't become confirmed acceptable or defective examples while the physical review remains unresolved.

I've checked the image folders and found that exposure and presentation both changed between them. Their improved appearance can't be attributed to either change individually.

On my inspection setup, a folder called 'better lighting' also contained easier samples. The folder name was doing a lot of unearned work.

Which details did you retain with each image? I can record sample identity and settings, but I'd like a practical record people will actually maintain.

On my setup we kept sample ID, exposure, lighting arrangement and presentation condition. Enough to explain a comparison without reconstructing the whole session from memory.

Use those details to ask your vision specialist for controlled comparisons on representative surfaces. Keep the sample set fixed when assessing an acquisition change.

Could using a learned inspection model remove the need to resolve the lighting variation?

@BethAli0237 Any learned model still depends on reliable labels and representative evaluation. Changing the method doesn't resolve unconfirmed physical defects or an uncontrolled comparison of image conditions.

Keep physical sample identity in mind when forming evaluation sets. Different images of a sample used for tuning don't provide an independent sample-level evaluation.

I'll group images by physical sample before dividing the tuning and evaluation sets. The current filenames wouldn't reliably expose the same sample appearing in both.

Use the held-out samples to report defect misses and false rejections independently. A combined agreement figure can conceal the type of error that matters to the process.

@AnikaAli0247 My starting point is clear: establish physical references and compare image conditions in a controlled way. That resolves the planning question while leaving inspection suitability dependent on evaluation.

You've settled the starting method, with inspection suitability still tied to the physical-reference and controlled-image evaluation.