Some labels came from the pictures alone. We still have the samples, but those labels aren't tied to physical inspection results. That's irritatingly circular.

Mark those labels uncertain and have the samples inspected properly. Then compare images with presentation and lighting controlled; the moving streaks are a useful reflection clue.

That clue needs a limit: presentation can also change how a real scratch appears. Movement in the image isn't enough to reclassify the sample as acceptable.

Correct. I meant a reason to investigate the imaging, not a new defect label. Physical acceptance stays with the agreed inspection method.

So uncertain isn't the same as good? Where do those images go?

Keep them explicitly undecided, with the sample link. Don't use them as confirmed examples of either class until the physical review settles them.

I've checked the image folders and found that exposure and presentation both changed between them. Their improved appearance can't be attributed to either change individually.

@RachelChan1131 On my inspection setup, a folder called 'better lighting' also contained easier samples. The folder name was doing a lot of unearned work.

What did you keep beside each image? I can manage sample ID and settings, but I don't want a form nobody fills in.

On my setup we kept sample ID, exposure, lighting arrangement and presentation condition. Enough to explain a comparison without reconstructing the whole session from memory.

@FionaChen1195 Use those details to ask your vision specialist for controlled comparisons on representative surfaces. Keep the sample set fixed when assessing an acquisition change.

@HanaAllen0274 Would a learned model just cope with the lighting differences?

It still needs trustworthy labels and representative evaluation. A different model doesn't establish whether your reference scratches are real or your acquisition comparison is fair.

@HanaAllen0274 Keep physical sample identity in mind when forming evaluation sets. Different images of a sample used for tuning don't provide an independent sample-level evaluation.

@LouisChan1094 I'll keep all images of one sample together when we split tuning and evaluation sets. Our file names alone wouldn't have caught that overlap.

Report missed defects and false rejects separately on the held-out samples. One agreement percentage can hide which mistake your method is making.

Should undecided physical samples count as misses in that report? I don't want them quietly disappearing just because they're awkward.

Show them as undecided with their count, outside confirmed-class error rates. That keeps the coverage limit visible without inventing a ground-truth label.

That helped on my setup. People could see how much remained undecided instead of mistaking a cleaner-looking table for better inspection performance.