@YasminAdams0093 Some labels came from the pictures alone. We still have the samples, but those labels aren't tied to physical inspection results. That's irritatingly circular.

Mark those labels uncertain and have the samples inspected properly. Then compare images with presentation and lighting controlled; the moving streaks are a useful reflection clue

A scratch's visibility can change with presentation too. Don't let 'moving streak' quietly become the rule for relabelling something acceptable

Correct. I meant a reason to investigate the imaging, not a new defect label. Physical acceptance stays with the agreed inspection method

@YasminAdams0093 Should uncertain images be treated as acceptable, or kept somewhere separate while their status is established?

Retain their sample identities and undecided status. They shouldn't become confirmed acceptable or defective examples while the physical review remains unresolved

Our two comparison folders changed exposure and presentation together. i can't tell which change made the images look better. That comparison needs a caveat too.

I had a separate inspection setup where the 'better lighting' folder used easier samples too. Its name implied a lighting conclusion that the comparison didn't support.

What did you keep beside each image? i can manage sample ID and settings, but i don't want a form nobody fills in.

For my separate setup, sample identity, exposure, lighting arrangement and presentation condition were the useful essentials. They let us understand comparisons without relying on recollection.

Use those details to ask your vision specialist for controlled comparisons on representative surfaces. Keep the sample set fixed when assessing an acquisition change

Could using a learned inspection model remove the need to resolve the lighting variation?

It still needs trustworthy labels and representative evaluation. A different model doesn't establish whether your reference scratches are real or your acquisition comparison is fair

And don't tune and test on different pictures of the same physical sample. That isn't the independence the result would suggest

@JamieBrown0904 i'll group images by physical sample before dividing the tuning and evaluation sets. The current filenames wouldn't reliably expose the same sample appearing in both.

Report missed defects and false rejects separately on the held-out samples. One agreement percentage can hide which mistake your method is making

Should undecided physical samples count as misses in that report? i don't want them quietly disappearing just because they're awkward.

Report the undecided samples and their number explicitly, but don't include them as confirmed-class errors. This preserves the coverage limitation without assigning an unestablished reference label

@YasminAdams0093 i've settled the review method: physical reference checks plus controlled image comparisons. That answers where to start; whether our inspection concept is suitable still depends on the resulting evaluation.