Some labels came from the pictures alone. We still have the samples, but those labels aren't tied to physical inspection results. That's irritatingly circular.

@RaviChan1085 Mark those labels uncertain and have the samples inspected properly. Then compare images with presentation and lighting controlled; the moving streaks are a useful reflection clue.

@JackBrooks0863 A scratch's visibility can change with presentation too. Don't let 'moving streak' quietly become the rule for relabelling something acceptable.

@EllaAli0207 You're right to qualify that. The streak behaviour motivates an imaging investigation; it doesn't replace physical classification under the agreed inspection procedure.

So uncertain isn't the same as good? Where do those images go?

@MeiBrooks0864 Retain their sample identities and undecided status. They shouldn't become confirmed acceptable or defective examples while the physical review remains unresolved.

Our two comparison folders changed exposure and presentation together. i can't tell which change made the images look better. That comparison needs a caveat too.

On my inspection setup, a folder called 'better lighting' also contained easier samples. The folder name was doing a lot of unearned work.

What did you keep beside each image? i can manage sample ID and settings, but i don't want a form nobody fills in.

On my setup we kept sample ID, exposure, lighting arrangement and presentation condition. Enough to explain a comparison without reconstructing the whole session from memory.

@HenryAli0177 Those records can support the vision specialist's comparison of image conditions. Holding the representative sample set constant will make an acquisition change easier to assess.

Would a learned model just cope with the lighting differences?

@MeiBrooks0864 Any learned model still depends on reliable labels and representative evaluation. Changing the method doesn't resolve unconfirmed physical defects or an uncontrolled comparison of image conditions.

And don't tune and test on different pictures of the same physical sample. That isn't the independence the result would suggest.

@EllaAli0207 i'll keep all images of one sample together when we split tuning and evaluation sets. Our file names alone wouldn't have caught that overlap.

Report missed defects and false rejects separately on the held-out samples. One agreement percentage can hide which mistake your method is making.

Should undecided physical samples count as misses in that report? i don't want them quietly disappearing just because they're awkward.

Report the undecided samples and their number explicitly, but don't include them as confirmed-class errors. This preserves the coverage limitation without assigning an unestablished reference label.

@JackBrooks0863 That helped on my setup. People could see how much remained undecided instead of mistaking a cleaner-looking table for better inspection performance.