@NaomiAllen0332 I've found that some labels were assigned from the images themselves. The samples are still available, but those labels have no corresponding physical inspection results, making the reference circular.

@SaraAllen0287 Keep the image-only labels uncertain pending physical inspection. A controlled comparison of lighting and presentation can then investigate whether the moving streaks indicate reflection effects.

That clue needs a limit: presentation can also change how a real scratch appears. Movement in the image isn't enough to reclassify the sample as acceptable.

Correct. I meant a reason to investigate the imaging, not a new defect label. Physical acceptance stays with the agreed inspection method.

So uncertain isn't the same as good? Where do those images go?

Keep them explicitly undecided, with the sample link. Don't use them as confirmed examples of either class until the physical review settles them.

I've checked the image folders and found that exposure and presentation both changed between them. Their improved appearance can't be attributed to either change individually.

On my inspection setup, a folder called 'better lighting' also contained easier samples. The folder name was doing a lot of unearned work.

@TobyAbbott0057 Which details did you retain with each image? I can record sample identity and settings, but I'd like a practical record people will actually maintain.

On my setup we kept sample ID, exposure, lighting arrangement and presentation condition. Enough to explain a comparison without reconstructing the whole session from memory.

@TobyAbbott0057 Use those details to ask your vision specialist for controlled comparisons on representative surfaces. Keep the sample set fixed when assessing an acquisition change.

@NaomiAllen0332 Could using a learned inspection model remove the need to resolve the lighting variation?

Any learned model still depends on reliable labels and representative evaluation. Changing the method doesn't resolve unconfirmed physical defects or an uncontrolled comparison of image conditions.

Keep physical sample identity in mind when forming evaluation sets. Different images of a sample used for tuning don't provide an independent sample-level evaluation.

@CallumChen1143 I'll group images by physical sample before dividing the tuning and evaluation sets. The current filenames wouldn't reliably expose the same sample appearing in both.

Use the held-out samples to report defect misses and false rejections independently. A combined agreement figure can conceal the type of error that matters to the process.

@NaomiAllen0332 How should the evaluation report handle samples whose physical status is undecided? I want them visible without assigning an unsupported error label.

Show them as undecided with their count, outside confirmed-class error rates. That keeps the coverage limit visible without inventing a ground-truth label.

The reference problem is clearer, but I still can't judge inspection performance from our example images. The physical labels and image comparisons need more work.