I've found that some labels were assigned from the images themselves. The samples are still available, but those labels have no corresponding physical inspection results, making the reference circular.

Mark those labels uncertain and have the samples inspected properly. Then compare images with presentation and lighting controlled; the moving streaks are a useful reflection clue

@BethBaker0498 That clue needs a limit: presentation can also change how a real scratch appears. Movement in the image isn't enough to reclassify the sample as acceptable.

@NinaArcher0414 Correct. I meant a reason to investigate the imaging, not a new defect label. Physical acceptance stays with the agreed inspection method

@BethBaker0498 Should uncertain images be treated as acceptable, or kept somewhere separate while their status is established?

Keep them explicitly undecided, with the sample link. Don't use them as confirmed examples of either class until the physical review settles them

I've checked the image folders and found that exposure and presentation both changed between them. Their improved appearance can't be attributed to either change individually.

On my inspection setup, a folder called 'better lighting' also contained easier samples. The folder name was doing a lot of unearned work.

Which details did you retain with each image? I can record sample identity and settings, but I'd like a practical record people will actually maintain.

On my setup we kept sample ID, exposure, lighting arrangement and presentation condition. Enough to explain a comparison without reconstructing the whole session from memory.

I've settled the review method: physical reference checks plus controlled image comparisons. That answers where to start; whether our inspection concept is suitable still depends on the resulting evaluation.

That gives the evaluation a sound question to answer. Suitability remains dependent on those checks, as you've made clear