Some labels came from the pictures alone. We still have the samples, but those labels aren't tied to physical inspection results. That's irritatingly circular.

Mark those labels uncertain and have the samples inspected properly. Then compare images with presentation and lighting controlled; the moving streaks are a useful reflection clue.

A scratch's visibility can change with presentation too. Don't let 'moving streak' quietly become the rule for relabelling something acceptable.

@GraceBell0668 You're right to qualify that. The streak behaviour motivates an imaging investigation; it doesn't replace physical classification under the agreed inspection procedure.

Should uncertain images be treated as acceptable, or kept somewhere separate while their status is established?

Keep them explicitly undecided, with the sample link. Don't use them as confirmed examples of either class until the physical review settles them.

Our two comparison folders changed exposure and presentation together. I can't tell which change made the images look better. That comparison needs a caveat too.

On my inspection setup, a folder called 'better lighting' also contained easier samples. The folder name was doing a lot of unearned work.

I've settled the review method: physical reference checks plus controlled image comparisons. That answers where to start; whether our inspection concept is suitable still depends on the resulting evaluation.

That gives the evaluation a sound question to answer. Suitability remains dependent on those checks, as you've made clear.