Did verbose logging freeze my dashboard? - read-only diagnostic sampling

AnilBell0636 · 22 May 2026, 20:13 UTC

Reply to discussion
AN
AnilBell0636
I enabled detailed logging for read-only diagnostic sampling, and now our diagnostic display sometimes stops updating while the log folder is busy. We're watching Fairino FR5 in an inspection cell with a separate monitoring laptop, with a reference coupon as a reference. The cell controls are separate. I suspect synchronous file writes, but I haven't ruled out communication delays.

20 replies

FE
FelixBarnes0545
Replying to AnilBell0636

Can you replay saved responses offline with a slow log destination? Measure receipt and processing separately.

8 points
AN
AnilBell0636
Replying to FelixBarnes0545

It does. My replay keeps supplying responses, but processing stops inside the file write. Finally, something I can reproduce.

13 points
FE
FelixBarnes0545
Replying to AnilBell0636

Move the replay's file writing behind a bounded queue and let a separate writer handle it. That should preserve both display progress and the diagnostic history.

9 points
SA
SaraBrooks0809
Replying to FelixBarnes0545

Your proposed queue has a limit, so preserving the entire history isn't guaranteed. A persistent storage blockage needs an explicit overflow policy.

8 points
FE
FelixBarnes0545
Replying to SaraBrooks0809

You're right about my preservation claim. The design needs a deliberate overflow policy and visible dropped-sample counts so missing diagnostics aren't concealed.

13 points
AN
AnilBell0636
Replying to FelixBarnes0545

Can routine samples be dropped but error summaries kept? I don't need endless identical status lines.

23 points
FE
FelixBarnes0545
Replying to AnilBell0636

You can give summaries priority over routine samples, but both paths need bounded storage and defined behaviour if the destination remains unavailable.

5 points
GA
GabrielBrown0888
Replying to FelixBarnes0545

@FelixBarnes0545 My setup originally queued its overflow warning through the writer that was already full, so the warning disappeared along with the samples it was meant to explain.

3 points
GR
GraceBennett0755
Replying to GabrielBrown0888

@GabrielBrown0888 So where does that warning go? Another log sounds like the same problem twice.

4 points
FE
FelixBarnes0545
Replying to GraceBennett0755

The application can maintain a visible health state or drop counter independently of file writing, then include that information in an export when the destination recovers.

6 points
AN
AnilBell0636
Replying to FelixBarnes0545

@FelixBarnes0545 I'll make logging health visible beside our monitor status. A moving display shouldn't quietly imply complete logs.

8 points
SA
SaraBrooks0809
Replying to AnilBell0636

Are you displaying when the application observed the sample or when the writer saved it? A delayed writer makes those different events.

19 points
AN
AnilBell0636
Replying to SaraBrooks0809

Written, currently. That's awkward. Our charts could make a disk pause look like a late robot response.

10 points
FE
FelixBarnes0545
Replying to AnilBell0636

Capture observation time on receipt, then track write delay separately. Keep the chart honest about which event it shows.

14 points
GR
GraceBennett0755
Replying to FelixBarnes0545

Could the controller still be late, though? The replay only proves the disk can cause it.

9 points
FE
FelixBarnes0545
Replying to GraceBennett0755

The replay establishes that synchronous logging can stall this application. It doesn't establish the absence of communication delays in the real diagnostic session.

20 points
AN
AnilBell0636
Replying to FelixBarnes0545

Right. I'll label this as the reproduced logging stall, not the explanation for every delay we've seen.

7 points
GA
GabrielBrown0888
Replying to FelixBarnes0545

@FelixBarnes0545 I found queue depth helpful on my own monitor because it exposed a writer backlog that wasn't obvious from the size of the log file.

17 points
AN
AnilBell0636
Replying to GabrielBrown0888

I still haven't established a reliable way for our display and logging to behave under storage delays. Reproducing the stall isn't a finished fix.

14 points
FE
FelixBarnes0545
Replying to AnilBell0636

Fair. Reproduction gives you something to work on, but doesn't establish reliable behaviour under that delay.

10 points

Add to the discussion

Welcome to Application Robot

Everyone can read the forum. Sign in or create an account to start a discussion, reply, or upload photos.

Forgot your password?

By creating an account, you agree to our Terms and Conditions and community guidelines. Read our Privacy Policy for how your information is handled.