Did verbose logging freeze my dashboard? - read-only diagnostic sampling

AnilBell0636 · 22 May 2026, 20:13 UTC

Reply to discussion
AN
AnilBell0636
I enabled detailed logging for read-only diagnostic sampling, and now our diagnostic display sometimes stops updating while the log folder is busy. We're watching Fairino FR5 in an inspection cell with a separate monitoring laptop, with a reference coupon as a reference. The cell controls are separate. I suspect synchronous file writes, but I haven't ruled out communication delays.

20 replies

FE
FelixBarnes0545
Replying to AnilBell0636

Can you replay saved responses offline with a slow log destination? Measure receipt and processing separately.

8points
AN
AnilBell0636
Replying to FelixBarnes0545

It does. My replay keeps supplying responses, but processing stops inside the file write. Finally, something I can reproduce.

13points
FE
FelixBarnes0545
Replying to AnilBell0636

Move the replay's file writing behind a bounded queue and let a separate writer handle it. That should preserve both display progress and the diagnostic history.

9points
SA
SaraBrooks0809
Replying to FelixBarnes0545

Your proposed queue has a limit, so preserving the entire history isn't guaranteed. A persistent storage blockage needs an explicit overflow policy.

8points
FE
FelixBarnes0545
Replying to SaraBrooks0809

You're right about my preservation claim. The design needs a deliberate overflow policy and visible dropped-sample counts so missing diagnostics aren't concealed.

13points
AN
AnilBell0636
Replying to FelixBarnes0545

Can routine samples be dropped but error summaries kept? I don't need endless identical status lines.

23points
FE
FelixBarnes0545
Replying to AnilBell0636

You can give summaries priority over routine samples, but both paths need bounded storage and defined behaviour if the destination remains unavailable.

5points
GA
GabrielBrown0888
Replying to FelixBarnes0545

@FelixBarnes0545 My setup originally queued its overflow warning through the writer that was already full, so the warning disappeared along with the samples it was meant to explain.

3points
GR
GraceBennett0755
Replying to GabrielBrown0888

@GabrielBrown0888 So where does that warning go? Another log sounds like the same problem twice.

4points
FE
FelixBarnes0545
Replying to GraceBennett0755

The application can maintain a visible health state or drop counter independently of file writing, then include that information in an export when the destination recovers.

6points
AN
AnilBell0636
Replying to FelixBarnes0545

@FelixBarnes0545 I'll make logging health visible beside our monitor status. A moving display shouldn't quietly imply complete logs.

8points
SA
SaraBrooks0809
Replying to AnilBell0636

Are you displaying when the application observed the sample or when the writer saved it? A delayed writer makes those different events.

19points
AN
AnilBell0636
Replying to SaraBrooks0809

Written, currently. That's awkward. Our charts could make a disk pause look like a late robot response.

10points
FE
FelixBarnes0545
Replying to AnilBell0636

Capture observation time on receipt, then track write delay separately. Keep the chart honest about which event it shows.

14points
GR
GraceBennett0755
Replying to FelixBarnes0545

Could the controller still be late, though? The replay only proves the disk can cause it.

9points
FE
FelixBarnes0545
Replying to GraceBennett0755

The replay establishes that synchronous logging can stall this application. It doesn't establish the absence of communication delays in the real diagnostic session.

20points
AN
AnilBell0636
Replying to FelixBarnes0545

Right. I'll label this as the reproduced logging stall, not the explanation for every delay we've seen.

7points
GA
GabrielBrown0888
Replying to FelixBarnes0545

@FelixBarnes0545 I found queue depth helpful on my own monitor because it exposed a writer backlog that wasn't obvious from the size of the log file.

17points
AN
AnilBell0636
Replying to GabrielBrown0888

I still haven't established a reliable way for our display and logging to behave under storage delays. Reproducing the stall isn't a finished fix.

14points
FE
FelixBarnes0545
Replying to AnilBell0636

Fair. Reproduction gives you something to work on, but doesn't establish reliable behaviour under that delay.

10points

Add to the discussion

Welcome to Application Robot

Everyone can read the forum. Sign in or create an account to start a discussion, reply, or upload photos.

Forgot your password?

By creating an account, you agree to our Terms and Conditions and community guidelines. Read our Privacy Policy for how your information is handled.