A watchdog that can only stop things turns a short storm into a long outage

My agents run day and night on a server of my own. A storm of a few minutes cost me more than two hours of standstill. Not because of the storm, but because of the watchdog.

What a fork storm is

Every program does its work through processes. A fork storm is a server that starts and stops new processes so fast that it has no time left for anything else. Memory and CPU look free, and yet everything is slow.

Normally my server starts about 70 processes per second. In a storm I saw more than 900, and in an earlier one more than 1,200. With people this rarely happens. With agents it does: an agent running a test does it with dozens of small commands at once, and at night several agents work side by side without knowing about each other.

What went wrong

I have a watchdog that spots this, points out the cause and freezes it. It did exactly that. But freezing is an intervention that someone has to undo. It was evening, nobody was looking, and the heart of the system stayed frozen. Every agent stood still. The storm itself lasted minutes. The standstill lasted more than two hours.

A watchdog that can only stop things turns a short storm into a long outage.

What I changed

• After five quiet minutes the watchdog thaws things itself.
• The heart of the system is never frozen for more than ten minutes, even if the storm continues.
• If the storm returns quickly from the same cause, I stop only that cause, not the heart.

I test this for real: the test starts a fork loop itself, demands that it gets frozen and then runs again.

Why no hard limit

The simplest fix is a ceiling on processes per second. I deliberately did not do that. The tests I run every half hour sometimes reach well over 3,000 per second, and that is intended work. A hard limit either throttles that, or is so generous that it does not catch a real storm. I want to use the server to the full.

What does work: telling intended heavy work from unintended heavy work, and making every intervention reversible and temporary.

What to take from this

1. Measure the rhythm of your server, not just its load. One line per minute is enough.
2. Have an agent bundle its work: not one process per piece of work, and not all at once.
3. A watchdog that can stop things must also be able to restart them.
4. Give every intervention a time limit. "Until a human looks" is not a limit.

It is the same pattern as in Het autonome AI-bedrijf: a role with a power, a protective measure and a human as the last step. A human who does not always look calls for a system that lets go by itself.

All pieces