A watchdog that can only stop things turns a short storm into a long outage

My agents run day and night on a server of my own. A storm of a few minutes cost me more than two hours of standstill. Not because of the storm, but because of the watchdog.

What a fork storm is

Every program does its work through processes. A fork storm is a server that starts and stops new processes so fast that it has no time left for anything else. Memory and CPU look free, and yet everything is slow.

On an ordinary day my server starts 200 to 300 processes per second. In a storm I saw more than 900, and in an earlier one more than 1,200. With people this rarely happens. With agents it does: an agent running a test does it with dozens of small commands at once, and at night several agents work side by side without knowing about each other.

What went wrong

I have a watchdog that spots this, points out a suspect and freezes it. It did exactly that. But freezing is an intervention that someone has to undo. It was evening, nobody was looking, and the heart of the system stayed frozen. Every agent stood still. The storm itself lasted minutes. The standstill lasted more than two hours.

A watchdog that can only stop things turns a short storm into a long outage.

What I changed

• After five quiet minutes the watchdog thaws things itself.
• The heart of the system is never frozen for more than ten minutes, even if the storm continues.
• If the storm returns quickly from the same cause, I stop only that cause, not the heart.

I test this with a test that starts a fork loop itself, demands that it gets frozen and then runs again.

Six freezes, six times intended work

I went back through the measurements of a few days. In that time the watchdog stepped in six times. All six times it froze work I wanted to run myself: a test inside an agent's turn, the rollout queue, an agent assembling a video. Not once was it a real loop. Once it even picked the wrong target, because the blame landed on the process that cleans up the small processes, instead of on whoever started them.

The reason is simple: the number of processes per second says almost nothing. Intended work peaks higher than the problems. On an ordinary day my server stays far below those peaks, and intended heavy work goes well above them. A count cannot tell them apart.

What a loop gives away

The actual problem on my server looked quite different. A check that ran at the start of every turn of every agent over all tickets started one process per ticket. That was tens of thousands per round, for every agent, again and again, day and night. There was no storm to see, just a server constantly busy doing nothing. You recognise a loop like that by something other than speed:

• the same command, over and over, from the same parent, where intended work starts many different commands;
• one process per piece of work, or a check that runs every few seconds, where intended work runs in bounded parallel and stops after a few minutes;
• sustained or growing, without progress, and children that fail instead of succeed;
• a load that stays far above the number of cores.

So these are patterns over time and repetition, not one number. My watchdog therefore measures per command and per parent, looks at the number of running processes per group, and steps in only on the group that behaves like this. Heavy work that registers itself gets room.

Why no hard limit

The simplest fix is a ceiling on processes per second. I deliberately did not do that. The tests I run every half hour sometimes reach well over 3,000 per second, and that is intended work. A hard limit either throttles that, or is so generous that it does not catch a storm. I want to use the server to the full.

What does work: telling intended heavy work from unintended heavy work, and making every intervention reversible and temporary.

What to take from this

1. Measure the rhythm of your server, not just its load. One line per minute is enough.
2. Have an agent bundle its work: not one process per piece of work, and not all at once.
3. A watchdog that can stop things must also be able to restart them.
4. Give every intervention a time limit. "Until a human looks" is not a limit.

It is the same pattern as in Het autonome AI-bedrijf: a role with a power, a protective measure and a human as the last step. A human who does not always look calls for a system that lets go by itself.

All pieces