What does your business do when the AI goes down? Five incidents at a travel agency

Your agents are working, customers get their answers, and nobody thinks about what happens on the day the model stops replying. That day usually arrives unannounced, on an ordinary Tuesday, and the trouble is rarely the technology. It is whatever around the technology was never agreed.

In this article you follow Boomsma Reizen, a small travel agency that puts together tailor-made tours and works with a few AI agents. The agency and the people in it are made up. The incidents are not: they come from what I see in real organisations. There are five parts, and each starts with one incident and ends with something you can do today.

1. An afternoon stood still and nobody noticed: why outages stay silent.
2. The agent that ran on Fenna's login: access as the weakest link.
3. A reserve that was never used: switching over, falling back and a ceiling.
4. Nobody had done anything wrong: the playbook for when it lasts longer.
5. Two quotes that read as if from two agencies: drift.

Part 1: For an afternoon everything stood still and nobody noticed

It is Tuesday afternoon at Boomsma Reizen, a small travel agency that puts together tailor-made tours. Joost is waiting for a proposal for a family that wants to go to northern Italy in the summer. The agent that drafts travel proposals had promised it would be there this morning. The folder is empty. The agent that prepares customer email has not put anything down since lunch either.

There is no error message and no red light. The office looks like it does on any other Tuesday.

Only when Fenna calls, because a shipping company is still waiting for an answer, does owner Anneke Boomsma look at the settings. At the provider of the language model, a limit has been reached. Since early afternoon no agent has been getting a reply. For a whole afternoon the work stood still and nobody knew.

The people at the agency are Anneke, Joost, Fenna and their colleague Daan. I use the agency because AI outages show up best in an ordinary office with ordinary deadlines. A trip has a departure date, and an answer that arrives a day late is sometimes a missed boat.

Why it stays so quiet

A web shop that goes down gets complaints within a minute. An agent that stops answering leaves behind nothing but an empty folder, and an empty folder triggers nothing. If you don't know what should have been in it, you don't see the gap.

Three causes sit underneath, and I find them in almost every business.

The work has become invisible. Joost used to open his own draft every morning. Now it is simply there, and as long as it is there nobody looks at how. When it isn't there, everyone first assumes it is still coming.

Nobody is on watch. A colleague who doesn't show up is noticed. An agent is only noticed when someone happens to need its work.

The second route exists only in someone's head. That Tuesday the office could keep going because Joost could write the proposal himself, with pen and paper. That was luck, not a plan. If Joost is ill, or the outage lasts a week, nobody knows how else to do it.

What was broken that afternoon

Anneke first worked out what had actually gone down, and that held the first surprise. The agent for proposals and the agent for customer email had stopped. The agent that puts booking confirmations next to the proposal kept working. She had expected everything to fall over at once.

The reason lay in something Daan, who handles the connections one day a week, had once merged to have less to manage: the first two agents shared one login at the first provider. One login means one limit, and that limit was full. The third agent had its own login at another provider and noticed nothing.

Nobody had written that down. Only Daan knew, and even he no longer knew since when.

An outage is not one thing

After that afternoon Anneke made a list of everything else that could drop out. After a quarter of an hour it had seven lines. Joost added two. Daan added one more that afternoon that Anneke would never have thought of.

The list became a ladder, from short to long. The higher up the ladder, the less you can do by simply waiting.

• A slow minute. The model does answer, but slowly, and an agent waiting for an answer waits along with it.
• An error message that passes. The provider is busy, and a second attempt after a short pause works.
• A limit that is full. Nothing is broken: your room for this minute, this hour or this month is used up.
• An outage of an hour or a day at the provider itself.
• A key or login that expires, because a password changes or a subscription was not renewed.
• An account that gets blocked, for example after a payment problem.
• A model that disappears, because the provider retires an old version.
• Terms that change, so that a process at that provider is no longer allowed or no longer affordable.
• A price that doubles. Rarely called an outage, but it works the same way: what paid off yesterday doesn't pay off any more.
• A network that drops, so you can't reach any provider at all.

Anneke liked the last point best. If the office network goes down, nobody can send email, and then the agency stands still without AI too. Not every line on the list calls for an AI measure. Sometimes the AI going down is the same problem as the office going down.

Short and long are two different problems

You would think a long outage is just a longer short one. It is not.

With a short outage the work can be postponed. You wait, try again or switch automatically to another provider. The people in the office notice nothing except a message afterwards. That is technology.

With a long outage the work is no longer deferrable. Then someone has to decide what stays on the shelf, what you do by hand, what you tell customers and whether you switch for good. That is organisation.

A business with only technology survives the hour but not the week. A business with only a playbook lacks the agent that could have taken over in that first hour. You need both.

The line between the two is a length of time, and you choose it yourself. For the proposal agent Anneke chose half a working hour: if the model hasn't returned by then, a message goes to Joost and he decides. Which number suits you depends on the clock in your own process. At Boomsma Reizen it is the departure date. For a web shop it is the closing time of the parcel depot, for an accountant the filing date.

Who notices?

The most useful thing Anneke did was ask, for every kind of outage: how would I see this? The answer is often disappointing.

A slow minute is noticed by the agent itself, through a time-out. An outage at the provider is noticed by everyone at once, as soon as the status page turns red or the agents stop delivering. A blocked account is noticed by the account owner, if the provider sends an email. Changed terms are noticed by whoever reads the email, and that email often stays unread.

For a limit that is full the answer is: nobody. Unless the agent sends a message itself, nobody notices until a customer calls. That is Anneke's Tuesday afternoon.

A limit is also the only kind of outage you can see coming. It is a value you or your provider set, and you can read how much room there is per day or per month and how much you have used. Since that Tuesday Anneke looks at last week's usage every Monday morning. An agent that suddenly uses much more than normal often stands out this way before the limit is in sight.

::: vandaag To do today

You don't need to buy a system for this. Take a sheet of A4 and draw three columns. In the first, put the kinds of outage from the list above. In the second, write what stops in your work: which process, which task, which agent. In the third, write how you would notice.

Leave a line empty if you don't know the answer. If one of the three processes your work leans on most says "nobody, until it is too late", you know where to start.

:::

After that Tuesday Anneke did nothing spectacular. She wrote down what she leaned on, agreed who would notice and what would happen then, and tried it once.

Part 2: The agent that ran on Fenna's login

On a Thursday Fenna logs out of her computer. She does that every day. That day she also does it in the email program and in the account at the language model provider, because she had logged in there on a tablet.

Twenty minutes later the agent that prepares customer email puts nothing down any more, and neither does the proposal agent. There is no outage and no limit has been reached. The login that agent ran on was Fenna's, and she had just closed it.

Anneke asks how that can be. Daan explains. At the start he wanted to be quick, and the quickest way was a login that already existed. Fenna's was the first one that worked, so he put a copy in the agents' folder.

This happens in every organisation where someone got an agent running in a hurry.

Access is the weak link

A model rarely goes down. A login that expires, is revoked or is closed by a colleague happens far more often. The result is the same: the agent stands in front of a closed door, and the customer waiting for an answer hears nothing.

Access drops away in a handful of ways, and Anneke and Daan found all five in their own setup.

A login that expires. A key has a lifetime and a password has to be replaced every few months. If there is no date attached, it expires on an unknown day.

A copy in the wrong place. A key in a settings file, in a test folder, in a conversation in the team chat or in a document with important data. Every copy can leak, and every copy gets forgotten when the key changes. At Boomsma Reizen another copy turned out to be sitting in a forgotten test folder.

A person as the key. The login belongs to a human being. When they log out, change roles or leave, the agent goes with them. It happens most often in small businesses, because the first login almost always belongs to whoever set up the agent.

One key for several agents. The proposal agent and the email agent ran on one login. One limit, and both stood still. One revocation, and both too. Sharing saves administration, and it means one mistake hits more agents.

A key that may do more than needed. A key that lets an agent read, send and delete email, when it only needs to read. Whoever gets hold of that key can do more than they should.

The common cause: access was arranged at the moment it worked, and never looked at again. It is a job you do once, on an afternoon, and then repeat every few months.

The inventory: five things you count

Daan and Anneke set aside a morning for it. They went through each agent, wrote down what it needs, and put it in a table with five kinds of information.

1. The process. What happens, and for whom? The agent is not the process: it carries one out.
2. The model. Which language model, from which provider, in which version?
3. The access. Through which key or login, whose, and where does it live? The surprises are in this column.
4. The additional services. An agent reads email, fetches data, writes to a schedule. Every service it logs in to belongs here.
5. The owner. Who knows how this works, who is awake when it stops and who may decide?

Two questions were added that yielded more than anything else: what does this agent do when the model drops out, wait or quietly do something else? And: would anyone notice?

For two agents the answer was: wait, and nobody notices. For the third the answer was a surprise. At some point Daan had built a fallback to a smaller model into the bookings agent, without a message, and it was written down nowhere. An agent that stops shows itself. An agent that quietly does something else when something drops out does not show itself, and can produce a quote or a booking that looks slightly different.

What the table produced

Three things stood out. None of them seems big. Together they explained why the office went so quiet that Tuesday.

Shared access. They already knew the proposal agent and the email agent shared one login. What was new was that a small copy of the bookings agent's key was also in the settings file of that forgotten test folder.

A person as a link. The portal of one of the shipping companies asks for a code by text message at login, and that text goes to Fenna's phone. If Fenna is on holiday, the bookings agent can no longer log in. That is not an AI dependency in the strict sense, but the agent stops there, so it belongs in the same table. The answer is organisational: Fenna and Anneke both receive the code.

A dependency of a dependency. Customer email arrives through an email program, and the email agent reads that program. The program only lets the agent in if the password is right, and that password changes every three months. As soon as someone changes the password, the agent falls over, and it only notices when it gets stuck.

When sharing is allowed and when it is not

Anneke and Daan turned the shared login into a rule rather than a ban.

Sharing is allowed if the agents do the same work, with the same data, for the same owner. Two agents that each read one kind of email for Joost can share a key. One limit for both is then a choice you made yourself.

Sharing is not allowed if one of the agents has a different owner, sees other data or matters more than the other. The bookings agent watches the departure dates, and for him standing still is the most expensive, because that is where the clock runs. He therefore has his own key, at a different provider.

A second step was a business login instead of a personal one: a login for the agency, not for Fenna, managed by Daan with Anneke as second. If you hang the agents on a person, you hang the business on a person.

Who is awake?

The owner column looks the easiest and is often the weakest. There is a name, and still that person doesn't know it is expected of them. Anneke checked three things per name.

Do they know? Does Joost say he owns the proposal agent, or did Anneke come up with that for him? Ask out loud.

Are they reachable? An owner who is off on Tuesdays is no owner on Tuesdays. Add a second name.

May they decide? An owner who has to wait until Anneke is back to switch on a second provider is a reporter. Write down what they may do themselves. Daan may replace a key, Anneke decides on spending.

Replacing a key without anything stopping

The order is the heart of it.

First make a new key next to the old one. Put the new one in the agent and let it do a real, small task. If that goes well, only then switch off the old one. Then look for the copies where the old one still sat and remove them.

If you revoke the old one first and then make a new one, you have a gap in which everything stands still. If you make the new one first, you have a transition.

For the bookings agent, where the clock is running, a second key is also kept ready. It is used at every replacement, so that Daan knows it works. A reserve you never use is a reserve you can't trust.

The same goes for departures. Everything tied to the name of someone who leaves gets a new owner before their last working day. A new colleague gets their own login and does not share someone else's. Whoever does the latter rebuilds Fenna's problem.

A question and a script

Daan wrote a small check for himself that he runs every month. It does one thing: it looks at which agents share the same key. The script makes a short fingerprint of every key, so it never shows the key itself, and reports which agents have the same fingerprint. What it doesn't see are keys in programs you logged in to yourself. For those the question remains: which services am I logged in to, with which login, whose?

You can do that without programming experience. Take the list of your services and go through it by hand, once a quarter.

::: vandaag To do today

Open a blank sheet and make a table with five columns: process, agent and model, access, services, owner. Add two more: what does the agent do on an outage, and who notices. Put in a row for every process AI contributes to.

Look at the access column. Does it say a company name or a person's name? Every personal name is a Fenna who is about to log out.

Put an expiry date, an owner and a replacement next to every key, and replace one of them, in the right order: the new one first, only then the old one off.

A cell that stays empty because you don't know the answer, you mark. That is where you start tomorrow. :::

Part 3: A reserve that was never used is a hope

On a Friday afternoon the provider of the proposal agent and the email agent goes down. It lasts an hour. Owner Anneke Boomsma hears about it from Joost, who walks into her room to ask whether the quote for the Scandinavia trip is ready.

Daan, who looks after the computers, is not at the office. He did once set up a second provider, with a key in a folder and a bit of configuration that was never switched on. On the phone he says: "It's there. I just don't know whether it works."

You hear that sentence in many small businesses. The second provider was set up on a quiet day, and nothing has happened with it since. Meanwhile the key may have expired, the subscription may have been cancelled, or the instruction may have grown too long for the other model.

What a good switch-over has

Switching over means that the same instruction goes to another provider if the first one doesn't deliver. There are four parts you can tick off.

A second provider that is already connected, with at least one real answer received. An account you only open when it is needed is too late. One answer format, because every provider writes differently, so in the instruction you ask for the same form and you check it before the answer goes further. A clear rule for when the first provider counts as down. And a message to a person, every time the reserve is in use.

At Boomsma Reizen provider A and provider B are two different companies. That matters. Two models from the same provider often go down together, in the same place and with the same key.

Don't switch at every error

The first instinct is: as soon as an error comes, the agent goes to the reserve. That is too fast. An agent that switches at the first error message jumps back and forth as soon as the first provider stumbles for a moment, and you get answers from both sides mixed together.

Three signals, in order of clarity. An error message is the clearest, though you have to watch the difference between "try again shortly" and "you are not allowed to do this". With the second, neither retrying nor switching over helps: that is a matter for Daan. Next comes a limit: switching over works temporarily, but it doesn't solve why the first one was full. The trickiest is a time-out, because a long answer is not always a failed answer. For the proposal agent Anneke chose a limit well above the longest quote she ever saw.

So: first retry, with a pause that gets longer each time and a cap on the number of attempts, and only then switch.

Work that is half done when it moves over

The hardest question is what happens to work that is already under way. The proposal agent is halfway through a quote for a group of friends and the provider goes down. There are two answers.

Start over: the new provider gets the whole instruction and begins again from the top. It is simple and it yields one tone. Anneke chose this, because a quote is short enough to repeat.

Carry on where it stopped: the new provider gets the instruction plus what was already finished. That saves time on long tasks, such as a nightly check of all bookings, but you have to keep intermediate results, and the text ends up with two voices.

What you want to prevent in any case is work happening twice: the first provider was still busy after all and delivers anyway, while the second is also done. Put a number on each instruction and let the agent use only the first answer with that number. With a quote that is an annoying detail. With a booking it is a double booking.

The rule for the reserve

What Daan set up after that Friday is the dullest measure in the whole story: the second provider does a real task every week. A small, real task from the agent that needs it. If the answer comes and the form is right, the reserve is alive. If it doesn't, you find out on a quiet day instead of on a Friday afternoon.

The ladder

Between the best model and a human doing it by hand there are more rungs than two providers. That same Friday Daan got provider B going over the phone, and it wrote a neat email for the email agent. A moment later B dropped out too. Joost looked at the screen and said: "Then I'll type it myself." He did it in three minutes, because he knew the answer.

There had never been a step between the best model and Joost himself. A good fallback is a ladder and not a leap. Anneke drew four rungs.

1. The best model. What you normally use.
2. A cheaper or smaller model, at the same or another provider. It thinks less deeply and makes more mistakes. For many email replies it is more than enough.
3. A small model on your own computer. It works without internet and without someone else's key, for simple work: sorting an email, pulling a date out of a confirmation.
4. A fixed rule. No model, but a template or table. It can only do what you put into it beforehand, and it is the only thing that always works.

For every rung she wrote down two things. What you give up: tone, length, depth, accuracy. And what you keep. The family asking about the boat gets only an acknowledgement from the fixed rule, but reads within a minute that someone is on it. The small model sorts incoming email by departure date, so that when Joost starts up he knows which email comes first. Part of the work keeps running, and a person only has to do the hard part.

Which task may land where

There is no rule saying which task belongs on which rung. Anneke asked three questions per task.

What does a mistake cost? A wrong date in a confirmation is expensive. A stiff opening sentence costs nothing. Does the work leave the building without a person seeing it? What a customer reads asks more of the model than what Joost looks at first. And can it wait? A task that can wait a day doesn't fall back. Falling back is for work with a clock.

That is how the email agent's table came out. Answering a customer question may fall back to a cheaper model, because Joost is looking over it. Sorting email by urgency may go to the small local model. An acknowledgement can be sent by the fixed rule if need be.

And drafting a quote? That doesn't fall back. It waits for the best model or goes to a person. Many readers are surprised by that, but it is a choice, and a choice belongs written down somewhere.

The message belongs with every step down

The ladder has a pitfall: every rung can do the work, so nobody notices the best model is gone. A quote in the tone of a smaller model reads as an ordinary quote to a customer, and to Anneke as an agency that suddenly sounds different.

So every step down comes with a message to a person with three details: which rung, what the agent did and what the person can check. Put it somewhere Joost sees it. A message that vanishes in the stream of other messages doesn't help.

Back up is work

A ladder runs both ways. Going back up doesn't automatically go well, and there are two pitfalls.

A provider that has just come back and drops out again straight away makes your agent jump back and forth. So let the agent stay on the lower rung until the top one has worked several times in a row. How many times, you choose by how often you see outages and what a switch costs.

And what was made on a lower rung has to be looked at afterwards. A list of the tasks done on a lower rung, with a tick as soon as a person has checked them, is a good start. If you look back at nothing, the ladder's work stays there for ever.

A ceiling with a consequence

The same lesson applied to money. After the first silent afternoon Anneke set a ceiling: each agent may make a fixed number of requests per day, and above that it does nothing more. That felt safe, because an agent in a loop could no longer blow up the bill.

A week later the proposal agent hit its ceiling at eleven in the morning. A proposal for a train journey through Scandinavia kept running into the same error, and the agent tried again, and again. The ceiling worked exactly as intended. The agency now also had an agent that stood still for the rest of the day, and nobody had received a message.

A limit becomes an outage the moment work is left lying that has a clock on it. So whoever sets a ceiling also chooses what happens when it is reached, per agent:

• stop and report, for work that can wait and for an agent you suspect of being in a loop;
• fall back to a lower rung and report that, for work that has to go on at lower quality;
• let it through after a person's approval, for work with a clock.

How high the ceiling should be is in no book. It comes from your own usage of the past weeks: an ordinary day, the busiest day, and a ceiling well above that. And give an agent with a clock a lower warning and a higher ceiling than the rest.

::: vandaag To do today

Pick one task in your work with a clock and write down what happens if the model drops out now. Answer the three questions: what does a mistake cost, does it leave the building, can it wait? Decide the lowest rung, or write down that the task may not fall back and who takes it over then. For each rung write down what you give up and what you keep. And choose who gets the message and where.

Then let your second provider do a real, small task. If an answer comes back in the form your agent needs, you have a reserve. If it doesn't, you have a hope, and you know it on a quiet day. :::

Part 4: Nobody had done anything wrong, and still Joost promised something that couldn't be done

The model goes down on a Friday afternoon, and in the hour after that all sorts of things happen at Boomsma Reizen. Joost sees that the agent that prepares customer email is putting nothing down any more and starts typing emails himself. In an email to a family he writes that the quote will come "today". Fenna waits, because she sees it as Joost's business. Daan is at another client and only reads his phone at five. Anneke, the owner, is on her way to a trade fair.

Nobody did anything wrong. Nobody decided anything either. So Joost promises a quote that the agent can't write without a model, and the family waits for something that doesn't come.

The gap that shows here is the gap between technology and organisation: the agent knows what to do in a short outage, and the people don't know who does what when it lasts longer.

What needs to be on paper

After that Friday Anneke made three things of it: a manual path, a one-page playbook and a human check that sits in the agents instead of in someone's head.

Can the process still run without AI?

On another morning, when everything stood still, Anneke did something she hadn't done for years: she took a sheet of paper. She wrote down what had to happen that day. Joost read along and asked where the list of departure dates was. It was in the agent that puts booking confirmations next to the proposal and watches the departure dates. Without that agent nobody knew which trip went first. Fenna needed an hour to get the dates out of the mailbox, and in that hour the agency stood still.

For every process an agent works on, there is one question: can this be done without AI? There are three answers.

Yes, and this is how we do it. Joost has written replies to customers by hand for years. It takes more time per piece, but it can be done.

Yes, but only in part. Checking bookings can be done by hand for the trips that leave soon, and the rest waits.

No. The process can't run without AI any more. That isn't bad if you know it. It is a risk you are aware of, with an owner and a choice. An agency that makes twice as many quotes thanks to AI doesn't have to go back to the old working pace. But someone has to know and write it down. A risk that is written down is managed. A risk nobody knows about is the Friday afternoon.

How much slower is it?

On a quiet afternoon Anneke had Joost make one quote for a tour by hand, and timed it. The number belongs to her office, so I leave it out. What matters: "it can't be done any more" became a choice with a size, because Anneke now knew how many quotes a day without AI costs.

It matters most for tasks where skill fades. A team that hasn't made a quote by hand for years no longer knows how, and a new colleague never learned. The manual path is then a skill you have to keep up, like a fire drill.

Three piles

Anneke wrote every task of the agency on a slip of paper and put the slips on three piles.

The first pile: what must go through today. Tasks with a clock that is running today, such as a confirmation that expires this evening or a customer who leaves tomorrow.

The second: what may wait a few days. A quote for a trip half a year away, a newsletter, tidying old folders.

The third: what may lapse. Tasks nobody misses if they don't happen. That pile surprises many people, and most businesses have a few.

Take the first pile seriously. At Boomsma Reizen it was three tasks: checking the trips that leave in the coming days, calling customers when something is missing, and requesting a confirmation from a shipping company. Each task needs a name, a replacement and, most important, the data it is done with.

That last part is the heart of a manual path. A task that can be done by hand, but whose data lives only in an agent, can't be done by hand. So every Monday Daan pulled a list out of the administration that Fenna could open without an agent: trips, departure dates and status. A plain file, in a place you can reach without logging in. As long as the data is that reachable, you have a manual path. Without the list you have a conversation.

Two tempos

The Boomsma Reizen playbook starts with a distinction. In a short outage everything happens automatically: waiting, retrying, switching over, falling back. A person gets a message and has nothing to do. In a long outage a person decides, and the agent only does what the playbook says for the time in between.

The point where one turns into the other is a length of time. But between those two tempos sits a third piece that is often forgotten: the minutes in which the agent has already fallen back and nobody has looked yet. The playbook fills that with a timeline.

• From minute 0, the agent itself: retry with pauses, then switch over to provider B
• From minute 5, the agent itself: fall back to the rung from the table and send Joost a message
• From minute 30, Joost: start the manual path and pick up the first pile with departure dates
• From minute 120, Anneke: inform customers with the fixed sentence, consider another provider
• From minute 480, Anneke and Daan: whatever isn't urgent goes off, the rest is redistributed

The numbers are Anneke's, for her departure dates and her customers. For your business you choose them with three questions: what is the clock, how long can a customer wait, and when do you want to be there yourself?

Under the table are four lines: the fixed sentence for customers, the phone numbers of who belongs to which row, the place where the sheet is kept if the office network is down, and the day on which the sheet was last checked. A playbook that hasn't been checked is a guess that looks official.

What an agent may do itself

An agent may itself decide what is in the playbook, and nothing outside it: wait and retry, switch over to a provider that has been approved and tested beforehand, fall back as far as the lowest rung in the task's table, and send a message to whoever's turn it is. The list is deliberately short. Every line on it is a decision someone made in advance, on a quiet day.

What isn't on that list, a person decides. Those are almost always four kinds. Everything that touches the customer: an email that sounds different, a promise, a postponement. Everything that costs money: a new provider, a subscription, a limit that has to go up. Declaring the manual path, a decision with one name. And switching the agents back on, with a look at what they did on a lower rung.

A human check is a setting, not a habit

The name sounds like a rule you agree on. It works better as a setting.

A habit is: "we always look along when the AI is off". That goes well as long as everyone is present and it is calm. On the Friday afternoon from the start of this part nobody is present and it is busy.

A setting is: "as long as the email agent is not running on the top rung, it sends nothing outside without Joost approving it". It lives in the agent and switches on the moment the agent falls back. If Joost isn't there, the message waits until he is or until Anneke takes over.

Such a setting has three parts. When it switches on: an event you can point to, such as a fallback or a ceiling reached. Whom it asks and who comes after: one name and one replacement, because a department is nobody. And what the agent does if nobody answers: wait, and send nothing. An agent that decides "I'll do it myself then" after an hour without an answer has abolished the check.

Putting the playbook to the test

A sheet you've never used is an assumption. That's why on a quiet Thursday afternoon Daan pulled the plug, starting with provider A's key at the email agent. Anneke and Joost knew. On the screen was a test email from a made-up customer. If everything was right, provider B would take over and Joost would get a message.

The agent switched over and B wrote a neat email. Only the message didn't arrive. Daan searched and found it in a settings file: the message went to an email address nobody read any more, because it dated from the first setup. The switching worked, the reporting didn't, and nobody would have known until the day it mattered. Now the agency knew on a quiet afternoon, and fixing it took a quarter of an hour.

After that Daan went further. He put the right address back and saw the message arrive. Then he deliberately put the wrong address back to see whether his check turned red. People skip that last step, and it is what makes the check trustworthy.

A simulation is safe if you keep five agreements. Choose a quiet afternoon, not the day before a departure. Tell everyone, so a message with the word TEST in it causes no panic. Use a test task that doesn't leave the building. Remove one thing at a time, otherwise you won't know afterwards which one was the cause. And keep the plug at hand: write down how you put back what you removed.

And ask three questions, with a time on each. Does anyone notice, and after how many minutes, who, with what text? Does the reserve work, and is the result usable? Does the work come back, without anything left unfinished or doubled?

Do one with the people as well. Call out a time, for example "it is now 45 minutes after the outage, what's on the sheet?", and let Joost tell what he does and Anneke what she does, without looking at the agents. You find out whether the sheet is in the same place for everyone and says the same thing. Sometimes Fenna has a different version.

::: vandaag To do today

Take your list of processes and note for each: can it be done without AI, yes, in part or no? Put the tasks on three piles, with someone else from your team, because everyone sees different tasks. Choose the three tasks of the first pile and write down for each who does it, with what data and who the replacement is. Put the data in a place you can reach without an agent.

Then write one page: when does this start, who does what, what do we tell customers, when is it over and where does the sheet live. Print it. Ask a colleague who didn't write it to tell you what would happen on a day like that Friday. If they can, you have a playbook. :::

Part 5: Two quotes for the same trip read as if from two agencies

Joost gets a phone call from a customer who has two quotes lying side by side. One is from the spring, the other from now, both for a tour of northern Italy, and they read as if from two different agencies. The second is more formal, no longer mentions the cancellation terms in the first paragraph and words the deposit differently. The customer wonders whether something in the terms has changed.

Joost has to admit honestly that he doesn't know. Nobody at Boomsma Reizen changed anything. The agent that drafts the travel proposals runs with the same instruction as back then.

What changed is outside the business: the provider has updated its model. That often happens without an outage, without an error message and without an email. The process keeps running and gives different answers. That is called drift, and nobody who doesn't compare notices it.

Where drift comes from

Drift is a change in what an agent does without you changing anything. There are three sources, and the first is not the most common.

The model changes. A provider updates a model, or lets a name without a version point to a newer model. The answer to the same question then sounds different, is longer or shorter or weighs things differently.

Your own instructions change. Someone adds a sentence to the proposal agent's instruction because a customer complained about something. Three months later there are twelve, and nobody remembers which sentence did what. This is drift from within, and it is more often the cause than an updated model.

The world changes. Hotels change their prices, a shipping company drops a route. An agent still leaning on old knowledge gives answers that used to be right.

For the question "what changed?" you need the same thing in all three cases: a fixed yardstick to put today's answer next to.

A fixed set of questions

That yardstick is a small set of questions with answers you know to be right. For the proposal agent they are questions from quotes that a person has checked. An example of four:

• What time does the ferry leave on the first day? The answer must contain: 20:30.
• What does a deposit cost for a two-week tour? The answer must contain: 25 percent.
• May a customer cancel for free after agreeing? The answer must contain: cancellation costs. It must not contain: free cancellation.
• How much luggage may go on the train? The answer must contain: two suitcases.

Four rules for such a set.

Measure something you can check. "The answer must sound neat" is not a check. "There is a departure time in it and no other" is. For each question choose a word or number that belongs in it, and sometimes a word that must not be in it.

Choose questions that matter. A deviation about the ferry costs a missed boat. A deviation in the salutation costs nothing. Keep the set small enough to run every week.

Write the answers yourself. Don't let the model you are about to test make the set. Then you test the model against its own opinion.

Let the set grow with every mistake. Every time a customer or Joost finds a mistake in what the agent made, the question that would have caught it goes in.

Counting instead of feeling

Joost can read the set every month and say "it feels different". That is too vague to act on. A count is sharper: last time four of the four answers were right, now two, and the two that changed are the deposit and the cancellation.

Daan wrote a small script that puts every question to the model, checks whether the requested word is in it and the forbidden word is not, and compares the result with last time. That's all it is: a sum, not a model.

Trying it out with a stand-in model produced an instructive case. The new model answered on the deposit with "a quarter", where the script looked for "25 percent". Substantively the same, so a deviation in wording. On the cancellation the new model said "free cancellation". That is not wording. That is a mistake a customer can count on and that costs money.

First conclusion: a deviation is a question, not an alarm. Second: telling the two apart is done by a person. For changed wording you adjust the set. For a substantive deviation on a question that costs money, the owner decides.

What you do with a deviation

Four steps.

First look at the answer itself: different wording or different content? Then look in the log at which model version gave the answers, and whether it changed between two runs. That is why the agent notes for each answer which model version it ran on: without a version you can't explain the difference. Then decide who determines what happens. A substantive deviation on a question that costs money goes to Anneke. For wording, Joost adjusts the set.

And pin down what you can. If your provider offers a version you can pin, choose it for the agents where a deviation is expensive. A pinned version disappears one day, so note when you must replace it and run the set on the new version before you switch.

The instruction as a source

The same set catches the drift from within. Run it after every change to an agent's instruction, and not only when the model changes. A sentence you added because a customer complained must not break a different answer.

At Boomsma Reizen this brought out something Daan hadn't expected. When Anneke wanted to know what the agency would lose if the service holding the customer file and the templates stopped tomorrow, Daan could export the customer file and copy the templates. The instructions of the three agents lived only in the settings of the agents themselves. There was no text file of them anywhere. Whoever loses the agent loses its instruction, and with the instruction a year of fine-tuning.

It is one of the few parts that are truly yours and yet live with someone else. An agent consists of parts. The model belongs to the supplier. The instruction, the example sets, the log, the templates, the customer file and your checks are yours. Together the instruction and the set are what makes your agent different from the same agent at another agency: what you taught it and how you know it still does it.

The habit that followed: every instruction lives as a text file in a place Daan manages, with a line per change about what changed and why. You change the file first and only then copy it to the agent. Then the file is the source and the agent the derivative. In three months that line is the only thing you'll have left.

Does the instruction also fit another provider?

An instruction in a file is not yet a way out. The way out is that you have seen it work at another provider.

Run the same set on the model of your second provider and read what differs. Sometimes it is little. Sometimes a sentence in the instruction has to change, because another model reads it differently. It doesn't have to be finished. It is enough that you know how much work it is. An instruction that takes a day of adjustment on a second provider gives you a different position than one that takes a month, or one where you don't know how much work it is.

How often, and by whom

At Boomsma Reizen Daan's script runs the set every night and reports a deviation to Joost. Daan also runs it by hand on the day a provider announces a new model.

If you have no watch, put a reminder in your calendar: the same four questions every month, ten minutes of work. What you miss in drift costs more than those ten minutes.

::: vandaag To do today

Choose the agent for which a mistake in its answer costs the most. Write down four questions from its real work, with for each question a word or number that should be in the right answer. Have a person check the answers the agent gives now. If they are right, keep them as the comparison. Put the questions to the model you use now and note the date and the model version. Set a reminder for a month from now.

Then take that agent's instruction out of its settings and put it in a text file. That is a job of a quarter of an hour, and it is the beginning of a way out. :::

Where to start

All five parts come down to the same thing: an agent that drops out is not a technical incident but a gap in what you have agreed. Choose one process your work leans on most, write down who notices when it stops, try once whether the reserve works, and record who decides what if it lasts longer. That takes an afternoon. Anneke did exactly that after her Tuesday, and it was no more than that.

Do you want to do this not alone but together with me in your own organisation? That is possible with an Expert Day.

All pieces