A green test proves nothing until you have seen it go red
A green test proves nothing until you have seen it go red.
A smoke detector with a green light tells you it is on. Not that it works. You only know it works when you hold a bit of smoke under it and it starts to beep.
With software tests and with checks on agents it is exactly the same. A green result feels like proof, but it is only proof if you know the same test would have gone red if something were wrong.
So after every new test or check I do the same thing: I deliberately break the assumption.
- A limit per hour on the number of messages? Set it to 600 and see whether the test raises the alarm.
- A gate that stops leaks? Remove it and see whether the leak is found.
- A check on attachments? Reverse it and see whether that gets noticed.
And then the most important part: does the right check go red? Not just anything. There is a difference between "the suite fails" and "the test about this fails". The first can be caused by something else. The second proves that the test guards the behaviour I wanted to guard.
That turns something up more often than you would think. In a large batch of tests one pattern kept coming back: a part that is completely built, with all the right pieces in it, but that nobody calls. All the separate tests are green, because each test calls the part itself. It works in the test and nowhere else. You only find a fault like that by counting whether anything actually refers to it, and even then without counting the comments, because a call that is commented out does not count.
Another example is a difference of two: 1400 was expected and 1398 came out. You would never see that by glancing at the result. It was only found because the test was broken on purpose and then gave exactly the right message.
How do you tackle this in practice?
- Write the test.
- Break the code it guards, in a copy.
- See whether the test goes red, and whether it is the right one.
- Restore it and see whether it goes green again.
Four steps, ten minutes, and afterwards you know something you only hoped before.
How do you know a check in your process would really notice something?
#AIagents #Testing #Quality #Software
See also
I expect my agents to buy property and hire people within five years
I expect my agents to buy property, take over businesses and hire staff within five years, both agents and people. That is not a promise and not a plan. It is an expectation, and everything I build now is a step in that…
A file service of your own, built by an agent in nine minutes
Sharing files with people outside your organisation is often a hassle. Permissions in OneDrive or SharePoint, an account for someone who wants to download something just once, or an outside service where you do not know…
Two things you lean on: a green light and a report
When you let agents do the work, you get two kinds of signals back as a human. A test that shows green, and a report of what was done. You lean on both. And both can put your mind at ease without saying anything.
