Backup Automation Code Crash

So… Many moons ago, I ended up writing a fairly short Perl program - maybe 150 lines. Its purpose was to automate several functions of our backup system that the software package didn’t provide in a reasonable manner.

So it goes into production, and runs every morning at 8:03AM. Works perfectly for some 6–7 years. And then we have a procedure change, and I have to take a line of code that said “if A is equal to B1, B2, or B3, do this”, and change it to “if A is equal to B1, B2, B3, or B4, do this”.

And while I’m there, I notice that another if statement a few lines down is backwards. And it’s the most crucial if statement in the whole 150 lines, and the code can’t possibly work - the program would obviously crash in a very specific way.

So I carefully quit out of the editor, and check… Yes, the datestamp on the production code is over 6 years back, and it’s run correctly at 8:03AM every morning for 6 years. And the if statement is backwards.

I get 4 co-workers to look at it individually, and they all agree the if statement is backwards, and it should be crashing in a very specific way, and nobody understands how this worked for 6 years.

The next morning, I get an email from our monitoring software at 8:04AM saying that the program crashed. And sure enough, the datestamp is 6 years back, the if statement is backwards, and it crashed exactly the way you’d expect it to crash if it was backwards.

I went in, reversed the if statement to be correct, added B4 to the other if statement, and the program continued to work properly until we decommissioned that backup system 5 years later.

I’m still mystified how that line of code worked until somebody looked at it.