My First Automated Report Was Wrong for Two Weeks

Every Monday, my report landed in my inbox looking exactly the way I’d designed it. Clean formatting, confident summaries, numbers that told a tidy story about which channels were working. It had been doing that for two weeks before I realized it was wrong.

I’d built it to pull numbers from a few sources and summarize them automatically. The problem was in one of those sources, which was double-counting a chunk of traffic. The report was quietly overstating a channel that wasn’t performing nearly that well.

How it got past me

The report didn’t look broken, and that’s the part worth sitting with. A broken report throws an error or shows a blank chart. This one showed a plausible number with a confident sentence next to it, so there was nothing on the page to make me doubt it.

I’d also built the summaries to sound sure of themselves, which I now think was a mistake. A summary that reads confidently makes a wrong number feel checked. The formatting did half the work of convincing me.

What caught it

It wasn’t a check I’d built in. One channel’s number felt too good, so I went and cross-referenced it by hand against the source, and it didn’t hold up. Only then did the double-counting show up.

I’d like to say a test would have caught it, but I hadn’t written that test yet, and I wouldn’t have known which one to write until I’d seen this failure. That’s the part nobody tells you about building your own reporting.

The automation isn’t the hard part. Trusting it is, and you only earn that trust by breaking it a few times first.

I’m not sure I’d have believed that before this happened. It reads like a platitude until you’ve watched a polished report mislead you for two weeks. I also don’t know how long it would have gone on if that one number had been a little less flattering, and that question still bothers me more than the bug did.

The standard I hold now

I’d rather ship something honest and a little rough than something polished and wrong. That’s the standard I use for my own reports and for the ones clients see. A report that admits a source looks unreliable, or flags that a number hasn’t been verified yet, is more useful than one that smooths everything into a clean story, because the owner is going to make real decisions from it.

In practice it means I check new numbers against their raw source before a report goes out, and I treat any figure that looks surprisingly good with more suspicion than one that looks bad. Good news gets questioned harder, since nobody goes looking for a reason to doubt it.

If you want to see how I build that kind of trust into a report before it reaches a client, I’m happy to walk through it. And when you look at your own numbers, do you trust them, or do you just hope they’re right?