Four bugs from my thesis that taught me more than the results
My thesis has a results chapter, but the bugs taught me more. Here are four.
The fake bottleneck. My storage step looked slow: a write took 47 ms. I started to look for a fix in my code. Then I measured the same write on Linux and it took 5 ms. The slowness was Docker Desktop on Windows, not my code. Lesson: measure on the machine that matters before you optimise.
More machines, less speed. I split the work equally over five machines. It ran at 29.7 messages per second, slower than two machines at 61.1. The fast machines finished their share and then waited for the slow ones. Sharing the work by machine power fixed most of it. Lesson: a cluster is as fast as its slowest equal share.
The silent Kafka limit. A Raspberry Pi needs about 800 ms per message. With the default max_poll_records=500, one batch took longer than the 5 minute poll limit, so Kafka decided the Pi was dead and gave the work out again, and the group kept thrashing. Setting the batch size lower for slow hardware fixed it. Lesson: a silent loop is often a timeout between two settings.
The network clash. The Raspberry Pi boards could not be reached. Docker’s default network, 172.17.0.0/16, collided with the Pi subnet. Moving Docker’s range made the boards reachable. Lesson: when machines cannot see each other, compare the network ranges first.
The full system and results are on the research page.