Divide and Conquer

We fixed the internet today. By we, I mean the network engineer Shawn and me. It was a long, somewhat painful process, but oh so much fun when we finally narrowed it down and found the problem. First thing Shawn wanted to know was where we received the internet. I showed him, and we began to run some tests. The main switch that receives the fiber internet from our ISP split into four different directions onto the main switch, and we disconnected all of them and plugged them in one by one until we discovered the problem. Three out of the four worked fine — they connected to the internet and everything seemed back to normal. The problem fiber branched out to the Middle School, and that was our next stop.

We went to the Middle School switch closet, and sure enough, we saw that one of our two switches wasn’t showing any network activity. Fortunately, that meant we narrowed it down. Unfortunately, this switch is the only one on the whole campus that has 48 ethernet ports; all the others one only have 24. Not only that, nothing was labelled, nothing was bundled together, and everything was messy and unwieldy. So what we did was take a pad of paper, follow each cord back to the patch panel, which luckily did have rudimentary labelling, and we wrote it down. All 48 cords. We then disconnected each one and started connecting them back one by one. He ran an infinite ping on his laptop, and we began with port 1. Then port 2. Then port 3, and we kept going until we saw which one locked up the network and gave us the problems. At around port 12, we noticed our first problem. We took note of it and kept the cord disconnected. A few ports later, we found another problem. Then around port twenty-something, we found another problem. So three ports were giving us problems, and we didn’t know why. We kept going. We reached port 43 when the power went out.

We were pissed and very disappointed. We were almost there! But instead of wallowing in pity, we decided to go investigate the three ports that were giving us issues. Unfortunately, each one was located in a different room, so we went to the first room that had the problem and we investigated. We checked out the computer, saw that everything looked okay, but we still marked the computer. We went into the next room, did the same thing, and marked that computer. While we went to look for the third room, the power came back on and we were back in business. We went to the third room and investigated the last port that was giving us problems. I found the right port, followed the cord out and noticed it plugged back into the port right next to it instead of to a computer. That was our issue. That was our loopback that was causing all our issues. Someone, instead of connecting that cord into a computer, connected it back into the network, and that loop buckled our switches, overloaded the internet, and brought our school to its knees. We corrected this error, finished plugging in all our cords back into the switch, and tested the network. Everything was up and running and working like it should.

This took us most of the day to figure out, and by dividing and conquering, we were able to trace our problem back to the source. It was so much fun and so enlightening and my plans for this network changed completely. I’m going to play the maintenance game, organizing everything so this doesn’t happen again. And, fortunately¹, I have the full backing of my superiors. I know what to do next time, even though I hope there’s no next time. I’m glad we don’t have to replace anything, that everything’s working like it’s supposed to, and we can all get back to our jobs.

What a fucking trip.