Was looking at flights for Mangocon, then this happened...
-
Slow day, was going to go home early. Decided not to. About 30 seconds after i was looking at flights to Rochester from Seattle, i start hearing users mumbling about things not working.
My stuff seems to be working fine, but i go out and see whats happening. Sure enough, people's Outlook is disconnected. then their phones stop working(voip). I go back into my office and suddenly nothing works. I mean nothing. Cant ping any device at all. However, our wifi network is working well. It is on a different vlan and subnet, gets its dhcp from Watchguard.I go in server room, check to make sure everything is still on. It is, so i then move my pc's network port to a different switch. Success, i can ping our exchange, firewall, and google... for about 3 seconds. Then nothing again.
I log into our sole remaining physical Domain Controller, take a look at dhcp, dns, network traffic. Something is sending out a ton of traffic from 192.168.xxx.255 "WTF" i said very loudly. Perhaps we got hit with a virus of some kind.I think to go downstairs and ask if the engineers have plugged anything into the network(we make network monitoring equipment). At any given time the engineers and tech support folks have at least half a dozen devices plugged in, mostly to unmanaged switch devices. Not ideal, but it has been like that for at least a decade and i just started here. I go to this one gal's workstation,
ping google
No resources.
No resources.
No resources.
No resources.Been doing this quite a while, never seen ping return that result. As such it give me an idea; i unplug the unmanaged switch at her desk. 30 seconds later, everything is back to normal. Got a bit lucky that i went to this engineers station first and did a ping test and saw something id never seen before, or I would probably be still be pulling hair out.
It's a good thing i didnt decide to go home early, or the office might have caught on fire or something else like that. -
Wow, that is an incredibly hard one to have found!
-
Yes, i think i am going home early today, for real this time. Last night i read more about icmp, there is really a lot going on with that old protocol. 99.99% of the time you can get your troubleshooting done with Reply and Unreachable. but there is much more.
-
So, did they create a loopback on the unmanaged switch? I had a client do that once, brought down the whole network for about an hour until I found it.
Definitely considered myself lucky finding it inside an hour.
-
@Dashrender said in Was looking at flights for Mangocon, then this happened...:
So, did they create a loopback on the unmanaged switch? I had a client do that once, brought down the whole network for about an hour until I found it.
Definitely considered myself lucky finding it inside an hour.
I would think that's the case as well.
