The Day The Internet Broke
-
It was mid-day on a normal Wednesday when I started getting alerts that devices have fallen offline at one of our locations. Seconds later the alerts about the firewall at this location falling offline were hitting my Inbox. Something was awry, and I needed to determine exactly what happened. My team and I are physically stationed at the company headquarters but support several remote sites in the area (and some out of state).
I called a user at the location in question (which is just over 30 miles from my office) to see if the facility still had power. He said there were no power issues and even went into our telco room at this facility to check equipment lights. Everything seemed normal on our UPS (not running on battery), firewall, switches, and the ISP's modem (all lights for US/DS and online solid blue).
This location has about 20 users and uses Time Warner coax as their internet connection. For the most part the connection has been reliable, but it is still coax and not as reliable as fiber. Since we couldn't ping the ISP's device at that location, I had the user power cycle the modem to see if that would fix the problem. Once the modem was online again I could ping it for about 15 seconds and then never again. Our firewall never came online, and folks at the location could not connect to the internet.
My mind immediately jumped to possible ISP issues, and I called Time Warner support to see if they might have an outage in the area. After giving the technician all the account information and describing the problem, I hear him pause and then say, "oh, there's the problem. Your account has been issued a soft disconnect because your account is overdue." Honestly that's the last thing I thought could have been the problem in this situation.
The technician transferred me to someone in billing who told me the account was 3 months overdue, and we owed over $1000 to them. I check with Finance, look through all of the billing statements, and find we issued a check on 8/18 for over $1000, which Time Warner had not received and had not yet processed. Even if they had received it, the processing of the check and applying it to the account takes time. Waiting on the check to process was not an option.
Luckily, I was able to pay the minimum amount to get us back online again (a little shy of $700). After the person in billing processed the payment, they had our internet circuit turned on again within 5-10 minutes. Once they receive our check and process it, we'll have some credit on our account for a couple of months.
After all of this I went and spoke with Finance about the situation one more time. It looked like we had received the invoices but that they had been bouncing them around to different people to approve before issuing the check, which caused a bit of a delay. In any case, they are going to look to pay this specific bill using an auto-draft so we won't run into this problem again.
At your company, have you ever had this happen? Do you have a role in approving the bills for recurring charges like internet and phone? What's your process?
-
I've had an approval process before to get things paid that are reoccurring. AV was one example, annual bill.
Every time "Dustin do we really need AntiVirus?" Yes, yes we do, Its approved "OK well I just have to check with one more person..." they come back to me "Dustin do we really need AntiVirus?"
See how this goes....
-
@DustinB3403 said in The Day The Internet Broke:
I've had an approval process before to get things paid that are reoccurring. AV was one example, annual bill.
Every time "Dustin do we really need AntiVirus?" Yes, yes we do, Its approved "OK well I just have to check with one more person..." they come back to me "Dustin do we really need AntiVirus?"
See how this goes....
It's amazing with all the news of attacks, virus, worms and such they want to know if you really need an AV program.
Ask them if they really need
- insurance on their car, house or family...
- a fire extinguisher or the fire department.
- etc...
-
@gjacobse That is what it essentially came down to.
"Do you really need a computer when you can work with pen and paper, Oh you do need a computer, OKAY renew the AV licensing and stop arguing about it."
-
@NetworkNerd I'm very lucky in this regard currently. I get "Travis, do we need this?" "Yes!" "OK, get it done!"
My first experience in IT, it was months of wrangling with people to get anything done. So many hands in the pie, the person that actually needed the widgets would give up. They had the main file server not completely boot over Y2K. The one box that every single person in the facility needed access to. It only took them 8 months to replace it.
-
Actually I'm starting to question the need for AV. That's not to say I don't use it, I do. But more and more attacks, modern ones, are getting around AV with little to no trouble.
Stopping things at the perimeter seems to be a better defense, but that's practically impossible.
-
@Dashrender said in The Day The Internet Broke:
Actually I'm starting to question the need for AV. That's not to say I don't use it, I do. But more and more attacks, modern ones, are getting around AV with little to no trouble.
Stopping things at the perimeter seems to be a better defense, but that's practically impossible.
It's all about a layered defense now. Perimeter firewall, IDS/IPS, local firewall and antivirus. The problem is that many anti-virus programs actually make the system more vulnerable. By their nature AV has to have access to everything on the system, and hacking tools can use them as a fast-lane to system access.
Just one of many reasons I'm glad we use Mint/CentOS wherever possible. The security risks aren't any less, but they are different. Different in a way that's easier for me to prevent against bad actors.
-
@travisdh1 said in The Day The Internet Broke:
Just one of many reasons I'm glad we use Mint/CentOS wherever possible. The security risks aren't any less, but they are different. Different in a way that's easier for me to prevent against bad actors.
At least you understand that running other OSes than Windows is not the panacea that a lot of people think it is.
-
@dafyre said in The Day The Internet Broke:
@travisdh1 said in The Day The Internet Broke:
Just one of many reasons I'm glad we use Mint/CentOS wherever possible. The security risks aren't any less, but they are different. Different in a way that's easier for me to prevent against bad actors.
At least you understand that running other OSes than Windows is not the panacea that a lot of people think it is.
What??? OSX can get viruses??? haha!
-
@dafyre said in The Day The Internet Broke:
@travisdh1 said in The Day The Internet Broke:
Just one of many reasons I'm glad we use Mint/CentOS wherever possible. The security risks aren't any less, but they are different. Different in a way that's easier for me to prevent against bad actors.
At least you understand that running other OSes than Windows is not the panacea that a lot of people think it is.
For sure. Nothing like comparing ls to what is in a repository to find it's been replaced by a rootkit. Yep, I've seen this happen before.
