Pros/Cons Dual Best Effort ISP vs Fiber/MPLS.
-
@scottalanmiller said:
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't feel that "reasonable" downtime is a thing. It's reasonable at what cost. Reasonable changes based on the costs and options. It's a curve, not a singular value.
Example: Manager says "reasonable downtime is 15 minutes."
But what if halving the risk only cost $1? Would having 15 minutes of risk for $800 a month be reasonable if a risk of 7 minutes was only $801? Probably not, what was stated as reasonable would almost certainly be reckless when the whole picture was taken account.
No argument there. I use reasonable as a point of reference not necessarily an indication of cost.
-
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't see it as being an ass, it's just been asked before (though perhaps not in this thread) - I thought I had addressed it.
-
But getting management to give a "reasonable" number is a bad idea because reasonable doesn't exist until the market analysis and business needs are done. Really, only IT can determine what is reasonable and only the financial department can provide the necessary information for IT to make that call. Unless the two work together, IT can, at best, make wild ass guesses.
-
@scottalanmiller said:
But getting management to give a "reasonable" number is a bad idea because reasonable doesn't exist until the market analysis and business needs are done. Really, only IT can determine what is reasonable and only the financial department can provide the necessary information for IT to make that call. Unless the two work together, IT can, at best, make wild ass guesses.
Good point. I hadn't thought of it that way. I was thinking more along the lines of cost of downtime vs cost of a solution.
-
@scottalanmiller said:
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't feel that "reasonable" downtime is a thing. It's reasonable at what cost. Reasonable changes based on the costs and options. It's a curve, not a singular value.
Example: Manager says "reasonable downtime is 15 minutes."
But what if halving the risk only cost $1? Would having 15 minutes of risk for $800 a month be reasonable if a risk of 7 minutes was only $801? Probably not, what was stated as reasonable would almost certainly be reckless when the whole picture was taken account.
Great - how do you quantify the actual risk?
All I know right now is I pay $880 and in 7+ years I've had 15 mins of unplanned downtime.
I pay $120 at another location and I 3 different days at 4 hours per day of downtime.I won't post publicly dollar values of lost revenue.. not that I know what it is anyhow, the power that be won't share with me.
-
@Dashrender said:
At this point I also have to ask myself - are you saying these things just to rial me up, like JaredBusch seemed to be to Scott the other day?
Nah, if I wanted to do that, I would do other things. Got better things to do with my time than just rile you up.
I've dealt with this kind of mentality with bosses before. Really, you gotta break out of their mold and start breaking it down into a much more concise set of requirements.
Your cloud based EHR is your most important application, and if they are saying that nothing else matters then its time to ignore anything with onsite resources. That shit can be set it in the colo cage and is not the problem here. You need bandwidth to the cloud but cannot abide by any downtime. Since it's public traffic, there is no need for MPLS, fancy fiber loops between sites, or even the VPN solution I mentioned with Peplink. Your sites don't care if they can talk with each other, so any interoffice communication is pointless and unnecessary. You need interwebs and you need it to be working when you need it working. It sounds as though bandwidth is a pointless consideration, you can probably get away with 1.5Mbps loops for the amount of traffic you are probably generating.
There are two ways to handle this. One way is to add an additional high quality loop into the mix. This can be from a third party vendor, so you wouldn't have any layer 3 issues upstream, like if Cogent decided to stop routing traffic from one ISP. This is gonna cost you money because quality loops are not cheap. But this doesn't eliminate any layer 2 issues, like a cut fiber or central office burning to the ground. Even if you went with different mediums, like cable plant versus fiber versus traditional telco, you will still have layer 2 potential issues. The setup would be fairly simple, even if you didn't have fancy load balancers behind it. Worst thing that would happen would be you reload your ASA or whatever firewall you have with a config for the other ISP.
The other way is to go cheap but go many. Swarm the problem with super cheap pipes from multiple locations. A mix of commodity pipes with MVNOs and any other wireless vendor you can find. Sure, one pipe might go down, but odds are the Sprint, AT&T, and Verizon towers won't and you would still have other pipes to fall back on. This is a much more complex setup, but could be much cheaper for what you have now or move to with a second quality pipe. Failover would require more complex equipment, like the Peplink, but would keep downtime to a minimal.
You need clear directives from the bosses. If they are gonna shoot down anything that cost more than their current setup or focus and spend money on things that don't matter, then this is an exercise in futility because I wouldn't dare guarantee a cheaper solution because I know what it will entail. If you are requiring more bandwidth, the costs are going to increase no matter what. If ANY other items come in, like local services, then internet services are only a small part of the issue here. Then it's time to take a look at all your services, the business workflow, and what can be done to streamline everything and start doing things right. Moving to a colo with better pipes locally would really help.
-
@Dashrender Well you can make another wild {censored} guess like the rest of us do, lol.
But Here, I think is where the rubber meets the road. If what you are doing has the ability to improve internet speeds and (both cut cost and improve reliability [through use of a backup connection]) , how is that a bad business decision?
-
@Dashrender said:
Great - how do you quantify the actual risk?
All I know right now is I pay $880 and in 7+ years I've had 15 mins of unplanned downtime.
I pay $120 at another location and I 3 different days at 4 hours per day of downtime.Risk is difficult to assess, especially in a case where you have to consider more than technology. The biggests risks that you have to address here are around scope (you are dealing only with service risk, but what if your building burns down or your own equipment fails!!), risk around businesses going under or changing policies, risk that you will be considered a pain to fix and traditional downtime could suddenly become insane downtime, lock in risk, risk because of service quality issues, potential downtime, likely downtime.... it's hard, very hard. That's why risk analysis is one of the absolute top skills for IT decision makers. It is complex and often the most important value that we provide to the business - without risk analysis they could just pick a solution themselves without IT being needed.
Risk is a bit of a soft science, but understanding the range of risks, a rough idea of likelihood, what risks apply how heavily to the business in question, how well mitigation factors can or should work, etc. is what IT provides and while far from perfect, it can be pretty decent.
And it is important to remember that we are only coming up with risk. And this is multidimensional risk not just a 20% kind of number.
-
@Dashrender said:
I won't post publicly dollar values of lost revenue.. not that I know what it is anyhow, the power that be won't share with me.
Ok, that answers part of it... you have no way to calculate the financial risk of downtime if those people won't work with you.
Just some basic math on the links alone (hopefully I did them correctly) The first one costs you ~$1.21/hour. Meaning you lost ~$0.30 on those 15 minutes of downtime. The second one costs you ~$0.16/hour. You've lost $1.92 for all those 12 hours you were down. In total though you've spent $73,920 on link 1 and $10,080 on link 2. Just something I thought was interesting.
-
@Dashrender said:
Great - how do you quantify the actual risk?
All I know right now is I pay $880 and in 7+ years I've had 15 mins of unplanned downtime.
I pay $120 at another location and I 3 different days at 4 hours per day of downtime.That's all very misleading data, sadly. Because a new connection will not necessarily have anything to do with what you have seen elsewhere. Might be the same, might be better, might be worse. And sadly, seven years of a fiber line is not enough to get insight into the failure rates because outages tend to be very long and very infrequent. Like a volcano.
Take this scenario..... if you were to have lived in San Francisco for three generations you would report that while there are tremors and little earthquakes, there is nothing scary ever. A little concerning maybe, but not scary of dangerous. Yet if you take a long view, scientists know that an earthquake levels the city and kills a huge number of people every 110 years or so. It's very regular and very dangerous. But 90 years of observation would make it seem like this wasn't a real risk.
When dealing with infrequent risks, observation is a dangerous thing.
-
I really don't think most business need dual links. These days a good ISP rarely goes down when it's not planned. The only reason we had it at the town was because of the police department there are things they need in an emergency, information on past crimes as well as facebook (which they can only get their special access to inside the network) Facebook is become more valuable that most people realize in detective work and active shooter situations.
-
@thecreativeone91 said:
I really don't think most business need dual links. These days a good ISP rarely goes down when it's not planned. The only reason we had it at the town was because of the police department there are things they need in an emergency, information on past crimes as well as facebook (which they can only get their special access to inside the network) Facebook is become more valuable that most people realize in detective work and active shooter situations.
I was thinking this as well. If there was a good wireless provider in the area get a 4G/3G modem in the mix as failover and you eliminate most ISP related downtime. I rarely hear of cell towers going "down".
-
@coliver said:
@thecreativeone91 said:
I really don't think most business need dual links. These days a good ISP rarely goes down when it's not planned. The only reason we had it at the town was because of the police department there are things they need in an emergency, information on past crimes as well as facebook (which they can only get their special access to inside the network) Facebook is become more valuable that most people realize in detective work and active shooter situations.
I was thinking this as well. If there was a good wireless provider in the area get a 4G/3G modem in the mix as failover and you eliminate most ISP related downtime. I rarely hear of cell towers going "down".
They do, but one of the great things is they tend to go down from completely different events than land lines tend to go down from.
-
@thecreativeone91 said:
I really don't think most business need dual links. These days a good ISP rarely goes down when it's not planned. The only reason we had it at the town was because of the police department there are things they need in an emergency, information on past crimes as well as facebook (which they can only get their special access to inside the network) Facebook is become more valuable that most people realize in detective work and active shooter situations.
I would agree with that, that most do not. Same factors that make most appropriate for consumer or prosumer links.
-
@scottalanmiller said:
@coliver said:
@thecreativeone91 said:
I really don't think most business need dual links. These days a good ISP rarely goes down when it's not planned. The only reason we had it at the town was because of the police department there are things they need in an emergency, information on past crimes as well as facebook (which they can only get their special access to inside the network) Facebook is become more valuable that most people realize in detective work and active shooter situations.
I was thinking this as well. If there was a good wireless provider in the area get a 4G/3G modem in the mix as failover and you eliminate most ISP related downtime. I rarely hear of cell towers going "down".
They do, but one of the great things is they tend to go down from completely different events than land lines tend to go down from.
A lot of cell towers around here aren't running generator backups so they go down often.
-
The thing about a cell tower outage is that nearly all people just see it as a "gap in coverage." It's not like losing a terrestrial line, it appears very differently to the end users.
-
@scottalanmiller said:
The thing about a cell tower outage is that nearly all people just see it as a "gap in coverage." It's not like losing a terrestrial line, it appears very differently to the end users.
But not to a non-mobile secondary link. In that case it would be considered down. Although as you said, generally ISPs and cell service don't go down at the same time.
-
@thecreativeone91 said:
@scottalanmiller said:
@coliver said:
@thecreativeone91 said:
I really don't think most business need dual links. These days a good ISP rarely goes down when it's not planned. The only reason we had it at the town was because of the police department there are things they need in an emergency, information on past crimes as well as facebook (which they can only get their special access to inside the network) Facebook is become more valuable that most people realize in detective work and active shooter situations.
I was thinking this as well. If there was a good wireless provider in the area get a 4G/3G modem in the mix as failover and you eliminate most ISP related downtime. I rarely hear of cell towers going "down".
They do, but one of the great things is they tend to go down from completely different events than land lines tend to go down from.
A lot of cell towers around here aren't running generator backups so they go down often.
Ouch, that could be an issue for sure.
-
@coliver said:
@scottalanmiller said:
The thing about a cell tower outage is that nearly all people just see it as a "gap in coverage." It's not like losing a terrestrial line, it appears very differently to the end users.
But not to a non-mobile secondary link. In that case it would be considered down. Although as you said, generally ISPs and cell service don't go down at the same time.
Still depends, coverage varies all the time and an outage of a tower might only look like a drop in quality.
-
I just had an interesting conversation at lunch.
So far, pretty much everyone here (as far as I can tell) have considered my idea of dumping the expensive fiber line and getting two commodity ISP links as a general good move. Of course a few of you have said something to the effect - prove to me that not being down is worth the extra costs - and I'm understanding that.
So, in my lunch conversation, and PSX will love this, my associates told me I was looking at it all wrong (granted you all have to, just for different reasons). To them the fact that I am even considering getting rid of the fiber connection is unthinkable. Why in god's name would I get rid of a pipe that has proven itself (yes I read your post Scott) so reliable. Instead of getting rid of that, if down time is really an issue, or as it appears, additional bandwidth is needed/desired, but the cheap best effort connection in addition to your current fiber connection, route your EHR and SIP traffic through that (with failover to the best effort) and push everything else off to the other pipe.
So instead of saving my company nearly $600/month, raise my price by $200.
Thoughts?
