Pros/Cons Dual Best Effort ISP vs Fiber/MPLS.
-
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
Something that PSX is clearly not understanding is that Email does not matter, local files do not matter, printing really doesn't matter. Two things in our environment really matter for operations to continue.
- phones. Yesteryear I'd say if the phones weren't working we'd close the office because we couldn't reach 911 if there was an emergency, but today that's not a problem because 80% or more of our staff have cellphones in their pockets and can call 911.
- Internet access. Our main app (EHR) that everyone in the company uses (except me, and I do to support them... soooo... ) is part of a service that we subscribe to. It's hosted, managed, etc all by them. So, as long as I have internet access, I can access my EHR.
PSX mentioned BOYD, my $11/hr employees are lucky if they have a desktop at home, let alone a laptop they can bring into the office. Furthermore, I can be assured that they do and will have constant problems (I know because they are always asking for help on their personal machines). Additionally, management does not feel right asking employees to use their personal equipment to do company business - and employees that have volunteered to use their personal equipment because it will make their lives easier have nearly demanded reimbursement for said usage.
Our external email usage is around 100 Megs a day, moving email to the cloud would actually drastically increase our internet usage because now all internal email would have to traverse our internet connection, granted it probably wouldn't be much, maybe increase to 500 megs a day, but it would be much worse.
We had a remote site that lost internet about a month ago, clinic was scheduled for one physician from 2-5. When the internet was not back in service at 2:15, the rest of the clinic was canceled, and the patients rescheduled. those patients that were already onsite were seen in a severally reduced capacity, and possibly rescheduled again if specific information wasn't available at the time. Fortunately this has only happened at our remote clinics about 3 times in 7+ years and has been considered acceptable. That said, there have been many more than 3 outages, there have been well over 15 between all of the remote locations, fortunately it seems the HFC plants don't all die at once in this city.
Now Scott will definitely tell me that I'm the wrong person to be driving redundant ISPs. The business should be telling me - hey it costs us x millions of dollars per hour that we are down, so it makes sense for you to spend y thousand dollars to help us reduce the downtime risk.
Unfortunately I'm not sure I can give you the lost revenue, and the reality is that 95% of canceled appointments that were going to happen (we have a noticeable no show rate) will definitely reschedule in the near future, and the cost is the physician's personal time while they squeeze those patients in to already booked days, usually at little or no staff time cost. Of course there's staff time lost during the outage itself which is not spent doing work because the system is down, but there is enough slack time in their schedules to do any additional work needed without paying overtime (yeah that means we are probably overstaffed - another topic).Have we been over paying for internet on fiber for the past 7+ years, well maybe not the whole time, but for most.. probably. The dual ISP solution has been available for quite some time I'm sure, but hasn't been a consideration until I started looking at my phone situation.
Things are getting a little disjointed.. hope you can just make due.
I feel comfortable telling the BOD that Best Effort ISP solutions WILL give them at least 3 downtime periods per year, probably averaging 4 hours. This seems to be the experience over all of our sites the last several years. With that in mind, and considering that our current Fiber ISP costs $880, and the new 50/10 connection will be $180, I feel comfortable given that information, that the BOD would tell me to get a second ISP installed at $120/month as a failover (12/2 best thing we can get at Best Effort at this location). The inconvenience factor alone would be worth saving to them, even if the clinic wasn't losing more than $1440/yr (plus taxes and the cost of the Peplink).
-
@Dashrender said:
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
If you don't have that info, who is making the financial decisions on technology spends?
-
@scottalanmiller said:
@Dashrender said:
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
If you don't have that info, who is making the financial decisions on technology spends?
Really, truly, me. I do what feel is the best solution at the cheapest price, I'm providing best effort considering the situation. That said, large expense, my phone project, this internet change, are run past the BOD. Sometimes they ask some questions, once they ask me to fine more information, but generally they say, what do you think, OK, do that.
Has it lead to overspending - clearly it has, we've had this expensive fiber for this whole time, perhaps never needed it, but like SMBs who get suckered into a SAN, I probably did get suckered into this Fiber because I was/am concerned about keeping us online as much as possible.
On a side note, when we cancel a clinic at a remote site, we're canceling no more than 10 patients, when we have problems at our main location that could could more like 40 in AM or PM, not to mention our ambulatory surgical center (ASC) (outpatient surgery), which if we are done, we finish the surgeries we are in, and stop until it's restored, or cancel the rest of the day. This is were the big dollars can be lost (I really should have mentioned this side earlier) there is only so much time available to book the ASC, and while I'm sure there is a small amount of slack time, it's generally pretty well booked, so we could definitely end up seeing some pretty substantial losses if we have to cancel those surgeries. So the justification of $1500/yr for a backup ISP is an easy one just considering the ASC.
-
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
If you don't have that info, who is making the financial decisions on technology spends?
Really, truly, me. I do what feel is the best solution at the cheapest price, I'm providing best effort considering the situation. That said, large expense, my phone project, this internet change, are run past the BOD. Sometimes they ask some questions, once they ask me to fine more information, but generally they say, what do you think, OK, do that.
Has it lead to overspending - clearly it has, we've had this expensive fiber for this whole time, perhaps never needed it, but like SMBs who get suckered into a SAN, I probably did get suckered into this Fiber because I was/am concerned about keeping us online as much as possible.
On a side note, when we cancel a clinic at a remote site, we're canceling no more than 10 patients, when we have problems at our main location that could could more like 40 in AM or PM, not to mention our ambulatory surgical center (ASC) (outpatient surgery), which if we are done, we finish the surgeries we are in, and stop until it's restored, or cancel the rest of the day. This is were the big dollars can be lost (I really should have mentioned this side earlier) there is only so much time available to book the ASC, and while I'm sure there is a small amount of slack time, it's generally pretty well booked, so we could definitely end up seeing some pretty substantial losses if we have to cancel those surgeries. So the justification of $1500/yr for a backup ISP is an easy one just considering the ASC.
Some items, like the ISP, is probably pretty easy as you have so few choices to work with. It's more when you have to decide on things like colocation, system architecture, hardware purchases, support contracts, etc. where there are far more granular choices and the decisions tend to come down to careful financial comparisons.
-
I've definitely learned a lot and seen mistakes I've made where I could have saved the company a ton of money recently. I will take those lessons with me to the next project and hopefully continue to make wiser choices for us.
That said, overall considering my experience I think I've done a pretty good job. We've had no major outages/issues since I've taken over IT here.
-
@Dashrender said:
Something that PSX is clearly not understanding is that Email does not matter, local files do not matter, printing really doesn't matter.
If that's the case, then your job is redundant. If phones and internet access are the ONLY driving factors in your situation then smurf it out with a set of phones. You get internet access and phones that will always work, and if one provider doesn't, another one will.
Your business case is cost is the only driving factor, to hell with everything else. Well, given that a bunch of cell phones will run ya ~$40 a line for some MVNOs, you are going even cheaper.
Here's the rub, you are so hyper-focused on cost you are lost on reliability, scalability, and the actual bandwidth usage of your environment. There is no way you are only using 100MB of emails a day unless you have two people using emails. If that's the case, then KILL THE MAIL SERVER and move to a cloud environment. And if internet is absolutely required, then going cheap is NOT the way to go. I would be calling some CLECs to get me multiple pipes into the environments. TWTelecom is always dropping pipes in, Covad/Megapath services almost every CO in the country.
Risk is something you get with cheap pipes. If your bosses are so risk adverse that they cannot stand to have even the millisecond of downtime, then it's gonna cost them not only in proper loops with SLAs and such but having multiple loops with SLAs and such. You gotta get real with them, lest they get screaming at you because they had a blip on these el-cheapo pipes.
-
Yes I'm (and my management) are extremely price conscious. But clearly we are willing to pay when we think (rightly or wrongly) we need expensive services. i.e. the fiber link we currently have.
We really don't use email at this company. Outlook is more about calendars for the physicians to sync to their mobile devices than any other purpose. If not for the need for calendar access, we'd go from 88 people having email to around 20.
Internet is critical, but we can suffer some downtime, we are a slightly higher requirement than average SMB (yeah I know "all SMBs think they can afford to suffer no down time" lol). I've already mentioned that we've suffered down time at the remote locations and yet, the BOD has not come to me saying - You need to make sure we don't have downtime. That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
At this point I also have to ask myself - are you saying these things just to rial me up, like JaredBusch seemed to be to Scott the other day?
-
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
-
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
Because the vendors won't provide failure rates on Best Effort solutions, the best you can go by is past experience. I can look at my tickets and see how many failures we've had on average over the past three years at each remote location, then make an educated guess as to what will probably happen in the future. Of course it's not perfect, it COULD be completely wrong, past performance can't predict future situations. But it's not simply a gut feeling though.
I know that on average we have 3 outages a year at each remote location, so assuming that to be the case, I hope and anticipate that to be what happens in the future. So now I consider options to mitigate those 3 failures, which is the purchase of a second line.
Let's leave the colo stuff out of this as it's not important to the current needs, which is protecting internet access for access to our EHR.
-
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
-
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
-
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
The starting point is knowing the cost of downtime. IT exists within a business context, so knowing the business is critical to understanding IT's place within it. Knowing what the impact of slowness, flakiness and outages is important (as they vary from company to company.) Once you have a good financial understanding of how telecommunications relates to the business then you can do risk analysis and cost analysis of the options and determine which make the most sense based on the business' unique needs.
-
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
Because the vendors won't provide failure rates on Best Effort solutions, the best you can go by is past experience. I can look at my tickets and see how many failures we've had on average over the past three years at each remote location, then make an educated guess as to what will probably happen in the future. Of course it's not perfect, it COULD be completely wrong, past performance can't predict future situations. But it's not simply a gut feeling though.
I know that on average we have 3 outages a year at each remote location, so assuming that to be the case, I hope and anticipate that to be what happens in the future. So now I consider options to mitigate those 3 failures, which is the purchase of a second line.
Let's leave the colo stuff out of this as it's not important to the current needs, which is protecting internet access for access to our EHR.
This doesn't really answer the question. You are looking a failure rates of a line not at the numbers associated with reasonable downtime.
Do you have good cell reception where you are?
-
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't feel that "reasonable" downtime is a thing. It's reasonable at what cost. Reasonable changes based on the costs and options. It's a curve, not a singular value.
Example: Manager says "reasonable downtime is 15 minutes."
But what if halving the risk only cost $1? Would having 15 minutes of risk for $800 a month be reasonable if a risk of 7 minutes was only $801? Probably not, what was stated as reasonable would almost certainly be reckless when the whole picture was taken account.
-
@scottalanmiller said:
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't feel that "reasonable" downtime is a thing. It's reasonable at what cost. Reasonable changes based on the costs and options. It's a curve, not a singular value.
Example: Manager says "reasonable downtime is 15 minutes."
But what if halving the risk only cost $1? Would having 15 minutes of risk for $800 a month be reasonable if a risk of 7 minutes was only $801? Probably not, what was stated as reasonable would almost certainly be reckless when the whole picture was taken account.
No argument there. I use reasonable as a point of reference not necessarily an indication of cost.
-
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't see it as being an ass, it's just been asked before (though perhaps not in this thread) - I thought I had addressed it.
-
But getting management to give a "reasonable" number is a bad idea because reasonable doesn't exist until the market analysis and business needs are done. Really, only IT can determine what is reasonable and only the financial department can provide the necessary information for IT to make that call. Unless the two work together, IT can, at best, make wild ass guesses.
-
@scottalanmiller said:
But getting management to give a "reasonable" number is a bad idea because reasonable doesn't exist until the market analysis and business needs are done. Really, only IT can determine what is reasonable and only the financial department can provide the necessary information for IT to make that call. Unless the two work together, IT can, at best, make wild ass guesses.
Good point. I hadn't thought of it that way. I was thinking more along the lines of cost of downtime vs cost of a solution.
-
@scottalanmiller said:
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't feel that "reasonable" downtime is a thing. It's reasonable at what cost. Reasonable changes based on the costs and options. It's a curve, not a singular value.
Example: Manager says "reasonable downtime is 15 minutes."
But what if halving the risk only cost $1? Would having 15 minutes of risk for $800 a month be reasonable if a risk of 7 minutes was only $801? Probably not, what was stated as reasonable would almost certainly be reckless when the whole picture was taken account.
Great - how do you quantify the actual risk?
All I know right now is I pay $880 and in 7+ years I've had 15 mins of unplanned downtime.
I pay $120 at another location and I 3 different days at 4 hours per day of downtime.I won't post publicly dollar values of lost revenue.. not that I know what it is anyhow, the power that be won't share with me.
-
@Dashrender said:
At this point I also have to ask myself - are you saying these things just to rial me up, like JaredBusch seemed to be to Scott the other day?
Nah, if I wanted to do that, I would do other things. Got better things to do with my time than just rile you up.
I've dealt with this kind of mentality with bosses before. Really, you gotta break out of their mold and start breaking it down into a much more concise set of requirements.
Your cloud based EHR is your most important application, and if they are saying that nothing else matters then its time to ignore anything with onsite resources. That shit can be set it in the colo cage and is not the problem here. You need bandwidth to the cloud but cannot abide by any downtime. Since it's public traffic, there is no need for MPLS, fancy fiber loops between sites, or even the VPN solution I mentioned with Peplink. Your sites don't care if they can talk with each other, so any interoffice communication is pointless and unnecessary. You need interwebs and you need it to be working when you need it working. It sounds as though bandwidth is a pointless consideration, you can probably get away with 1.5Mbps loops for the amount of traffic you are probably generating.
There are two ways to handle this. One way is to add an additional high quality loop into the mix. This can be from a third party vendor, so you wouldn't have any layer 3 issues upstream, like if Cogent decided to stop routing traffic from one ISP. This is gonna cost you money because quality loops are not cheap. But this doesn't eliminate any layer 2 issues, like a cut fiber or central office burning to the ground. Even if you went with different mediums, like cable plant versus fiber versus traditional telco, you will still have layer 2 potential issues. The setup would be fairly simple, even if you didn't have fancy load balancers behind it. Worst thing that would happen would be you reload your ASA or whatever firewall you have with a config for the other ISP.
The other way is to go cheap but go many. Swarm the problem with super cheap pipes from multiple locations. A mix of commodity pipes with MVNOs and any other wireless vendor you can find. Sure, one pipe might go down, but odds are the Sprint, AT&T, and Verizon towers won't and you would still have other pipes to fall back on. This is a much more complex setup, but could be much cheaper for what you have now or move to with a second quality pipe. Failover would require more complex equipment, like the Peplink, but would keep downtime to a minimal.
You need clear directives from the bosses. If they are gonna shoot down anything that cost more than their current setup or focus and spend money on things that don't matter, then this is an exercise in futility because I wouldn't dare guarantee a cheaper solution because I know what it will entail. If you are requiring more bandwidth, the costs are going to increase no matter what. If ANY other items come in, like local services, then internet services are only a small part of the issue here. Then it's time to take a look at all your services, the business workflow, and what can be done to streamline everything and start doing things right. Moving to a colo with better pipes locally would really help.
