Pros/Cons Dual Best Effort ISP vs Fiber/MPLS.
-
-
SOOOOOooooo.. no other Pros or Cons ???
-
@Dashrender said:
@thecreativeone91 said:
@scottalanmiller said:
@Dashrender said:
But just as bad, in the case of failure, how are you suppose to get back online? It would take days or more to download all of the data back in most cases, and that's assuming you left the connection alone for nothing but that.
We will call this problems that seem obvious when you are at a company with a 10Mb/s WAN. Lots of companies, certainly not all, have huge pipes and can restore systems really quickly. Lots of even homes now are starting to get 1Gb/s. Think about how fast a restore could be for critical systems over 100Mb/s to 1Gb/s. In many cases, companies with good WANs have faster WAN links thatn @mattspeller has LAN speed!!
The last company I interviewed at had backups from AppAssure replicated to a second location (they have 16) plus to the cloud. As well as the SANs replicated between two locations and backuped to Azure. Cloud backups when planned properly seems to be a good alternative (much better) than keeping tape or harddrives off site in a vault.
Sure, if you have 100Mb+ internet connection.
Granted I'm behind the times because I was worried about outages, but I'm working to solve that now, so soon I could see myself having 5 to 10 time the bandwidth I have now.
They don't have 100mb internet connection. It's metro between locations. Internet is like 40meg at each location. SANs are 42TB but they upload the data tranactionally as it happens so it doesn't make a hit on the connections. Backups are done hourly.
-
@Dashrender said:
@thecreativeone91 said:
@scottalanmiller said:
@Dashrender said:
But just as bad, in the case of failure, how are you suppose to get back online? It would take days or more to download all of the data back in most cases, and that's assuming you left the connection alone for nothing but that.
We will call this problems that seem obvious when you are at a company with a 10Mb/s WAN. Lots of companies, certainly not all, have huge pipes and can restore systems really quickly. Lots of even homes now are starting to get 1Gb/s. Think about how fast a restore could be for critical systems over 100Mb/s to 1Gb/s. In many cases, companies with good WANs have faster WAN links thatn @mattspeller has LAN speed!!
The last company I interviewed at had backups from AppAssure replicated to a second location (they have 16) plus to the cloud. As well as the SANs replicated between two locations and backuped to Azure. Cloud backups when planned properly seems to be a good alternative (much better) than keeping tape or harddrives off site in a vault.
Sure, if you have 100Mb+ internet connection.
Granted I'm behind the times because I was worried about outages, but I'm working to solve that now, so soon I could see myself having 5 to 10 time the bandwidth I have now.
You can bring down a ton with a lot less than 100Mb/s. At 100Mb/s you are getting some companies' LAN speeds. Remember that restores are often compressed and only need to be data. So 30Mb/s will often let you restore a ton. And keep in mind that only live data, not archives, need to be back before you are up and running.
-
-
@MattSpeller said:
@Dashrender said:
SOOOOOooooo.. no other Pros or Cons ???
Cat pics download way faster on 100mbit?
You know I laughed.. but I didn't put in the pro column that staff would be able to use streaming media more if management was OK with it. If I skip 50/10 and go to 100/15 I could probably even create a VLAN for patient internet access (of course throttle that sucker).
-
The more you mention the situation you are in, the more it smells like you need to get out of the closet and into the datacenter!
Get cheap crap pipes, move yer shit into a colo cage somewhere. That comes with a 100Mbps or even a 1Gbps Cogent unmetered pipe out to the interwebs. Have both sites VPN into it using as best as you can. I would take me a Peplink, break out a VPN connection to the colo, then route all the HTTP/HTTPS traffic over the cheapest pipe I can find.
Price on pipes and such would probably equal out on onsite versus offsite for a colo cage. That's when you move into the fun of counting power costs, cooling costs, even equipment costs if you move to a leased managed hosting model versus owning equipment. Then you will get good savings there in the long run.
As long as you can let go of the control of the physical machine, you can make some serious inroads into better network management. Hell, have you thought about cloud services? Don't even need a location, just be in the clooooooooooooooooud!
-
PSX is right. Hosted is the obvious answer. Once you move to fast pipes and redundancy, being hosted will be almost certainly a slam dunk.
-
HUH? How did you come to this conclusion? Sure eventually I'll probably push email offsite and to O365, then I'm left only with file and print onsite - no apps.
Currently today the only app I have onsite is a copy of my old EHR for reference purposes, email and file and print.
Going Colo (other than possibly saving me on power and cooling) wouldn't save me anyplace else.... I'd still need the exact same highly available or dual ISP setup as my original post.
Even if I go hosted today (never going to happen, the boss is anti remote - doubly admitted to me just yesterday), I'd still want/need very reliable fast links to the internet for my EHR with is my daily driver of an app that is already in the 'cloud'.
-
@Dashrender said:
Going Colo (other than possibly saving me on power and cooling) wouldn't save me anyplace else.... I'd still need the exact same highly available or dual ISP setup as my original post.
Do you ever need to physically be there off hours? Colo gives you 24x7 physical support. It also increases reliability my a dramatic amount.
-
@scottalanmiller said:
@Dashrender said:
Going Colo (other than possibly saving me on power and cooling) wouldn't save me anyplace else.... I'd still need the exact same highly available or dual ISP setup as my original post.
Do you ever need to physically be there off hours? Colo gives you 24x7 physical support. It also increases reliability my a dramatic amount.
You're right, reliability would be potentially better (at least the servers wouldn't suffer power loss when our building does), but unless the cost is exactly what we pay today, or less, they'd never go for it. I'd have show that the power use in my room and the heating/cooling cost would reduce by the amount of the rent at the Colo to even consider it.. because the reliability we have today is adequate.
Internet access is the single most important thing to us. If all of our internal servers just died.. yet we could still access the internet and our cloud based EHR, we would be able to continue to function.
-
@Dashrender said:
HUH? How did you come to this conclusion? Sure eventually I'll probably push email offsite and to O365, then I'm left only with file and print onsite - no apps.
What's email then? That's a serious driver of traffic, and a main reason you need bandwidth locally now. Why not move it to a box in the sky with massive redundant links that you could only dream about having locally?
Going Colo (other than possibly saving me on power and cooling) wouldn't save me anyplace else.... I'd still need the exact same highly available or dual ISP setup as my original post.
Nopes, this would reduce your need for bandwidth and could even get you to a single loop. A pipe with your stuff in colo if it goes offline isn't that big of a deal. Your services will still run, they will "never" go down because of problems in the main site. Site goes down for internet? Oh well, just smurf it out with cell data or failover to an el-cheapo pipe until the other pipe is back online.
Even if I go hosted today (never going to happen, the boss is anti remote - doubly admitted to me just yesterday), I'd still want/need very reliable fast links to the internet for my EHR with is my daily driver of an app that is already in the 'cloud'.
The only 100% reliable link I can guarantee is my LAN. How often does an internal LAN link go down?
Your EHR is your driver, but you are choking it with cheap bad pipes on hot sweaty hardware stuck in a closet for your users to access the internet. If it's already cloud based, setting up a terminal server or VDI farm on the colo cage would make sure anyone can work anywhere. BYOD, further reducing costs, by eliminiating the need to buy equipment for those guys to do their work or having to buy bleeding edge or newer equipment. A dumb terminal is cheap as shit, buy a bunch of Pi's and go to town! See, I've saved you even more money
And your boss is a dumbass.
-
@Dashrender said:
You're right, reliability would be potentially better (at least the servers wouldn't suffer power loss when our building does), but unless the cost is exactly what we pay today, or less, they'd never go for it.
Why worry about downtime if that's not a factor? Seems like the consensus is that cost, not uptime, is the only important factor. This suggests that the powers that be see the operational situation as having low value.
Colocation is cheaper in cases that we have measured. Better uptime, cost savings, less work for you. Pretty big win.
-
@Dashrender said:
.. because the reliability we have today is adequate.
"good enough" is an odd way to measure uptime. It should be a cost to risk scale. Not a "good enough" or "not good enough" scale. How does the CFO determine what is good enough without it being tied to money?
-
@PSX_Defector said:
The only 100% reliable link I can guarantee is my LAN. How often does an internal LAN link go down?
Had a customer lose their LAN last month
-
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
Something that PSX is clearly not understanding is that Email does not matter, local files do not matter, printing really doesn't matter. Two things in our environment really matter for operations to continue.
- phones. Yesteryear I'd say if the phones weren't working we'd close the office because we couldn't reach 911 if there was an emergency, but today that's not a problem because 80% or more of our staff have cellphones in their pockets and can call 911.
- Internet access. Our main app (EHR) that everyone in the company uses (except me, and I do to support them... soooo... ) is part of a service that we subscribe to. It's hosted, managed, etc all by them. So, as long as I have internet access, I can access my EHR.
PSX mentioned BOYD, my $11/hr employees are lucky if they have a desktop at home, let alone a laptop they can bring into the office. Furthermore, I can be assured that they do and will have constant problems (I know because they are always asking for help on their personal machines). Additionally, management does not feel right asking employees to use their personal equipment to do company business - and employees that have volunteered to use their personal equipment because it will make their lives easier have nearly demanded reimbursement for said usage.
Our external email usage is around 100 Megs a day, moving email to the cloud would actually drastically increase our internet usage because now all internal email would have to traverse our internet connection, granted it probably wouldn't be much, maybe increase to 500 megs a day, but it would be much worse.
We had a remote site that lost internet about a month ago, clinic was scheduled for one physician from 2-5. When the internet was not back in service at 2:15, the rest of the clinic was canceled, and the patients rescheduled. those patients that were already onsite were seen in a severally reduced capacity, and possibly rescheduled again if specific information wasn't available at the time. Fortunately this has only happened at our remote clinics about 3 times in 7+ years and has been considered acceptable. That said, there have been many more than 3 outages, there have been well over 15 between all of the remote locations, fortunately it seems the HFC plants don't all die at once in this city.
Now Scott will definitely tell me that I'm the wrong person to be driving redundant ISPs. The business should be telling me - hey it costs us x millions of dollars per hour that we are down, so it makes sense for you to spend y thousand dollars to help us reduce the downtime risk.
Unfortunately I'm not sure I can give you the lost revenue, and the reality is that 95% of canceled appointments that were going to happen (we have a noticeable no show rate) will definitely reschedule in the near future, and the cost is the physician's personal time while they squeeze those patients in to already booked days, usually at little or no staff time cost. Of course there's staff time lost during the outage itself which is not spent doing work because the system is down, but there is enough slack time in their schedules to do any additional work needed without paying overtime (yeah that means we are probably overstaffed - another topic).Have we been over paying for internet on fiber for the past 7+ years, well maybe not the whole time, but for most.. probably. The dual ISP solution has been available for quite some time I'm sure, but hasn't been a consideration until I started looking at my phone situation.
Things are getting a little disjointed.. hope you can just make due.
I feel comfortable telling the BOD that Best Effort ISP solutions WILL give them at least 3 downtime periods per year, probably averaging 4 hours. This seems to be the experience over all of our sites the last several years. With that in mind, and considering that our current Fiber ISP costs $880, and the new 50/10 connection will be $180, I feel comfortable given that information, that the BOD would tell me to get a second ISP installed at $120/month as a failover (12/2 best thing we can get at Best Effort at this location). The inconvenience factor alone would be worth saving to them, even if the clinic wasn't losing more than $1440/yr (plus taxes and the cost of the Peplink).
-
@Dashrender said:
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
If you don't have that info, who is making the financial decisions on technology spends?
-
@scottalanmiller said:
@Dashrender said:
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
If you don't have that info, who is making the financial decisions on technology spends?
Really, truly, me. I do what feel is the best solution at the cheapest price, I'm providing best effort considering the situation. That said, large expense, my phone project, this internet change, are run past the BOD. Sometimes they ask some questions, once they ask me to fine more information, but generally they say, what do you think, OK, do that.
Has it lead to overspending - clearly it has, we've had this expensive fiber for this whole time, perhaps never needed it, but like SMBs who get suckered into a SAN, I probably did get suckered into this Fiber because I was/am concerned about keeping us online as much as possible.
On a side note, when we cancel a clinic at a remote site, we're canceling no more than 10 patients, when we have problems at our main location that could could more like 40 in AM or PM, not to mention our ambulatory surgical center (ASC) (outpatient surgery), which if we are done, we finish the surgeries we are in, and stop until it's restored, or cancel the rest of the day. This is were the big dollars can be lost (I really should have mentioned this side earlier) there is only so much time available to book the ASC, and while I'm sure there is a small amount of slack time, it's generally pretty well booked, so we could definitely end up seeing some pretty substantial losses if we have to cancel those surgeries. So the justification of $1500/yr for a backup ISP is an easy one just considering the ASC.
-
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
If you don't have that info, who is making the financial decisions on technology spends?
Really, truly, me. I do what feel is the best solution at the cheapest price, I'm providing best effort considering the situation. That said, large expense, my phone project, this internet change, are run past the BOD. Sometimes they ask some questions, once they ask me to fine more information, but generally they say, what do you think, OK, do that.
Has it lead to overspending - clearly it has, we've had this expensive fiber for this whole time, perhaps never needed it, but like SMBs who get suckered into a SAN, I probably did get suckered into this Fiber because I was/am concerned about keeping us online as much as possible.
On a side note, when we cancel a clinic at a remote site, we're canceling no more than 10 patients, when we have problems at our main location that could could more like 40 in AM or PM, not to mention our ambulatory surgical center (ASC) (outpatient surgery), which if we are done, we finish the surgeries we are in, and stop until it's restored, or cancel the rest of the day. This is were the big dollars can be lost (I really should have mentioned this side earlier) there is only so much time available to book the ASC, and while I'm sure there is a small amount of slack time, it's generally pretty well booked, so we could definitely end up seeing some pretty substantial losses if we have to cancel those surgeries. So the justification of $1500/yr for a backup ISP is an easy one just considering the ASC.
Some items, like the ISP, is probably pretty easy as you have so few choices to work with. It's more when you have to decide on things like colocation, system architecture, hardware purchases, support contracts, etc. where there are far more granular choices and the decisions tend to come down to careful financial comparisons.
-
I've definitely learned a lot and seen mistakes I've made where I could have saved the company a ton of money recently. I will take those lessons with me to the next project and hopefully continue to make wiser choices for us.
That said, overall considering my experience I think I've done a pretty good job. We've had no major outages/issues since I've taken over IT here.
