Was mangolassi down earlier today?
-
@WLS-ITGuy said in Was mangolassi down earlier today?:
@Dashrender said in Was mangolassi down earlier today?:
@Pete-S said in Was mangolassi down earlier today?:
@JaredBusch said in Was mangolassi down earlier today?:
Yes.
Also talked about here.
Ahh, so server hardware failure. A little surprised it is only running on one server/VM. Not because there is a need to do otherwise but because it could :grinning_face:
is it worth the expense just because it can?
Always
Could build a fully replicated ML cluster with the Vultr instance and something in their Lab.
-
I doubt enough revenue and future potential revenue was lost to warrant HA.
-
Downtime was quite minimal, I don't think HA is even worth thinking about for ML as of yet.
-
Yes, Linode had a hardware failure.
-
@Obsolesce said in Was mangolassi down earlier today?:
I doubt enough revenue and future potential revenue was lost to warrant HA.
Especially given that no competitor of ML has HA, even ones with lots of sponsors, and the actual downtime is tiny, so the amount of time for HA to cover is really small. And if we moved to Vultr, we'd likely cut the existing downtime by 90% of so as it is.
-
@dafyre said in Was mangolassi down earlier today?:
@WLS-ITGuy said in Was mangolassi down earlier today?:
@Dashrender said in Was mangolassi down earlier today?:
@Pete-S said in Was mangolassi down earlier today?:
@JaredBusch said in Was mangolassi down earlier today?:
Yes.
Also talked about here.
Ahh, so server hardware failure. A little surprised it is only running on one server/VM. Not because there is a need to do otherwise but because it could :grinning_face:
is it worth the expense just because it can?
Always
Could build a fully replicated ML cluster with the Vultr instance and something in their Lab.
Cost would be high, though. ML needs a lot of horsepower, so cloud instances don't come cheaply.
-
@Pete-S said in Was mangolassi down earlier today?:
@JaredBusch said in Was mangolassi down earlier today?:
Yes.
Also talked about here.
Ahh, so server hardware failure. A little surprised it is only running on one server/VM. Not because there is a need to do otherwise but because it could :grinning_face:
Someone has to pay for that, though.
-
@scottalanmiller said in Was mangolassi down earlier today?:
@Pete-S said in Was mangolassi down earlier today?:
@JaredBusch said in Was mangolassi down earlier today?:
Yes.
Also talked about here.
Ahh, so server hardware failure. A little surprised it is only running on one server/VM. Not because there is a need to do otherwise but because it could :grinning_face:
Someone has to pay for that, though.
I think @Pete-S and @dafyre are volunteering to pay for ML HA...
-
I was thinking if you have load balancers in front of two webservers and you cluster the databases you get about half the load on each server (since it's mostly read) and you get redundancy as well. So you can run on two smaller VMs instead of one larger and thereby get HA for almost no additional cost (maybe).
If you use cloudflare isn't load balancers part of the package?
BTW @scottalanmiller, don't know who pays or hosts ML but if the site requires lots of horsepower wouldn't it be better suited as a colo workload?
-
@Pete-S said in Was mangolassi down earlier today?:
I was thinking if you have load balancers in front of two webservers and you cluster the databases you get about half the load on each server (since it's mostly read) and you get redundancy as well. So you can run on two smaller VMs instead of one larger and thereby get HA for almost no additional cost (maybe).
If you use cloudflare isn't load balancers part of the package?
BTW @scottalanmiller, don't know who pays or hosts ML but if the site requires lots of horsepower wouldn't it be better suited as a colo workload?
It used to be Minion Queen's personal project, but when that whole thing went south... NTG took over ML. As for CoLo workload - possibly, but someone would have to front the capital expense then. and HA becomes a zero chance thing at that point due to costs.
-
@Dashrender said in Was mangolassi down earlier today?:
@Pete-S said in Was mangolassi down earlier today?:
I was thinking if you have load balancers in front of two webservers and you cluster the databases you get about half the load on each server (since it's mostly read) and you get redundancy as well. So you can run on two smaller VMs instead of one larger and thereby get HA for almost no additional cost (maybe).
If you use cloudflare isn't load balancers part of the package?
BTW @scottalanmiller, don't know who pays or hosts ML but if the site requires lots of horsepower wouldn't it be better suited as a colo workload?
It used to be Minion Queen's personal project, but when that whole thing went south... NTG took over ML. As for CoLo workload - possibly, but someone would have to front the capital expense then. and HA becomes a zero chance thing at that point due to costs.
When using cloud services you can build out the infrastructure with the smallest discrete step being a VM.
In colo, where you have your own hardware, the smallest discrete step is a server. You either have one or you don't. If a server can run 100 VMs you will have somewhere between 0 to 99 VMs available as it's impossible to utilize it 100%.
It's also impossible for cloud providers to utilize their data centers 100%, but that is not something the users can see.So it's almost guaranteed that colo equipment isn't 100% utilized and then adding one more workload doesn't trigger any capital expense, only operational expense - primary in the form of electricity and management and possibly bandwidth.
So that's one way a colo is different.
Another way it's different is that colo can be lower cost - if you need enough capacity all the time to have colo equipment.
-
@Pete-S said in Was mangolassi down earlier today?:
I was thinking if you have load balancers in front of two webservers and you cluster the databases you get about half the load on each server (since it's mostly read) and you get redundancy as well.
DBs don't work that way, we'd need, for all of that, quite a bit more power. Load balancing works well at scale, but not when you fit into a single host. To load balance, rather than just have a failover option, we'd be increasing the cost even moreso than doubling it.
-
@scottalanmiller said in Was mangolassi down earlier today?:
@Pete-S said in Was mangolassi down earlier today?:
I was thinking if you have load balancers in front of two webservers and you cluster the databases you get about half the load on each server (since it's mostly read) and you get redundancy as well.
DBs don't work that way, we'd need, for all of that, quite a bit more power. Load balancing works well at scale, but not when you fit into a single host. To load balance, rather than just have a failover option, we'd be increasing the cost even moreso than doubling it.
I was assuming more than one physical host.
-
@Pete-S said in Was mangolassi down earlier today?:
BTW @scottalanmiller, don't know who pays or hosts ML but if the site requires lots of horsepower wouldn't it be better suited as a colo workload?
No, some easy math there....
Minimum colo cost is $50 / mo not including the hardware, and doesn't have backups which you need to coordinate on your own. So even a cheap server (which is all that would be needed) would be say $2,000 which is $33/mo. So you'd need to spend somewhere around $90/mo (time value of money) to have a single server in a colo with minimum performance.
And you still need any external costs like backups, CloudFlare, etc. That doesn't change.
ML costs about $80/mo to run on Linode or Vultr. Which isn't crazy, but as the site doesn't produce revenue, that's a lot to donate to the world when you start talking about doubling or tripling it because that's just more money being donated, it isn't like there is any financial advantage to more uptime.
-
@Pete-S said in Was mangolassi down earlier today?:
@scottalanmiller said in Was mangolassi down earlier today?:
@Pete-S said in Was mangolassi down earlier today?:
I was thinking if you have load balancers in front of two webservers and you cluster the databases you get about half the load on each server (since it's mostly read) and you get redundancy as well.
DBs don't work that way, we'd need, for all of that, quite a bit more power. Load balancing works well at scale, but not when you fit into a single host. To load balance, rather than just have a failover option, we'd be increasing the cost even moreso than doubling it.
I was assuming more than one physical host.
Right, that's where the growth in cost comes from.
-
@Pete-S said in Was mangolassi down earlier today?:
Another way it's different is that colo can be lower cost - if you need enough capacity all the time to have colo equipment
But the minimum scale for colo is decently high unless you have lots of workloads to put there.
-
@scottalanmiller said in Was mangolassi down earlier today?:
@Pete-S said in Was mangolassi down earlier today?:
Another way it's different is that colo can be lower cost - if you need enough capacity all the time to have colo equipment
But the minimum scale for colo is decently high unless you have lots of workloads to put there.
Yes, it is. And you have all the surrounding network gear as well.
I was just thinking perhaps there already was colo servers and infrastructure in place. Otherwise it would economic suicide
-
@scottalanmiller said in Was mangolassi down earlier today?:
ML costs about $80/mo to run on Linode or Vultr
So that would that be something like 16GB RAM, 6 vCPUs?
-
@Pete-S said in Was mangolassi down earlier today?:
@scottalanmiller said in Was mangolassi down earlier today?:
ML costs about $80/mo to run on Linode or Vultr
So that would that be something like 16GB RAM, 6 vCPUs?
I believe it is multiple instances. The database server is a separate incidents from the web server
-
@Pete-S said in Was mangolassi down earlier today?:
@scottalanmiller said in Was mangolassi down earlier today?:
ML costs about $80/mo to run on Linode or Vultr
So that would that be something like 16GB RAM, 6 vCPUs?
Yes.
