Testing oVirt...
-
@travisdh1 said in Testing oVirt...:
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
Right, Fedora has been faster and more stable for us. CentOS was much slower, lacked solid features, and had support issues (because it was unable to continue to support living software that was still updating while the OS had stagnated.)
What exactly was CentOS slower at?
What exactly is CentOS slower at?
Every major web server, many database servers.
What features were lacking? How exactly it could not support "living software"?
We know that PHP 5.X is many times slower than PHP 7.X. Meaning that any admin actually deploying a web server with PHP 5.X is acting maliciously. Adding support for PHP 7.X onto Red Hat/CentOS is an additional process that isn't needed in Fedora at the least, and creates package dependency headaches quite often.
And once you start replacing the "tested, supported" pieces of EL with Fedora pieces, um..... that defeats the purpose.
I don't think even RH management would argue that RHEL is better tested with Fedora components than Fedora is.
-
What I'm hearing here is that we should move to Ubuntu, because their current release IS their key production release with the testing and support.
If RH really treats Fedora so haphazardly, and RHEL is so pathetically out of date, that makes both sound like bad platforms to be on.
In reality, I simply don't believe that Fedora isn't stable. I've seen zero evidence of this, I know of no one having or having had Fedora stability issues. Nor CentOS / EL stability issues. Both are quite stable.
EL has the advantage of having "so much testing", and Fedora has the benefit of "much more updates". Ten years ago, we pretty much all agreed that the additional testing outweighed being up to date. But I don't see that as being the case any more.
-
@scottalanmiller said in Testing oVirt...:
Yes, but I would never pay for EL support or do business with RH on ethics grounds.
Care to elaborate?
The "support" is how much they support the OS and make sure that it works,
Oh no, you clearly never worked with RH proper. I've spent 4 years in RH support, and there is so much more there
and they do a pretty amazing job with Fedora. Bottom line, Fedora gets "support" to make it work in the real world, EL gets "support" that you pay for to help you when you don't know what you are doing.
Nope. Fedora gets built by professionals to be an OS that has all the latest code that is capable of working at least somehow. Not well, not in a stable manner, but work. Working well and stabilization is all the work that goes into EL afterwards.
Support for EL isn't about handholding, it's about making integrations work, about providing customers with very quick fixes to their specific problems at the code level sometimes, and about working with the customers on what they need in their distribution. I've gone through T4 support to product management and seen exactly that all the way. Simple handholding is something you get from ISP support with a bunch of kids who read from a script. Don't get confused between the two.
In the real world, Fedora is supported and works. EL does not for production workloads I deal with. When I have the choice between the two, Fedora every time.
In the real world, especially at scale, nobody has the time to deal with the bugs you get in the new untested code. Not everything can be solved by restarting a microservice, and not many organizations keep talent onboard, who can fix a kernel bug on the fly. This is why people rely on EL and other enterprise grade software.
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
Yes, but I would never pay for EL support or do business with RH on ethics grounds.
Care to elaborate?
Nothing secret. RH used posts on social media to go to human resources and try to get internal Linux engineers fired (they did this to me at a Fortune 100 bank) and then immediately (because they thought that this would work) turn around and try to hire the same person for far, far less than the position that they [failed] to get them fired for.
As an IT pro, you should never take the risk of letting RH account reps have access to your business. They are the enemies of the IT team and put you personally at risk.
I was lucky, I had done nothing wrong and the employer was livid with Red Hat for breaching professionalism and trying to screw their employee and the bank. And the funniest part was that I was RH's biggest proponent internally and now have removed them from other banks because I won't do business with them - and it was all because RH didn't like that I had said how great RH support was... but that you never needed it as the product always worked. So when they tried to get me fired, it brought attention all the way up the food chain that with tens of thousands of servers, and a decade of management, we'd never once used the support and that CentOS would have saved us a fortune.
And the pay that RH was offering was $60K for their engineers. What a joke. If that's who is providing the vendor support, no one should be paying for it.
This also exposed the lack of internal support to the bank. Since the bank was paying hundreds of thousands for a fleet of top engineers, they found out that their "vendor support" that they paid extra for weren't people they would ever consider hiring internally.
-
@dyasny said in Testing oVirt...:
The "support" is how much they support the OS and make sure that it works,
Oh no, you clearly never worked with RH proper. I've spent 4 years in RH support, and there is so much more there
We were a top tier customer, we had everything RH offered.
-
@dyasny said in Testing oVirt...:
In the real world, Fedora is supported and works. EL does not for production workloads I deal with. When I have the choice between the two, Fedora every time.
In the real world, especially at scale, nobody has the time to deal with the bugs you get in the new untested code. Not everything can be solved by restarting a microservice, and not many organizations keep talent onboard, who can fix a kernel bug on the fly. This is why people rely on EL and other enterprise grade software.
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
-
@scottalanmiller said in Testing oVirt...:
And vice versa. Having worked at the vendor in question, it feels like you are seeing this through the eyes of the sales team, and not thinking about it from the perspective of the customers needing to actually run the product. Where is your conviction coming from, other than it being the sales mantra of the vendor?
Never worked in sales, but I have worked for other enterprise vendors and their customers quite a bit. From government to automotive and aerospace, to oil and gas to hipster-ish startups. No Fedora anywhere but desktops sometimes. Ever.
Having used both in the real world, what problems do you see with Fedora and what old code are you running that EL always has what you need?
I can install EL, keep it updated without moving versions, and be certain everything I build on top of it will keep working throughout the OS lifecycle, which is very long. That means I can concentrate on developing my software instead of wasting time on keeping up with what the underlying OS is doing, introducing bugs into my code which used to work before the OS update. Without losing the safety of important updates coming in on time. Yes, some of the prepackaged stuff is outdated, but I can get the newer code, if I need it, from repositories that are code-specific (in my case - mostly pypi). I've come into a place once, where Fedora was installed on several hundred servers which were rendering CGI. They had two people there just making sure the cluster task dispatcher was able to work after every update. Two expensive coders, doing nothing but test fedora releases versus their task scheduler. A test cluster on CentOS turned out to be able to do exactly the same job, but the guys dealing with fedora quirks became free to develop the task scheduler, which boosted their bottom line productivity after a few months.
Are you really seeing Fedora instabilities that the rest of us are not? Are you really not running any modern code that benefits from current libraries and packages?
Of course I am. pip, cpan, (whatever php has for that same purpose) etc do the job perfectly. There isn't much I cannot do on an EL distribution, one way or another
-
@dyasny said in Testing oVirt...:
This is why people rely on EL and other enterprise grade software.
Not really. In every case I've ever encountered, people use EL due to a policy of "buying vendor support" without actual investigation into a need. It's political finger pointing in nearly all cases. That's how businesses work. Go up the chain and dig in at real world customers, how many actually evaluated risk and got support for that reason, versus get it for every product and never even look into why for a specific one.
When I got into the hedge fund world, the idea of "evaluating support" was new and just started happening. And there too, RH lost out because they found that there was no need for support. It just wasn't a thing that came up in the real world.
People running into those kinds of bugs are edge cases and generally causing them by doing other things poorly.
-
@dyasny Yep, a great example
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
And vice versa. Having worked at the vendor in question, it feels like you are seeing this through the eyes of the sales team, and not thinking about it from the perspective of the customers needing to actually run the product. Where is your conviction coming from, other than it being the sales mantra of the vendor?
Never worked in sales, but I have worked for other enterprise vendors and their customers quite a bit. From government to automotive and aerospace, to oil and gas to hipster-ish startups. No Fedora anywhere but desktops sometimes. Ever.
This is a time thing, though.
Remember you are talking to loads of people that were passionate about CentOS / EL in the past and switched recently because times have changed. So unless you worked for all of these places in the last 24 months, your experience is moot. All of our experiences would match that, as well.
When I was in banking, EL was king. But the entire point of the article I linked was that things have changed, code is more mature today, projects move faster, priorities in the world have changed.
Your looking to past experiences tells me you are thinking about the past, not the environment today. And you are missing that we had the same experiences back then, too. So it is based on the same experience, same knowledge, but bringing it up to date to today's situations, that led us from EL to Fedora.
-
@scottalanmiller this is foul play indeed, and knowing the company from within, I'm pretty sure such practices aren't smiled upon. I'm not with RH any longer, but in my 10 years there, they have always struck me as the most morally positive company I've ever worked with and for. I've seen business needs sacrificed to doing the right and moral thing quite a few times there. In the end - this is why they succeed, where those who attempt to be business sharks fail (waves at Canonical).
Do keep in mind, you have one single story, I've lived in there for years. I'd say I do have more stats
-
@dyasny said in Testing oVirt...:
Are you really seeing Fedora instabilities that the rest of us are not? Are you really not running any modern code that benefits from current libraries and packages?
Of course I am. pip, cpan, (whatever php has for that same purpose) etc do the job perfectly. There isn't much I cannot do on an EL distribution, one way or another
- But you can't do it as well, or as performant.
- If you replace any libraries to do it, I consider that acknowledgement that Fedora was better, but you were only on EL to make a point.
CAN you do things on EL? Of course. That's not the point. It's whether it is better than Fedora most of the time. And that answer seems to be "no". You make some good points about testing, but don't address that we had already considered that before coming to the conclusion that they weren't good enough to overcome the significant deficits of the process.
-
@scottalanmiller said in Testing oVirt...:
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
How does Fedora protect you? It's a distro packaged, with some bugfixes here and there, no formal QA besides the very basics, no support, nothing. I'm fine with that on my laptop, but on a thousand servers?
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
The apps don't run in a vacuum, they rely on layers of software. Do you test all those layers?
-
@dyasny said in Testing oVirt...:
@scottalanmiller this is foul play indeed, and knowing the company from within, I'm pretty sure such practices aren't smiled upon. I'm not with RH any longer, but in my 10 years there, they have always struck me as the most morally positive company I've ever worked with and for. I've seen business needs sacrificed to doing the right and moral thing quite a few times there. In the end - this is why they succeed, where those who attempt to be business sharks fail (waves at Canonical).
Do keep in mind, you have one single story, I've lived in there for years. I'd say I do have more stats
Inside stats are misleading, though. Vendors paint a different picture for employees than what customers see. That's not an RH thing, that's universal.
I guarantee as an employee, there were zero stats kept about how they were doing unethical, maybe illegal, things with customers. Account managers don't report to engineering.
That they had a process for this at all means it wasn't casual. This was a major effort to do what they did. If you didn't hear about it while there, then your stats aren't valid, if that makes sense. Because I bet you were told that nothing like that was happening with major customers (we had to be one of the five biggest customers.)
We had the same thing internally at where I was, so I get how it happens. We happen to catch it in our case. HR was trying to sabotage some departments and telling people that the jobs were awful before those applicants went fully into the system. So the "stats" internally said one thing, but if you could find someone who had turned down a job with the company, you learned all kinds of secrets that weren't officially recorded anywhere.
Same with RH - there is no chance that that activity was on the books. But it was a major effort to manipulate the customer. They could easily guarantee that their account would not be dropped if the heads of the engineering departments were working at RH and not at the customer.
-
@dyasny said in Testing oVirt...:
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
The apps don't run in a vacuum, they rely on layers of software. Do you test all those layers?
Individually? No. Together, yes.
-
@dyasny said in Testing oVirt...:
@black3dynamite said in Testing oVirt...:
Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.
Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?
Please tell me what the point is in CentOS running PHP 5.6?
I mean, Look how old it is, and look when it looses support!
Fedora 28 uses 7.2.x, FAR FROM BLEEDING EDGE (ffs!). And oh looky, supported for longer than 2 more months lol.
Have fun upgrading the CentOS LTS servers you use to the next CentOS LTS... EVERYTHING will break, including all of your PHP apps.
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
How does Fedora protect you? It's a distro packaged, with some bugfixes here and there, no formal QA besides the very basics, no support, nothing. I'm fine with that on my laptop, but on a thousand servers?
How doesn't matter, that it does it and does it reliably is what matters.
We can postulate any number of reasons "why" or "how". Maybe having the individual packages more up to date simply matters that much. Maybe extensive testing is a waste and not as valuable as it sounds. Maybe Fedora tests more than you realize (maybe unofficially.)
Bottom line, though, is that Fedora has reliably protected. EL has, too. It's not that EL is bad, it's that Fedora just has come out better in real world use.
There are lots of things that matter at the end of the day...
- Reliability
- Speed
- Security
- Compatibility
And more, of course. When taken as a whole, EL might win on reliability, but the margin of winning is nominal. But Fedora seems to win on all of the others. Sometimes nominally, sometimes by quite a margin.
-
@obsolesce said in Testing oVirt...:
@dyasny said in Testing oVirt...:
@black3dynamite said in Testing oVirt...:
Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.
Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?
Please tell me what the point is in CentOS running PHP 5.6?
I mean, Look how old it is, and look when it looses support!
Fedora 28 uses 7.2.x, FAR FROM BLEEDING EDGE (ffs!). And oh looky, supported for longer than 2 more months lol.
Have fun upgrading the CentOS LTS servers you use to the next CentOS LTS... EVERYTHING will break, including all of your PHP apps.
One COULD argue that RHEL goes out of support when PHP does. From an application perspective, using RHEL 7 would be "unsupported".
By that logic, which is pretty solid in reality, Fedora is the supported OS, not RHEL, come January.
-
@scottalanmiller said in Testing oVirt...:
Remember you are talking to loads of people that were passionate about CentOS / EL in the past and switched recently because times have changed. So unless you worked for all of these places in the last 24 months, your experience is moot. All of our experiences would match that, as well.
So one second you say business is slow to change, and then you say that is moot. OK.
When I was in banking, EL was king. But the entire point of the article I linked was that things have changed, code is more mature today, projects move faster, priorities in the world have changed.
Code is, if anything, less mature today. With the current craze of moving as fast as possible, release as fast as possible and skip testing everywhere ("we've got devs writing unit tests for that" - sounds familiar?) code is getting crappier every day. And with solutions like k8s and mesos, people stopped debugging, their answer is "just respawn, who cares if it fails every 5 minutes".
Your looking to past experiences tells me you are thinking about the past, not the environment today. And you are missing that we had the same experiences back then, too. So it is based on the same experience, same knowledge, but bringing it up to date to today's situations, that led us from EL to Fedora.
Oh leave the slogans aside, neither of us has any future experience, just the past. You can guess what will happen in the future, but you dont know, no matter how much you "look to the future". You found Fedora to be working for your use case - well and good. This doesn't mean EL is less stable or has less support, you simply found a niche use case which works for you. Huge difference
-
@obsolesce said in Testing oVirt...:
Have fun upgrading the CentOS LTS servers you use to the next CentOS LTS... EVERYTHING will break, including all of your PHP apps.
This is a big one. Fedora has a consistently rock solid update path. EL does not. Moving EL versions has always been a hassle and tends to cause apps to break like crazy. Fedora isn't perfect, nothing is, but the difference is shocking. Keeping Fedora up to date is trivial, EL is an effort (akin to Windows.)
