Testing oVirt...
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
Right, Fedora has been faster and more stable for us. CentOS was much slower, lacked solid features, and had support issues (because it was unable to continue to support living software that was still updating while the OS had stagnated.)
What exactly was CentOS slower at? What features were lacking? How exactly it could not support "living software"?
PHP packages are the glaring one lots of us ran into recently.
Living software, meaning software that companies were actively updating and releasing tended to eventually require that we bolt on Fedora libraries to CentOS to keep it working - 100% defeating the purpose of CentOS since now we are using Fedora anyway, but without as much unified testing from either side.
Ran into this with a lot of packages.
Those that still worked on CentOS, did so at a fraction of the speed. Not ideal. Bottom line, CentOS hasn't been up to the job. It's too much like old Windows - a great solution with the goal being of supporting bad third party products that aren't current themselves (and are often ghost ships.)
In the Windows world, abandoned software is the norm, not the exception. It's so common, no one thinks much of it. The entire Windows ecosystem embraces this traditionally (this is starting to change as MS wants to start being more competitive) and much of the Windows super slow release schedule and the way they traditionally treated an update more like a new product that would stand on its own forever were focused around providing an aging, never-updating platform for non-living software packages that needed to just "keep running" without real updates for possibly decades.
-
@dyasny said in Testing oVirt...:
@black3dynamite said in Testing oVirt...:
Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.
Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you.
Remember no one wants bleeding edge. Current stable and bleeding edge are worlds apart.
Long Term Release < Current Stable < Cutting Edge < Bleeding Edge
Fedora is very production ready, very stable. It's very, very far away from bleeding edge. Even Tumbleweed is only cutting edge.
-
@scottalanmiller said in Testing oVirt...:
@dyasny said in Testing oVirt...:
@black3dynamite said in Testing oVirt...:
Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.
Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?
Support. Fedora has insanely broad vendor (meaning RH) and third party (the software makers) support. Possibly the broadest in the industry, or maybe second after Ubuntu. But Ubuntu support leans towards the unsupported LTS releases making Ubuntu products questionably supported at all (since Ubuntu's official stance is that if you need LTS support beyond consulting, meaning actual fixes, you might have to leave LTS and go to Current and if your software vendor is LTS only, the resulting product is unsupported.)
Alpine is great, but not many vendors test against it.
Actual vendor support for Fedora is tiny compared to EL, you are really describing an alternate reality here. I have never seen anyone doing any production work on Fedora, beyond development and desktop computing.
-
@scottalanmiller said in Testing oVirt...:
Remember no one wants bleeding edge. Current stable and bleeding edge are worlds apart.
Long Term Release < Current Stable < Cutting Edge < Bleeding Edge
Fedora is very production ready, very stable. It's very, very far away from bleeding edge. Even Tumbleweed is only cutting edge.
I really don't know where you are getting this from. I've never seen such an opinion expressed from anyone in the business, neither in my 10 years within Red Hat or in my over 20 years in IT in general. Fedora is supposed to be as close to the development code as possible, there is no QA there, just basic testing to make sure stuff can be compiled/built/installed, nothing else. Even the layered upstream products get way more testing on EL than on Fedora (hence the dumping of Fedora for oVirt, and IIRC RDO too).
I really wonder how you managed to come up with this conviction of yours and what makes you believe it besides your own experiences
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
@dyasny said in Testing oVirt...:
@black3dynamite said in Testing oVirt...:
Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.
Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?
Support. Fedora has insanely broad vendor (meaning RH) and third party (the software makers) support. Possibly the broadest in the industry, or maybe second after Ubuntu. But Ubuntu support leans towards the unsupported LTS releases making Ubuntu products questionably supported at all (since Ubuntu's official stance is that if you need LTS support beyond consulting, meaning actual fixes, you might have to leave LTS and go to Current and if your software vendor is LTS only, the resulting product is unsupported.)
Alpine is great, but not many vendors test against it.
Actual vendor support for Fedora is tiny compared to EL, you are really describing an alternate reality here. I have never seen anyone doing any production work on Fedora, beyond development and desktop computing.
Yes, but I would never pay for EL support or do business with RH on ethics grounds. The "support" is how much they support the OS and make sure that it works, and they do a pretty amazing job with Fedora. Bottom line, Fedora gets "support" to make it work in the real world, EL gets "support" that you pay for to help you when you don't know what you are doing.
In the real world, Fedora is supported and works. EL does not for production workloads I deal with. When I have the choice between the two, Fedora every time.
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
Remember no one wants bleeding edge. Current stable and bleeding edge are worlds apart.
Long Term Release < Current Stable < Cutting Edge < Bleeding Edge
Fedora is very production ready, very stable. It's very, very far away from bleeding edge. Even Tumbleweed is only cutting edge.
I really don't know where you are getting this from. I've never seen such an opinion expressed from anyone in the business, neither in my 10 years within Red Hat or in my over 20 years in IT in general. Fedora is supposed to be as close to the development code as possible, there is no QA there, just basic testing to make sure stuff can be compiled/built/installed, nothing else. Even the layered upstream products get way more testing on EL than on Fedora (hence the dumping of Fedora for oVirt, and IIRC RDO too).
I really wonder how you managed to come up with this conviction of yours and what makes you believe it besides your own experiences
And vice versa. Having worked at the vendor in question, it feels like you are seeing this through the eyes of the sales team, and not thinking about it from the perspective of the customers needing to actually run the product. Where is your conviction coming from, other than it being the sales mantra of the vendor?
Having used both in the real world, what problems do you see with Fedora and what old code are you running that EL always has what you need?
Are you really seeing Fedora instabilities that the rest of us are not? Are you really not running any modern code that benefits from current libraries and packages?
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
Right, Fedora has been faster and more stable for us. CentOS was much slower, lacked solid features, and had support issues (because it was unable to continue to support living software that was still updating while the OS had stagnated.)
What exactly was CentOS slower at?
What exactly is CentOS slower at?
Every major web server, many database servers.
What features were lacking? How exactly it could not support "living software"?
We know that PHP 5.X is many times slower than PHP 7.X. Meaning that any admin actually deploying a web server with PHP 5.X is acting maliciously. Adding support for PHP 7.X onto Red Hat/CentOS is an additional process that isn't needed in Fedora at the least, and creates package dependency headaches quite often.
-
@travisdh1 said in Testing oVirt...:
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
Right, Fedora has been faster and more stable for us. CentOS was much slower, lacked solid features, and had support issues (because it was unable to continue to support living software that was still updating while the OS had stagnated.)
What exactly was CentOS slower at?
What exactly is CentOS slower at?
Every major web server, many database servers.
What features were lacking? How exactly it could not support "living software"?
We know that PHP 5.X is many times slower than PHP 7.X. Meaning that any admin actually deploying a web server with PHP 5.X is acting maliciously. Adding support for PHP 7.X onto Red Hat/CentOS is an additional process that isn't needed in Fedora at the least, and creates package dependency headaches quite often.
And once you start replacing the "tested, supported" pieces of EL with Fedora pieces, um..... that defeats the purpose.
I don't think even RH management would argue that RHEL is better tested with Fedora components than Fedora is.
-
What I'm hearing here is that we should move to Ubuntu, because their current release IS their key production release with the testing and support.
If RH really treats Fedora so haphazardly, and RHEL is so pathetically out of date, that makes both sound like bad platforms to be on.
In reality, I simply don't believe that Fedora isn't stable. I've seen zero evidence of this, I know of no one having or having had Fedora stability issues. Nor CentOS / EL stability issues. Both are quite stable.
EL has the advantage of having "so much testing", and Fedora has the benefit of "much more updates". Ten years ago, we pretty much all agreed that the additional testing outweighed being up to date. But I don't see that as being the case any more.
-
@scottalanmiller said in Testing oVirt...:
Yes, but I would never pay for EL support or do business with RH on ethics grounds.
Care to elaborate?
The "support" is how much they support the OS and make sure that it works,
Oh no, you clearly never worked with RH proper. I've spent 4 years in RH support, and there is so much more there
and they do a pretty amazing job with Fedora. Bottom line, Fedora gets "support" to make it work in the real world, EL gets "support" that you pay for to help you when you don't know what you are doing.
Nope. Fedora gets built by professionals to be an OS that has all the latest code that is capable of working at least somehow. Not well, not in a stable manner, but work. Working well and stabilization is all the work that goes into EL afterwards.
Support for EL isn't about handholding, it's about making integrations work, about providing customers with very quick fixes to their specific problems at the code level sometimes, and about working with the customers on what they need in their distribution. I've gone through T4 support to product management and seen exactly that all the way. Simple handholding is something you get from ISP support with a bunch of kids who read from a script. Don't get confused between the two.
In the real world, Fedora is supported and works. EL does not for production workloads I deal with. When I have the choice between the two, Fedora every time.
In the real world, especially at scale, nobody has the time to deal with the bugs you get in the new untested code. Not everything can be solved by restarting a microservice, and not many organizations keep talent onboard, who can fix a kernel bug on the fly. This is why people rely on EL and other enterprise grade software.
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
Yes, but I would never pay for EL support or do business with RH on ethics grounds.
Care to elaborate?
Nothing secret. RH used posts on social media to go to human resources and try to get internal Linux engineers fired (they did this to me at a Fortune 100 bank) and then immediately (because they thought that this would work) turn around and try to hire the same person for far, far less than the position that they [failed] to get them fired for.
As an IT pro, you should never take the risk of letting RH account reps have access to your business. They are the enemies of the IT team and put you personally at risk.
I was lucky, I had done nothing wrong and the employer was livid with Red Hat for breaching professionalism and trying to screw their employee and the bank. And the funniest part was that I was RH's biggest proponent internally and now have removed them from other banks because I won't do business with them - and it was all because RH didn't like that I had said how great RH support was... but that you never needed it as the product always worked. So when they tried to get me fired, it brought attention all the way up the food chain that with tens of thousands of servers, and a decade of management, we'd never once used the support and that CentOS would have saved us a fortune.
And the pay that RH was offering was $60K for their engineers. What a joke. If that's who is providing the vendor support, no one should be paying for it.
This also exposed the lack of internal support to the bank. Since the bank was paying hundreds of thousands for a fleet of top engineers, they found out that their "vendor support" that they paid extra for weren't people they would ever consider hiring internally.
-
@dyasny said in Testing oVirt...:
The "support" is how much they support the OS and make sure that it works,
Oh no, you clearly never worked with RH proper. I've spent 4 years in RH support, and there is so much more there
We were a top tier customer, we had everything RH offered.
-
@dyasny said in Testing oVirt...:
In the real world, Fedora is supported and works. EL does not for production workloads I deal with. When I have the choice between the two, Fedora every time.
In the real world, especially at scale, nobody has the time to deal with the bugs you get in the new untested code. Not everything can be solved by restarting a microservice, and not many organizations keep talent onboard, who can fix a kernel bug on the fly. This is why people rely on EL and other enterprise grade software.
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
-
@scottalanmiller said in Testing oVirt...:
And vice versa. Having worked at the vendor in question, it feels like you are seeing this through the eyes of the sales team, and not thinking about it from the perspective of the customers needing to actually run the product. Where is your conviction coming from, other than it being the sales mantra of the vendor?
Never worked in sales, but I have worked for other enterprise vendors and their customers quite a bit. From government to automotive and aerospace, to oil and gas to hipster-ish startups. No Fedora anywhere but desktops sometimes. Ever.
Having used both in the real world, what problems do you see with Fedora and what old code are you running that EL always has what you need?
I can install EL, keep it updated without moving versions, and be certain everything I build on top of it will keep working throughout the OS lifecycle, which is very long. That means I can concentrate on developing my software instead of wasting time on keeping up with what the underlying OS is doing, introducing bugs into my code which used to work before the OS update. Without losing the safety of important updates coming in on time. Yes, some of the prepackaged stuff is outdated, but I can get the newer code, if I need it, from repositories that are code-specific (in my case - mostly pypi). I've come into a place once, where Fedora was installed on several hundred servers which were rendering CGI. They had two people there just making sure the cluster task dispatcher was able to work after every update. Two expensive coders, doing nothing but test fedora releases versus their task scheduler. A test cluster on CentOS turned out to be able to do exactly the same job, but the guys dealing with fedora quirks became free to develop the task scheduler, which boosted their bottom line productivity after a few months.
Are you really seeing Fedora instabilities that the rest of us are not? Are you really not running any modern code that benefits from current libraries and packages?
Of course I am. pip, cpan, (whatever php has for that same purpose) etc do the job perfectly. There isn't much I cannot do on an EL distribution, one way or another
-
@dyasny said in Testing oVirt...:
This is why people rely on EL and other enterprise grade software.
Not really. In every case I've ever encountered, people use EL due to a policy of "buying vendor support" without actual investigation into a need. It's political finger pointing in nearly all cases. That's how businesses work. Go up the chain and dig in at real world customers, how many actually evaluated risk and got support for that reason, versus get it for every product and never even look into why for a specific one.
When I got into the hedge fund world, the idea of "evaluating support" was new and just started happening. And there too, RH lost out because they found that there was no need for support. It just wasn't a thing that came up in the real world.
People running into those kinds of bugs are edge cases and generally causing them by doing other things poorly.
-
@dyasny Yep, a great example
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
And vice versa. Having worked at the vendor in question, it feels like you are seeing this through the eyes of the sales team, and not thinking about it from the perspective of the customers needing to actually run the product. Where is your conviction coming from, other than it being the sales mantra of the vendor?
Never worked in sales, but I have worked for other enterprise vendors and their customers quite a bit. From government to automotive and aerospace, to oil and gas to hipster-ish startups. No Fedora anywhere but desktops sometimes. Ever.
This is a time thing, though.
Remember you are talking to loads of people that were passionate about CentOS / EL in the past and switched recently because times have changed. So unless you worked for all of these places in the last 24 months, your experience is moot. All of our experiences would match that, as well.
When I was in banking, EL was king. But the entire point of the article I linked was that things have changed, code is more mature today, projects move faster, priorities in the world have changed.
Your looking to past experiences tells me you are thinking about the past, not the environment today. And you are missing that we had the same experiences back then, too. So it is based on the same experience, same knowledge, but bringing it up to date to today's situations, that led us from EL to Fedora.
-
@scottalanmiller this is foul play indeed, and knowing the company from within, I'm pretty sure such practices aren't smiled upon. I'm not with RH any longer, but in my 10 years there, they have always struck me as the most morally positive company I've ever worked with and for. I've seen business needs sacrificed to doing the right and moral thing quite a few times there. In the end - this is why they succeed, where those who attempt to be business sharks fail (waves at Canonical).
Do keep in mind, you have one single story, I've lived in there for years. I'd say I do have more stats
-
@dyasny said in Testing oVirt...:
Are you really seeing Fedora instabilities that the rest of us are not? Are you really not running any modern code that benefits from current libraries and packages?
Of course I am. pip, cpan, (whatever php has for that same purpose) etc do the job perfectly. There isn't much I cannot do on an EL distribution, one way or another
- But you can't do it as well, or as performant.
- If you replace any libraries to do it, I consider that acknowledgement that Fedora was better, but you were only on EL to make a point.
CAN you do things on EL? Of course. That's not the point. It's whether it is better than Fedora most of the time. And that answer seems to be "no". You make some good points about testing, but don't address that we had already considered that before coming to the conclusion that they weren't good enough to overcome the significant deficits of the process.
-
@scottalanmiller said in Testing oVirt...:
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
How does Fedora protect you? It's a distro packaged, with some bugfixes here and there, no formal QA besides the very basics, no support, nothing. I'm fine with that on my laptop, but on a thousand servers?
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
The apps don't run in a vacuum, they rely on layers of software. Do you test all those layers?
