Testing oVirt...
-
@scottalanmiller this is foul play indeed, and knowing the company from within, I'm pretty sure such practices aren't smiled upon. I'm not with RH any longer, but in my 10 years there, they have always struck me as the most morally positive company I've ever worked with and for. I've seen business needs sacrificed to doing the right and moral thing quite a few times there. In the end - this is why they succeed, where those who attempt to be business sharks fail (waves at Canonical).
Do keep in mind, you have one single story, I've lived in there for years. I'd say I do have more stats
-
@dyasny said in Testing oVirt...:
Are you really seeing Fedora instabilities that the rest of us are not? Are you really not running any modern code that benefits from current libraries and packages?
Of course I am. pip, cpan, (whatever php has for that same purpose) etc do the job perfectly. There isn't much I cannot do on an EL distribution, one way or another
- But you can't do it as well, or as performant.
- If you replace any libraries to do it, I consider that acknowledgement that Fedora was better, but you were only on EL to make a point.
CAN you do things on EL? Of course. That's not the point. It's whether it is better than Fedora most of the time. And that answer seems to be "no". You make some good points about testing, but don't address that we had already considered that before coming to the conclusion that they weren't good enough to overcome the significant deficits of the process.
-
@scottalanmiller said in Testing oVirt...:
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
How does Fedora protect you? It's a distro packaged, with some bugfixes here and there, no formal QA besides the very basics, no support, nothing. I'm fine with that on my laptop, but on a thousand servers?
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
The apps don't run in a vacuum, they rely on layers of software. Do you test all those layers?
-
@dyasny said in Testing oVirt...:
@scottalanmiller this is foul play indeed, and knowing the company from within, I'm pretty sure such practices aren't smiled upon. I'm not with RH any longer, but in my 10 years there, they have always struck me as the most morally positive company I've ever worked with and for. I've seen business needs sacrificed to doing the right and moral thing quite a few times there. In the end - this is why they succeed, where those who attempt to be business sharks fail (waves at Canonical).
Do keep in mind, you have one single story, I've lived in there for years. I'd say I do have more stats
Inside stats are misleading, though. Vendors paint a different picture for employees than what customers see. That's not an RH thing, that's universal.
I guarantee as an employee, there were zero stats kept about how they were doing unethical, maybe illegal, things with customers. Account managers don't report to engineering.
That they had a process for this at all means it wasn't casual. This was a major effort to do what they did. If you didn't hear about it while there, then your stats aren't valid, if that makes sense. Because I bet you were told that nothing like that was happening with major customers (we had to be one of the five biggest customers.)
We had the same thing internally at where I was, so I get how it happens. We happen to catch it in our case. HR was trying to sabotage some departments and telling people that the jobs were awful before those applicants went fully into the system. So the "stats" internally said one thing, but if you could find someone who had turned down a job with the company, you learned all kinds of secrets that weren't officially recorded anywhere.
Same with RH - there is no chance that that activity was on the books. But it was a major effort to manipulate the customer. They could easily guarantee that their account would not be dropped if the heads of the engineering departments were working at RH and not at the customer.
-
@dyasny said in Testing oVirt...:
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
The apps don't run in a vacuum, they rely on layers of software. Do you test all those layers?
Individually? No. Together, yes.
-
@dyasny said in Testing oVirt...:
@black3dynamite said in Testing oVirt...:
Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.
Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?
Please tell me what the point is in CentOS running PHP 5.6?
I mean, Look how old it is, and look when it looses support!
Fedora 28 uses 7.2.x, FAR FROM BLEEDING EDGE (ffs!). And oh looky, supported for longer than 2 more months lol.
Have fun upgrading the CentOS LTS servers you use to the next CentOS LTS... EVERYTHING will break, including all of your PHP apps.
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
How does Fedora protect you? It's a distro packaged, with some bugfixes here and there, no formal QA besides the very basics, no support, nothing. I'm fine with that on my laptop, but on a thousand servers?
How doesn't matter, that it does it and does it reliably is what matters.
We can postulate any number of reasons "why" or "how". Maybe having the individual packages more up to date simply matters that much. Maybe extensive testing is a waste and not as valuable as it sounds. Maybe Fedora tests more than you realize (maybe unofficially.)
Bottom line, though, is that Fedora has reliably protected. EL has, too. It's not that EL is bad, it's that Fedora just has come out better in real world use.
There are lots of things that matter at the end of the day...
- Reliability
- Speed
- Security
- Compatibility
And more, of course. When taken as a whole, EL might win on reliability, but the margin of winning is nominal. But Fedora seems to win on all of the others. Sometimes nominally, sometimes by quite a margin.
-
@obsolesce said in Testing oVirt...:
@dyasny said in Testing oVirt...:
@black3dynamite said in Testing oVirt...:
Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.
Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?
Please tell me what the point is in CentOS running PHP 5.6?
I mean, Look how old it is, and look when it looses support!
Fedora 28 uses 7.2.x, FAR FROM BLEEDING EDGE (ffs!). And oh looky, supported for longer than 2 more months lol.
Have fun upgrading the CentOS LTS servers you use to the next CentOS LTS... EVERYTHING will break, including all of your PHP apps.
One COULD argue that RHEL goes out of support when PHP does. From an application perspective, using RHEL 7 would be "unsupported".
By that logic, which is pretty solid in reality, Fedora is the supported OS, not RHEL, come January.
-
@scottalanmiller said in Testing oVirt...:
Remember you are talking to loads of people that were passionate about CentOS / EL in the past and switched recently because times have changed. So unless you worked for all of these places in the last 24 months, your experience is moot. All of our experiences would match that, as well.
So one second you say business is slow to change, and then you say that is moot. OK.
When I was in banking, EL was king. But the entire point of the article I linked was that things have changed, code is more mature today, projects move faster, priorities in the world have changed.
Code is, if anything, less mature today. With the current craze of moving as fast as possible, release as fast as possible and skip testing everywhere ("we've got devs writing unit tests for that" - sounds familiar?) code is getting crappier every day. And with solutions like k8s and mesos, people stopped debugging, their answer is "just respawn, who cares if it fails every 5 minutes".
Your looking to past experiences tells me you are thinking about the past, not the environment today. And you are missing that we had the same experiences back then, too. So it is based on the same experience, same knowledge, but bringing it up to date to today's situations, that led us from EL to Fedora.
Oh leave the slogans aside, neither of us has any future experience, just the past. You can guess what will happen in the future, but you dont know, no matter how much you "look to the future". You found Fedora to be working for your use case - well and good. This doesn't mean EL is less stable or has less support, you simply found a niche use case which works for you. Huge difference
-
@obsolesce said in Testing oVirt...:
Have fun upgrading the CentOS LTS servers you use to the next CentOS LTS... EVERYTHING will break, including all of your PHP apps.
This is a big one. Fedora has a consistently rock solid update path. EL does not. Moving EL versions has always been a hassle and tends to cause apps to break like crazy. Fedora isn't perfect, nothing is, but the difference is shocking. Keeping Fedora up to date is trivial, EL is an effort (akin to Windows.)
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
Remember you are talking to loads of people that were passionate about CentOS / EL in the past and switched recently because times have changed. So unless you worked for all of these places in the last 24 months, your experience is moot. All of our experiences would match that, as well.
So one second you say business is slow to change, and then you say that is moot. OK.
You are arguing for what businesses "do", which is irrelevant. That most companies do things badly is of no concern to us. What we care about is how should we do things to do them well.
That "what good looks like" has changed and that's what we as IT pros care about. Doing what "everyone else does" is a recipe for disaster. We discuss this all the time. The "average" IT shop is terrible, and the average business loses money and fails. So what "everyone else does" is interesting to note, and worth looking at, but never a reason to not evaluate needs and look at the real world.
But the point was, you used an example of "what people do" that exactly matched both cases - those that would switch to Fedora and those that don't. It's moot because you equally supported both points.
-
@scottalanmiller said in Testing oVirt...:
Inside stats are misleading, though. Vendors paint a different picture for employees than what customers see. That's not an RH thing, that's universal.
I've spent plenty of time on the customer side.
I guarantee as an employee, there were zero stats kept about how they were doing unethical, maybe illegal, things with customers. Account managers don't report to engineering.
That they had a process for this at all means it wasn't casual. This was a major effort to do what they did. If you didn't hear about it while there, then your stats aren't valid, if that makes sense. Because I bet you were told that nothing like that was happening with major customers (we had to be one of the five biggest customers.)
We had the same thing internally at where I was, so I get how it happens. We happen to catch it in our case. HR was trying to sabotage some departments and telling people that the jobs were awful before those applicants went fully into the system. So the "stats" internally said one thing, but if you could find someone who had turned down a job with the company, you learned all kinds of secrets that weren't officially recorded anywhere.
Same with RH - there is no chance that that activity was on the books. But it was a major effort to manipulate the customer. They could easily guarantee that their account would not be dropped if the heads of the engineering departments were working at RH and not at the customer.
In a large company, stuff done by some members can be unethical. I seriously doubt it was something decided at the top company level though, and if you cared to complain all the way to the top, heads would have flown.
when I worked at a hardware vendor we all know quite well, there was a sales team hired somewhere in Asia to do UK sales. They were told to sell as much as possible, provided training and left alone. A month later, a call came in from some old lady who was sold a rack of SAN equipment when she called for a printer. After an investigation, the entire team was sacked and the customers reimbursed. Does that make that vendor evil?
-
@dyasny said in Testing oVirt...:
This doesn't mean EL is less stable or has less support, you simply found a niche use case which works for you. Huge difference
Same in reverse. You started this by pushing that EL was so much better because it gets testing and more support. You made us explain why that doesn't work for us (and anyone of whom we know.) It has a bigger support process, but we aren't convinced the support is actually better in end results. We've shown that it may not even be able to claim to have support technically in a few weeks any longer. And that "EL works for you" is fine, but we are saying that they both work, but one works "better".
You are requiring the position be that EL doesn't work at all, but that's not the case. It's that we believe Fedora has pulled ahead. And that's where I feel you aren't reading what we are writing. You are giving lots of great examples of why EL has some great value and advantages. But you aren't explaining why you feel they are more important than the reasons why Fedora has advantages.
Everything you are saying about EL we already knew and accepted and used and believe fell in value in the last two years to the point where Fedora pulled ahead in real world desirability for most production environments. And in all of this, I feel like you are only arguing for getting up to the starting point of the discussion where we were before we first looked into this years ago.
So consider it that we accept everything you said, and thought that it was all assumed, and didn't realize it needed to be stated. Now from this point, explain why you feel that outweigh's Fedora's advantages such as actual package support, working applications, no significant kernel issues, higher speed, more mature code, more packages, more features, working upgrade paths, etc.
-
@dyasny said in Testing oVirt...:
In a large company, stuff done by some members can be unethical. I seriously doubt it was something decided at the top company level though, and if you cared to complain all the way to the top, heads would have flown.
Maybe, but they probably should have been following up with customers, too. This went very, very high at the customer and ended up affecting the RH relationship at multiple F100 companies. And has been published publicly more than a few times. RH has never once acknowledged, reprimanded, explained or apologized.
I think it's unreasonable that senior management doesn't know. Either they are putting their heads in the sand and don't want to know what customers are experiencing; or they are complacently allowing stuff like this to happen.
Maybe they realized the risk was too large and backed off. Maybe they didn't realize they would get caught. Who knows. What I know is, there's no plausible way they didn't either ignore it or outright condone it.
-
@dyasny said in Testing oVirt...:
when I worked at a hardware vendor we all know quite well, there was a sales team hired somewhere in Asia to do UK sales. They were told to sell as much as possible, provided training and left alone. A month later, a call came in from some old lady who was sold a rack of SAN equipment when she called for a printer. After an investigation, the entire team was sacked and the customers reimbursed. Does that make that vendor evil?
So the did something to fix the situation? That's the difference between them and RH. RH has never once attempting to rectify it to the customer, or to the employees.
So you see why I see RH as very, very different.
And overselling to someone is the job of sales, violating professional and business ethics is not. Nor is attempting to poach customer staff.
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
How does Fedora protect you? It's a distro packaged, with some bugfixes here and there, no formal QA besides the very basics, no support, nothing. I'm fine with that on my laptop, but on a thousand servers?
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
The apps don't run in a vacuum, they rely on layers of software. Do you test all those layers?
How much stuff are you running on the servers?
For example, I have a bunch of Fedora 28 web servers. They run the typical LAMP stack.
There was a PHP dev here who designed some LoB PHP app, but on old version of CentOS. And when I suggested we move to a new server (due to hundreds of complaints by users a month), I proposed Fedora, built one, and helped him migrate everything over. Of course, it was designed to use old AF PHP, so it needed to be fixed, and he didn't want to do that or take the time to do it.
But I emailed him back regarding PHP 7, and he was absolutely hooked. Just going to PHP 7 by itself was a several-fold performance improvement. And that's excluding going from old AF MySQL, to MariaDB, as well as all the CURRENT packages of software.
The end result was an incredibly improved end user experience, lots of compliments all around. Why? Because current software.
-
@obsolesce said in Testing oVirt...:
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
How does Fedora protect you? It's a distro packaged, with some bugfixes here and there, no formal QA besides the very basics, no support, nothing. I'm fine with that on my laptop, but on a thousand servers?
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
The apps don't run in a vacuum, they rely on layers of software. Do you test all those layers?
How much stuff are you running on the servers?
For example, I have a bunch of Fedora 28 web servers. They run the typical LAMP stack.
There was a PHP dev here who designed some LoB PHP app, but on old version of CentOS. And when I suggested we move to a new server (due to hundreds of complaints by users a month), I proposed Fedora, built one, and helped him migrate everything over. Of course, it was designed to use old AF PHP, so it needed to be fixed, and he didn't want to do that or take the time to do it.
But I emailed him back regarding PHP 7, and he was absolutely hooked. Just going to PHP 7 by itself was a several-fold performance improvement. And that's excluding going from old AF MySQL, to MariaDB, as well as all the CURRENT packages of software.
The end result was an incredibly improved end user experience, lots of compliments all around. Why? Because current software.
Oh, back to my initial point lol (forgot).
Are you using your Fedora servers to do so many different things, so many different roles and so many different softwares?
How many bugs are in current software that you're so worried about? No bugs in any of the stuff running on our Fedora servers.... so not sure where that fits.
-
Maybe this summary will help to explain why we feel the way that we do...
I think we all agree, and have always felt that...
- EL is an excellent product.
- EL has great support.
And then that...
- Fedora is an excellent product.
- Fedora support is good enough that we see nominal value in anything more.
But then that...
- Fedora includes features and performance benefits that are more than nominal.
-
@obsolesce said in Testing oVirt...:
How many bugs are in current software that you're so worried about? No bugs in any of the stuff running on our Fedora servers.... so not sure where that fits.
Right, this is my point. We run all kinds of workloads on Fedora and see zero concerns with bugs. PHP is a big one, but we run lots of other things, too. It's our database platform, it's our dev platform, it's our app platform. We haven't experienced any of these issues, we know of no one experiencing them.
And since the apps we use are tested on Fedora (when they are not, we use something else, like Zimbra on CentOS - which really shows the performance problems of being old and not kept up to date like we'd like) there isn't a lot of opportunity for new, unknown bugs to crop up since the entire stack is tested.
-
@scottalanmiller said in Testing oVirt...:
@obsolesce said in Testing oVirt...:
@dyasny said in Testing oVirt...:
@black3dynamite said in Testing oVirt...:
Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.
Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?
Please tell me what the point is in CentOS running PHP 5.6?
I mean, Look how old it is, and look when it looses support!
Fedora 28 uses 7.2.x, FAR FROM BLEEDING EDGE (ffs!). And oh looky, supported for longer than 2 more months lol.
Have fun upgrading the CentOS LTS servers you use to the next CentOS LTS... EVERYTHING will break, including all of your PHP apps.
One COULD argue that RHEL goes out of support when PHP does. From an application perspective, using RHEL 7 would be "unsupported".
By that logic, which is pretty solid in reality, Fedora is the supported OS, not RHEL, come January.
Yeah, but why purposely run bad performing software for years when there's a better and more stable option?
