Never Let the Vendor Set Up a Server
-
@Breffni-Potter said:
@scottalanmiller said:
As in.... they aren't concerned about wasting time and money?
Why would you waste money when you can invoice for on site engineer time? = Profit.
If you don't know what is available to you as a customer, how will you know what other alternatives you have?
That's very valid. I've seen a lot of service providers pull that kind of stuff. Customers don't push for good up front setup and it is financially in the interest of the service providers to not push the point so they don't. Although how many customers then react when they find out that that stuff is happening? Most seem to just not care.
This is why someone has to oversee that good practices are being followed, whether it is internal IT or an external function there needs to be somebody responsible for oversight. No one is perfect, of course, but there needs to be a role that is handling that or things are going to be missed all of the time.
-
@scottalanmiller said:
@dafyre said:
Or the vendors charge too much for the software licenses.
Not aware of anyone who charges for basic OOB.
We couldn't download the iLO software for our (umm...) HP servers that I recall without having to pay a few hundred dollars per server. Granted this was several years ago, and we were likely looking in the wrong places, but I don't recall us ever finding even the basic version for free... (I assume OOB means you are talking about things like a remote console type setup?)
-
@dafyre said:
We couldn't download the iLO software for our (umm...) HP servers that I recall without having to pay a few hundred dollars per server.
Download? What is there to download? It's built in. You just plug into it. What server(s) were these? I've used nearly everything from the Compaq era through the G8s and have not seen anything without a free, unlicensed ILO option in all of that time.
In recent times, we always pay for the upgraded ILO because KVMoIP is well worth it. But basic OOB has been free on every machine I've looked at since the DL145 G2 well over a decade ago (before that you had to add on the ILO physically as it was not integrated yet.)
-
@dafyre said:
Granted this was several years ago, and we were likely looking in the wrong places, but I don't recall us ever finding even the basic version for free... (I assume OOB means you are talking about things like a remote console type setup?)
Remote console is not always free, sadly. But remote monitoring and power cycling is always free AFAIK. The KVMoIP is so worth it, though. And you get remote DVD drive too. Which is a way better place to spend the money than paying for a physical DVD drive in the machines!
-
SuperMicro's IPMI is completely open and free.
-
Even before the ILOs were ILOs (before the I was there for Integrated) I was physically adding the ILO external cards to every one of my servers, even at home, since 2000 with the DL360 G1s that used Pentium III.
-
@Breffni-Potter said:
Here's an older video on what ILO is and why it is cool.
Thanks. I was more interested in how you would connect, but I'm guessing that you will need VPN access to our LAN. Not a problem, but not trivial either.
But that video is useful. iLO is definitely something I've ignored too long and I've put it on my task list to setup.
-
@Breffni-Potter said:
And the courier fee does go up and down like a yoyo but it has been £25 for next day with insurance at one time, ballpark £75 at most.
OK. Call it £75. So you're saving me £275 as it stands. Not bad, but not massive. I also forgot to mention that I need an old server removing from the rack, so you will probably want to add a bit on for that. If it sounds trivial, I should point out that it took our "engineer" about half-an-hour to get the rails out. Everyone seems to struggle with our rack.
And I would call him an engineer, regardless of what @scottalanmiller says. The hardest part of the last job was installing the SAS expander card. This is probably trivial once you've done it a couple of times, but our guy hadn't done one before and it did involve a call to HP for help. It seems to be a bit like putting an IKEA sofa
-
Anyway, I'm done. I've always been pretty open on here about my day job and I thought it would be helpful as there aren't that many end users on ML compared with people working for IT companies. But I'm tired of constantly getting told that I'm doing it all wrong. That my partners are "ripping me off" or that I have "unskilled screwdriver guys that you are paying for Exchange work". I respect different opinions, but opinions are all they are. I've been doing this job for the majority of my working life and I think I do a fine job. I'm going to try shutting up now and hopefully let someone else take the grief for a bit
<throws rest of toys out of pram>
-
@scottalanmiller Thanks for clearing that up! It may have been that we were after the remote console. These servers are about 10 years old and the administration at my last job didn't like spending money when IT asked at the time, so that may be why we never got it. I was unaware that ILO, et al did anything other than KVMoIP and Remote DVD.
-
@dafyre said:
@scottalanmiller Thanks for clearing that up! It may have been that we were after the remote console. These servers are about 10 years old and the administration at my last job didn't like spending money when IT asked at the time, so that may be why we never got it. I was unaware that ILO, et al did anything other than KVMoIP and Remote DVD.
ILO4, which is much newer than what you have, also adds full remote monitoring capability where you don't even need an HP SIM server to get that data. They keep adding more and more to the ILO.
-
@Carnival-Boy said:
@Breffni-Potter said:
Here's an older video on what ILO is and why it is cool.
Thanks. I was more interested in how you would connect, but I'm guessing that you will need VPN access to our LAN. Not a problem, but not trivial either.
But that video is useful. iLO is definitely something I've ignored too long and I've put it on my task list to setup.
VPN or you can punch a hole in the firewall to allow the port through or you can use a jump station like a desktop with LogMeIn, ScreenConnect or similar. VPN is not something I would start with for MSP or MSP-like support until you have dedicated external staff who are full time or nearly full time with you and their machines can be treated as part of your network.
-
@Carnival-Boy said:
@Breffni-Potter said:
Here's an older video on what ILO is and why it is cool.
Thanks. I was more interested in how you would connect, but I'm guessing that you will need VPN access to our LAN. Not a problem, but not trivial either.
But that video is useful. iLO is definitely something I've ignored too long and I've put it on my task list to setup.
We have dialup networks for routers, switches and DRACs. That's in addition to the VPN of course.
-
@Carnival-Boy said:
That my partners are "ripping me off" or that I have "unskilled screwdriver guys that you are paying for Exchange work".
I'm confused, I thought that these were things that you had told us, not us to you. You have complained several times about how you can't find an MSP that you like, have stated that your sales people have tried to rip you off by selling you things that clearly you didn't need (and you didn't buy) and that you were paying the same thing for both your bench and IT work. Those things specifically were things I remember you telling us (at least two of them in this thread, the SAN thing in another.)
Yes, we've pushed you to evaluate how you look at those relationships and are trying to get everyone to think more broadly and consider how to approach IT differently for everyone betterment. But specifically the things that you are thinking of are ones that you had told us and then we discussed.
-
@Carnival-Boy keep in mind that in several of these cases the question about how to approach this came up as an attempt to document best practices, based on tons of observation, peer review, etc. This thread itself, for example, is based on decades of experience and tons of peer review and tons of observations of companies making this exact mistake. The timing of this article was because yet another company had a major outage caused by not following this best practice. The article and the post had nothing to do with you, no one knew what you had been doing.
Your response here was to state how you were doing things differently and not following this practice. That's fine, maybe you have a case where the best practice would not apply or maybe it isn't really a best practice and that should be addressed. But you were essentially taking the tact of either "this is wrong" or "maybe this does not apply to me."
From the discussion, did we not determine that very key, basic server setup items like OOB management setup was being missed by your installation guys and that you were overpaying by a wide margin and that you had a support knowledge gap in the processes? Was this not helpful?
You say that you feel that you have been doing a good job and I'm sure that you have. But has this not been useful and insightful and allowed for you to potentially do an even better job with better IT coverage at lower cost? Would it not be disrespectful to you as an IT practitioner to not probe into these items and look for improvement?
At the end of this discussion, while finding out that something we did in the past could have been improved, didn't this make things better for the future or at very least provide important perspective? You seem upset, but I felt like we genuinely helped you here and provided value to you (and hopefully to others who come along and read this as well.)
-
And that last piece has to be noted - in cases like this where we are specifically working to develop or document best practices for the industry, or are reviewing them, discussing them, etc. and you post a dissenting viewpoint we really have no choice but to discuss that and prove because either the best practices are wrong and need to be adjusted or the fact that there is conflicting information needs to be cleared up so that someone who has not delved into this thoroughly does not get confused.
-
@Carnival-Boy said:
And I would call him an engineer, regardless of what @scottalanmiller says. The hardest part of the last job was installing the SAS expander card.
What would be his full title then? What kind of engineer do you feel that he is?
As installing a SAS card is no different than installing a graphics card, do you feel that every kid at Geek Squad that installs GPUs and hard drives is an engineer? If not, what's the difference and where do you draw the line to differentiate between the two tasks of plugging in parts to a computer?
Technically, the Geek Squad work is harder because they have to figure out compatibility, performance and other factors and work with a variety of designs. The server work is using pre-defined parts without that decision making process. But in both cases the job is just "plugging stuff in."
-
@Carnival-Boy said:
I also forgot to mention that I need an old server removing from the rack, so you will probably want to add a bit on for that. If it sounds trivial, I should point out that it took our "engineer" about half-an-hour to get the rails out. Everyone seems to struggle with our rack.
This is pretty common. We've had many servers get "stuck" even in major datacenters. It really can take a while especially with older gear and older racks. But all the more reason to not pay IT staff to do manual labour work too. You can be the best IT person in the world and have no clue how to get things out of a rack or even be strong enough to lift a server. The skill of lifting a computer and putting rails on a rack is very different than that of IT and has crossovers with lots of other fields. Music equipment is racked the same way, for example, and some manufacturing equipment.
-
@Carnival-Boy said in Never Let the Vendor Set Up a Server:
And I would call him an engineer, regardless of what @scottalanmiller says. The hardest part of the last job was installing the SAS expander card. This is probably trivial once you've done it a couple of times, but our guy hadn't done one before and it did involve a call to HP for help. It seems to be a bit like putting an IKEA sofa
You wouldn't call someone a furniture engineer just because they assemble IKEA stuff, right? My wife doesn't call herself an engineer, but she does that all the time. The definition of engineer is that they design the systems, that's the English word definition, right from Cambridge. Someone that just "puts in a card" isn't designing anything. In fact, they aren't even doing IT work (at that moment), it's just manual labour in the bench space.
https://mangolassi.it/topic/11651/comparing-the-roles-of-engineer-and-administrator
-
Great example came in today. Someone had a Dell server, four matching drives. The system arrived with no virtualization configured and the OS was installed without RAID on a single drive. Each drive was attached as an individual drive. Obviously Dell never intended someone to use the system like that, even for a desktop that's not an acceptable setup. It's pretty clear that it was just a test install to show that the hardware was working.
But several people said "but Dell set it up this way, obviously it is okay" and it has been running in production and is now a disaster.
