Server 2012R2 Long Boot Time Caused By Additional non-RAID HDD On RAID controller
-
so we have 3 servers this is affecting. Boot time is 25 minutes for server 1, and 45 minutes for server 2 & 3. Server 2 & 3 have taken up to 2 hours to boot at times. The server is hanging on the Windows Loading screen (Circling dots).
I was able to troubleshoot it this weekend, and have narrowed down the problem to hardware configuration. We have RAIDs on all 3 of these servers, and have 1 additional storage drive on the same controller that is not part of any RAID. If I unplug this drive from the board, the boot time reduces to 2-3 minutes. Once you plug the extra drive back in, the boot time spikes back to 45 minutes.
Why would having an additional HDD connected to the onboard SATA controller cause this? Does anyone have any troubleshooting tips?
Server Motherboards
- Server 1 - SuperMicro X9DR3-LN4F+
- Server 2 - Tyan S2927G2NR-E
- Server 3 - Tyan S2925A2NRF-E
Sata Controller / Chipsets
- LSI MegaRAID LSI00331 (9271-8i) PCI-Express 3.0 x8
- NVIDIA NFP3400
- NVIDIA NFP3400
-
The system is likely trying to start a service on the host, that is failing a system check. Delaying the system boot.
-
@DustinB3403 I am not sure how that is applicable given the facts above. If that was the case, the drive being disconnected would not affect this.
This drive is just being used as a temporary location for backups before they are shuffled over the network to our backup server.
-
The drive connected, is it acting as a file share?
So with the drive connected, reboot the system, go into services.msc and see if the "server" service is taking for ever to start.
Shutdown, remove the drive, and reboot.
See if the same service is starting, and compare the time difference.
-
@DustinB3403 The drive is not being shared and is only being used locally until a local task copies those files over to the backup server
-
Do me the favor then, test it.
Shut down your server, remove the drive. Start the system (and a stop watch) and stop it when the system is running. All while monitoring the Server service.
Then do the same thing, with the drive connected.
-
@DustinB3403 said:
Start the system (and a stop watch) and stop it when the system is running. All while monitoring the Server service.
So after the server has booted up (~45 minutes) you want me to see if the service "Server" is running? And you want me to do the same thing with the drive disconnected? I am not sure what that will accomplish. I am pretty sure it is running immediately after boot (It is running now on all 3 servers), but will not be able to do that until next weekend.
-
The reason that I'd ask you check, is because I have a feeling that at least 1 service (likely the server service) is being hung at startup.
And the only reason I say that is because when you remove the disk, it starts in 2-3 minutes.
I've had similar experiences with external USB drives, causing a server to "hang" at startup because it's trying to host that drive.
-
I've had similar problems with long boots when the NIC card was plugged in, but DNS was wrong. Unplugging would allow the machine to boot in 1 min, plugged in took over 5.
Not saying that you have an IP setting issue or NIC issue, just that as Dustin has suggested, you probably have a service that is hanging during boot due to that drive.
What point is it hanging? What exactly do you see on the screen when it appears to stall out?
-
@Dashrender It is hanging on the Windows Loading screen (Circling dots). After ~45 minutes it will get to the login screen (updated original description)
-
@DustinB3403 Thanks Dustin. I will take a look. I will check to see if that service is running after I log into Windows after the long boot.
-
It'll probably be much easier to test without the drive connected first, as this way you can jump right into services.msc while monitoring what services are actually starting.
Then to test with the drive afterwards.
-
So what exactly is he looking for, Dustin? Are you thinking the server service will be timed out (no running) when the drive is connected?
-
A extreme delay in the service(s) actually starting. Again, I'm just guessing in this case, but that service host all drives and network share fucntions, and has caused massive delays in my experience when a drive is having issues.
Sort the list by Start-Up Type and Automatic. Anything that should be starting automatically, will be at the top, and things that are taking a long time will just sit at the top of the list.
-
But let's assume that, like my NIC problem, that the system just gives up on whatever the problem is, and the service just starts anyway?
I know there is software out there that can track what processes take what time to start at bootup, though I don't know any names of them off the top of my head.
A quick search turned up these two pages
http://lifehacker.com/5089839/track-down-startup-programs-and-processes-with-a-comprehensive-guide
-
In our case (as I believe I said [not being snark]) the issue was specific to a failed External USB RAID device, which when the failed drive was replaced (or the USB RAID device was removed) the system booted without issue.
Once the drive was replaced and resilvered the boot issue was resolved.
You've added some good information on trying to pinpoint the issue though.
-
@DustinB3403 said:
In our case (as I believe I said [not being snark]) the issue was specific to a failed External USB RAID device, which when the failed drive was replaced (or the USB RAID device was removed) the system booted without issue.
Once the drive was replaced and resilvered the boot issue was resolved.
You've added some good information on trying to pinpoint the issue though.
No snark
- I am seriously asking what would I expect to find in your situation when troubleshooting it? -
Specifically, I would be monitoring for extremely slow service start-up due to a drive issue. (and pinpointing which service is the issue)
Compared to when the issue doesn't present it's self.
In my issue I simply removed the external drives rebooted, saw that the issue wasn't persistent. With this knowledge, I then connected the drive, rebooted the system, and pinpointed the cause (as in our case) the cause wasn't obvious.
-
@DustinB3403 said:
Specifically, I would be monitoring for extremely slow service start-up due to a drive issue.
Where do you look for that? Event logs?
-
In this recent issue, I was able to actually just watch the services.msc window without the drive
Compared watching the system start-up and services.msc (all network shares in a non-functional state) with the drive connected, as the system was trying to mount and share this external drive.
