Dell PERC Question (Server Down)

BRRABill

Well, over the weekend I got all the important data off this array, and this morning I TRASHED IT!

I unplugged all the drives, then re-plugged in all DELL-only drives.

The arrays built, initialized, and thus far seem fine.

I guess there is no way to test except actually putting the data back on there, so that will be the next step.

Oh, and prayer.

DustinB3403

Wish you the best.

BRRABill

@DustinB3403 said

Wish you the best.

Thanks!

BRRABill

When I plugged the DELL drives in last week, two of the three of them failed.

Today, those same DELL drive, in the same slots, are all working fine.

I wonder if there really was something corrupted in the array because of the other drives. DELL mentioned that as a good possibility ... but is that really possible?

But so far it seems like the PERC and the backplane are working fine.

StrongBad

What's the latest on this cluster (pun intended.)

BRRABill

I have not put any more data back on there yet, but the server has been up (with the array intact) for 24 hours now. Seems to be no issues, but the array had no issues for months until I started using it in production, so we'll have to wait and see. (Assuming it was the extra activity that got it.)

travisdh1

@BRRABill said in Dell PERC Question (Server Down):

I have not put any more data back on there yet, but the server has been up (with the array intact) for 24 hours now. Seems to be no issues, but the array had no issues for months until I started using it in production, so we'll have to wait and see. (Assuming it was the extra activity that got it.)

Is the array actually mounted somewhere? I'd try hammering it for a bit. IE:

dd if=/dev/random of=/dev/array bs=4k count=1000000000

will write 4gb of random stuff to the array.

BRRABill

As part of this discussion about booting off of USB, I decided that since I needed to redo my entire XS installation anyway, I would install to USB.

But as I was doing a little research, I found online people recommending against this, since XS (and other hypervisors, too, I'm not sure) writes a lot to the boot disk. The fear is that this constant writing with fry a USB stick that doesn't have the smarter controllers that SSDs have. I spoke with @scottalanmiller a little bit about this, and he said it made sense, and to definitely get the logs off there.

So my question is ... how many of you that boot off USB have had issues with your USBs not booting? Do you agree this might be an issue? (Yes, I know that you can clone another USB, but that;s not really the point. )

And how many of you that boot XS off USB are sending your logs elsewhere?

BRRABill

@travisdh1 said

Is the array actually mounted somewhere? I'd try hammering it for a bit. IE:
dd if=/dev/random of=/dev/array bs=4k count=1000000000
will write 4gb of random stuff to the array.

The problem is that it was pretty random. I'm not even sure that kind of test would make me feel any better.

Dashrender

I only just started using XS, and it is installed on an SD card. The logs have not be moved yet.. but really, the logs should be move regardless to your central log catching server.

DustinB3403

@Dashrender said in Dell PERC Question (Server Down):

I only just started using XS, and it is installed on an SD card. The logs have not be moved yet.. but really, the logs should be move regardless to your central log catching server.

You can even specify a default location (NFS / SMB) for your logs to get pushed too so they aren't on the boot device.

BRRABill

@DustinB3403 said

You can even specify a default location (NFS / SMB) for your logs to get pushed too so they aren't on the boot device.

Do you mean be using the syslog forwarding in XC? (And also in the CLI?)

I did set that up. However, I noticed that the logs were still recording themselves on my boot device.

I Googled this, and apparently you have to go into /var/lib/syslog.conf and comment out all the local locations. And then you have to restart the syslog daemon ... which overwrites the config file. It does this on reboot, too.

Is that what you meant? If so, take a look in your logs directory and you'll probably see the same thing I saw ... even though you are forwarding, it is still writing to the boot drive. Check it out!

(Unless you meant actually moving the local logging location to another storage device which is also something I saw mentioned, but it much more entailed.)

This is the article I am referencing:
http://xenserver.org/discuss-virtualization/virtualization-blog/entry/log-rotation-and-syslog-forwarding.html

scottalanmiller

@BRRABill said in Dell PERC Question (Server Down):

@DustinB3403 said

You can even specify a default location (NFS / SMB) for your logs to get pushed too so they aren't on the boot device.

Do you mean be using the syslog forwarding in XC? (And also in the CLI?)

No, he just means mounting an NFS device at the logging location.

scottalanmiller

@BRRABill said in Dell PERC Question (Server Down):

(Unless you meant actually moving the local logging location to another storage device which is also something I saw mentioned, but it much more entailed.)

It's actually very simple and standard. You just move the old log location from /var/log to /var/log_old. Then you make mkdir /var/log and then you mount an NFS share to that spot. That's all and then the logs go directly to the NFS device.

Dashrender

@scottalanmiller said in Dell PERC Question (Server Down):

@BRRABill said in Dell PERC Question (Server Down):

(Unless you meant actually moving the local logging location to another storage device which is also something I saw mentioned, but it much more entailed.)

It's actually very simple and standard. You just move the old log location from /var/log to /var/log_old. Then you make mkdir /var/log and then you mount an NFS share to that spot. That's all and then the logs go directly to the NFS device.

While Windows can do this, this is so rarely used to be more myth than anything

BRRABill

@Dashrender said

While Windows can do this, this is so rarely used to be more myth than anything

I used it once to set up a folder to my USB thumb drive.

BRRABill

@BRRABill said

This is the article I am referencing:
http://xenserver.org/discuss-virtualization/virtualization-blog/entry/log-rotation-and-syslog-forwarding.html

I ended up trying the "dirty, dirty" trick in the comments tonight. Worked like a charm, and thankfully didn't blow anything up.

BRRABill

@BRRABill said

I ended up trying the "dirty, dirty" trick in the comments tonight. Worked like a charm, and thankfully didn't blow anything up.

Still not logging. Sweet.

I wonder why he said he didn't recommend doing this? I can't think of any reason. (Other than it gets re-written on updates.)

BRRABill

File this under the "you learn something new every day" category...

I have two SanDisk Cruzer Fit USB sticks I am running my two XSs off of.

The one has no LED, and the other flashes every 3 seconds.

Odd, I thought. I contacted SanDIsk, thinking perhaps one was broken or something.

This is the chart of "acceptable" behaviors, and they said it's possible two of the same drives bought at the same time could exhibit different behaviors.

LED is a solid light and blinks only when transferring files to the device.
LED blinks rapidly during initialization then turns off. LED then only blinks when data is being transferred.
LED breaths (slowly fades in and out) when device is not being used and blinks rapidly when data is being transferred.

scottalanmiller

Basically they are saying that the lights are meaningless and pointless and even they don't know why they are there.