@olivier Wow thanks for that very detailed response. It’s a shame that Citrix isn’t playing nice with SMAPIv3. But it’s also great to hear you guys are working on it anyway!

I’m really impressed with the entire xcp-ng project. Really amazing some of the changes you guys have brought. You seem to have an excellent team of devs.

@Dashrender said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:

@biggen said in No way to create larger than 2TB virtual disk with Xen or XCP-NG?:

@scottalanmiller Ok and that's fine. That's what I need to do then. For the camera server VM I'm working on i want it to record to a couple 12TB Exos X drives. So I have to figure out how to pass them through directly. I think Pete S. had a tutorial I need to hunt dkwn.

Why stay with XCP-NG? why not move over to KVM?

I've used KVM and really like it. Been running it on a Debian host for yeras. For the new host build, I wanted to try something different mainly. I was impressed with Xen and when I found out about xcp-ng I had to give it a spin.

@olivier Great! I think I will just pass through the disks and be done with it. I could have gone the NAS route, but I have enough local storage to not have to use that option.

https://www.anandtech.com/show/14694/amd-rome-epyc-2nd-gen

What advantages does Intel have anymore? Reading this review reads like an obituary for Intel in the server market. Why spend double with Intel for half the cores vs Epyc?

I’m just wondering what Intels next move is here.

So I'm looking at the motherboard I'm pillaging from another system and it has two M.2 slots built into it! Doh! When I bought it last year I must have had a premonition I wanted a dual NVMe setup for a future lab!

@scottalanmiller @Pete-S

Excellent. Thanks for that explanation guys and that nifty diagram Pete!

I guess I was skeptical I had correct what @Pete-S said because I've seen so many reports that its taken days/weeks to rebuild [insert whatever size] TB Raid 6 arrays in the past. But I guess that was because those systems weren't just idle. There was still IOPS on those arrays AND a possible CPU/cache bottleneck.

Pretty cool. I’ll have to try it and see how it goes.

I’ve been using Unbound for several years running on a Raspberry Pi and using a custom black list. Love not having to run ad blockers on each computer browser since it’s all taken care of with Unbound.

@StorageNinja No personal experience with it. I've only ever run RAID 1 or 10. Just the reading I've done over the years from people reporting how long it took to rebuild larger RAID 6 arrays.

BTW, are you the same person who is/was over at Spiceworks? I always enjoyed reading your posts on storage. I respect both you and @scottalanmiller in this arena immensely.

@scottalanmiller said in Reconsidering ProxMox:

@biggen said in Reconsidering ProxMox:

@scottalanmiller What’s your storage configuration like?

I’ve been playing with it on ZFS Raid 1 mirror. Proxmox OS and VMs all on same mirror. Performance is “OK”. Not as good as MD with same setup though.

Wonder if it’s better to create separate Raid 1 ZFS pools. One for the Proxmox OS and one for the VMs.

We don't use ZFS - slow and we don't want its features (few actually do.) LVM is what we use. What is making you want to look at ZFS? It's not meant for speed and has little generally purpose these days. It's not bad, but mostly it's deployed by accident when people aren't sure what it is. Then people swear by "features" that everything has thinking they are unique to ZFS.

ZFS is a great system, with niche applicability.

I wanted to just mirror a SSD pair but thought the only way to "officially" to that with Proxmox was ZFS since they don't support MD.

@scottalanmiller Are you installing these at customer locations? Since you aren't using ZFS with Proxmox are you doing hardware RAID and then using LVM backed storage on top of that for customers?

@Pete-S said in RAID5 SSD Performance Expectations:

@Pete-S said in RAID5 SSD Performance Expectations:

@scottalanmiller said in RAID5 SSD Performance Expectations:

@Pete-S said in RAID5 SSD Performance Expectations:

Having a drive failure will become such an odd failure like having a raid controller, a motherboard or a CPU fail. You'd just replace it and restore the entire thing from backup.

I think drives already fail less than RAID controllers. From working in giant environmnts, the thing that fails more than mobos or CPUs is RAM. That's the worst one as it does the most damage and is hard to mitigate.

The difference though is that mobo, controllers, PSUs, are stateless to the system but drives are stateful. So their failure has a different type of impact, regardless of frequency.

Well, the stateful-ness of the drives is not something we can count fully on, hence the saying "raid is not backup".

What I'm proposing is that when it becomes very unlikely that a drive fails we could rethink our strategy and go for single drives instead of raid arrays. In the very unlikely event that a failure did occur, we are restoring from backup, which we are prepared to do anyway.

With HDDs the failure rate is too high but with enterprise SSDs it's starting to get into the "will not fail" category.

As an example assume we have 4 servers with a RAID10 array of 4 x 2TB drives each. Annual failure rate of HDDs are a few percent, say 3% for arguments sake. With 16 drives in total, every year there is about 50% chance that a drive will fail. So over the lifespan of the servers it's very likely that we will see one or more drive failures.

Now assume the same 4 servers with a single enterprise 4TB NVMe drive in each. Annual failure rate is 0.4% (actual number a few years back). With 4 drives in total, every year there is less than 2% chance that any drive will fail. So over the lifespan of the server it's very unlikely that we will ever see a drive failure at all. Sure, if it does happen anyway, we are restoring from backup instead of rebuilding the array.

As long as you can justify the downtime in the event that a single drive failure takes an entire server down (albeit with a low statistical chance).

If that isn't a concern no use running RAID anyway.

@brianlittlejohn So I guess the days of having a Jr. Admin blind swap are over then? It takes much more care and instruction to use software RAID.

At home I have a Promox host that I have a Win10 VM running Blue Iris (IP camera VMS). I've had it running for some time now (~year) with no issues. It's a simple setup. All my Proxmox VMs are hosted on a SSD. However, I do pass through two large spinning disks to the Blue Iris VM so that I can record video to the mechanical drives and not to the actually SSD that the VM is hosted on. Those two mechanical drives are only used by the Blue Iris VM. No other VM touches them.

So yesterday I'm in the room where the host is located and I hear disk thrashing. You know, the sound of a hard drive being defragged? That kind of disk thrashing. Lots of reads and lots of writes. This is quite unusual as I've never heard those drives make noise ever. I open up my Proxmox GUI to see what is going on and I see that VM is showing a lot of of reads and writes. Proxmox is showing 50MB/s writes and 50MB/s reads all going at the same time. This is highly irregular. At any given time, Blue Iris is writing no more than 4MB/s (from my cameras to the drives) and reading virtually nothing so 50MB/s both ways is waaay of the charts. So I decide to quickly RDP into the VM and see what the hell is accessing the disk and as soon as I RDP in, the drives stop thrashing and disk access returns to normal levels.

So I look around in task manager and everything looks fine and I chalk it up to nothing and log out of the RDP session. Well as soon as I log out, the drive thrashing fires right back up again. I finally figure out its tied to an RDP sessions after a couple times of logging in and out. Here is what Promox is showing for drive I/O for that VM with my annotations:

You can see from the chart that whenever I'm logged out of RDP, disk I/O is real high. But as soon as I login, it dissapears and resumes normal I/O levels that I'm used to seeing which is about 3 - 4 MB/s write and little to no reads.

How can I pin this down since whenever I log in, the task that is causing the issues pauses and nothing shows in task manager?

I came home this afternoon after lunch and noticed the server was back to whisper quiet. Checked the Proxmox I/O graphs and it back to its regular 3MB/s write and barely any reads. So I don't know what the hell Windows was doing for the last 24 hours but it seems to be done.

I'm going to research this further because I have a feeling I'll see it again one day.

@dbeato Gold is reads and teal is writes. Writes were higher (slightly) than reads so you can't make out the gold reads very well.

They use shading for those graphs. It would make more sense to make them transparent so you can see the line colors better.

Who is going to be tasked with changing the tapes if using a single drive? Is that something they are going to keep up with in house??

@scottalanmiller Yeah I just wonder if this is only surveillance related, do they even have an IT dept that can do it?

That autoloader that was linked above is only double the price of a single deck. Unreal how much decks cost.

I use RevelPOS our retail business. Had it for about 7 years now. It’s ok. Uses iPads.

I’d probably do Clover if I had to do it over again but it’s too much of a PITA to switch now.

@travisdh1 Yes, I saw they have the option to enable a "guest network". I was just reading about that here: https://help.ui.com/hc/en-us/articles/115000166827-UniFi-Guest-Network-Guest-Portal-and-Hotspot-System

I'll have to play with it at home to see exactly what it does.

All of my alarms are installed with a cellular communicator only. My alarm guy says he only wires in telephone in the event there is no cellular coverage (which is non-existent unless extremely rural). I haven't had a telephone cellular communicator in over 5 years.