@biggen said in RAID rebuild times 16TB drive:

@scottalanmiller @Pete-S

Excellent. Thanks for that explanation guys and that nifty diagram Pete!

I guess I was skeptical I had correct what @Pete-S said because I've seen so many reports that its taken days/weeks to rebuild [insert whatever size] TB Raid 6 arrays in the past. But I guess that was because those systems weren't just idle. There was still IOPS on those arrays AND a possible CPU/cache bottleneck.

We don't see any bottlenecks on our software RAID-6 arrays but they run bare metal on standard servers. That might be atypical, I don't know.

But I think regular I/O has a much bigger effect than any bottleneck. I can see how MB/sec takes a nose dive when rebuilding and there is activity on the drive array.

If you think about it, when the drive only does rebuilding it's just doing sequential read/writes and hard drives are up to 50% as fast as SATA SSDs at this. But when other I/O comes in, it becomes a question of IOPS. And hard drives are really bad at this and only have about 1% of the IOPS of an SSDs.

@StorageNinja said in Dell PERC H740 with SSDs?:

Ehhh, be careful here. The PM8xx is the bargain basement cheap Samsung SATA devices. Samsung in the earlier versions of this series had some... interesting firmware bugs so make sure you patch your drives.

Always good to update the firmware if you're having problems.

I wouldn't call the PM series "bargain basement cheap Samsung SATA devices". It's their datacenter series for read-intensive workloads. What most workloads are when you use the SSDs as local storage on a VM host. It's going to be running SQL server with 30 users. That's the OPs use case - not as cache for an enterprise storage array.

Of course there are faster and better products but you got to put things in perspective.

@black3dynamite said in Dell PERC H740 with SSDs?:

@biggen said in Dell PERC H740 with SSDs?:

@scottalanmiller Just not as commonplace to find enterprise gear to whitebox with. Where are you looking for enterprise motherboards?

Supermicro

And Tyan, Gigabyte, Intel etc. Goto newegg and look under server motherboards. Maybe you can find suppliers on ebay and amazon as well.

Vendors that have enterprise stuff are not always selling directly to consumers. Consumers requires a lot of support and hand holding, while system integrators who are their usual customers are suppose to know what they are doing.

I was looking for a distro for the kids to use for their school work.

Ubuntu Desktop was suggested and it has worked perfectly. It's easy to use, nice to look at and stable. Easy to support too.

Here's another test. It shows that IPsec is more than 50% faster than WireGuard.

https://www.pcwrt.com/2020/02/performance-comparisons-of-three-vpn-protocols-on-a-budget-router/

Thing is that WireGuard uses the ChaCha20 cipher which is very efficient and fast on non-dedicated hardware. IPsec, and sometimes also OpenVPN, can however often use hardware acceleration on AES and is then faster.

It's the devices in each end and their architecture (ARM, x86 etc) and any hardware offloading that will determine what to pick for maximum performance on a VPN. And often it just doesn't matter - for instance when the hardware can handle encryption at WAN speed.

Nice thing about WireGuard is that it has just been included in the 5.6 kernel so soon enough it will be available by default on every linux system.

@scottalanmiller said in Import a QCOW2 Into Proxmox:

If you already have a QCOW2 file, either coming from another KVM system or converted from another format, to use it in Proxmox you need to import it because you cannot copy the files directly into the storage location.

It sure looks like you can copy image files straight in. It just has to be in the right directory / have the right file name according to the target VM ID.
https://pve.proxmox.com/wiki/Moving_disk_image_from_one_KVM_machine_to_another

@CCWTech said in Associate existing drive with new Proxmox VM:

@scottalanmiller said in Associate existing drive with new Proxmox VM:

@Pete-S said in Associate existing drive with new Proxmox VM:

You have that info in the thread below from the other day:
https://mangolassi.it/topic/21751/import-a-qcow2-into-proxmox

Basically you need to put it in the right place and it has to have the right name (depending on config file, what filesystem, VM ID)

Right, it seems that you just go into the config and edit the name of the drive to point to the correct volume name and ta da.

What do you need to do in order to change the boot order?

Boot order should logically speaking be in the VM settings since it's a BIOS setting in a non-virtual machine.

What is your recommendation year 2020 for a password manager for ordinary users running O365 and such?

Has to support desktop and mobile use. Has to be able to share passwords as well. Has to be simple to use and as close to zero maintenance as possible.

@scottalanmiller said in Looking to Buy a SAN:

Yeah, just checked, it iSER. It puts iSCSI directly over RDMA so it's crazy fast and straight through the RAM channel. You can do it on Infiniband so that it is faster than FC or Ethernet can do.

https://www.snia.org/sites/default/files/ESF/FCoE-vs-iSCSI-vs-iSER-Final.pdf

iSER was presented at MangoCon 2018. Max gave a good talk on it.

I believe NVMe-oF (NVMe over Fabric) has the highest performance nowadays and is expected to replace iSCSI (and iSER). It can also use RDMA but uses the much leaner NVMe protocol so it will always be faster over the same type of connection.
https://www.snia.org/sites/default/files/news/iSCSI-Future-Cloud-Storage-Doomed-NVMe-oF.pdf

Thanks guys. I'm activating it right now and will start playing with it.

Of the options you listed, S/MIME is the only one that is standard. Meaning you can send secure mails to others as well.

@scottalanmiller said in encrypted email options?:

That's secure email. That it's transparent makes it even more powerful. My point has been for years - all standard email is fully secure.

Well, it doesn't prevent your email provider from reading your emails. Google and their ilk will use your emails to profile you and whomever you email. So when your doctor sends an email about your cancer treatment, you are going to start seeing ads about that on every site.

You'll get a much higher degree of security when you have real encrypted email and especially so when the email provider doesn't have your private key to decrypt. But then any web mail solution is out.

OpenPGP for instance requires both a private key and a passphrase to be able to decrypt emails. Works great with native emails clients that support OpenPGP. But I wouldn't want to be the guy supporting that for general end users.

@Dashrender said in encrypted email options?:

@scottalanmiller said in encrypted email options?:

With Zix, for example, they have your key.

yep

PGP/GPG options are definitely way, way more secure. And they use real email to do it. But no one likes them.

Right, normal users, a cashier at the grocery store, will likely never setup a GPG key, support it, etc to get medical records via email from their doctor. that's to much work.

But they will create a logon to an EMR portal and a "secure email portal" to access/retrieve messages from their doctors. Of course, generally, they will use the same password they use everywhere else, but that's not my problem.

I know cases where communication between health care professionals and patients is done through a website that requires 2FA for the patients. It has absolutely nothing to do at all with email except that patients can get email and sms notifications. Patients can view their journals as well. I'm sure it's custom built but maybe there are COTS systems that are made for exactly this.

@brandon220 said in RAID5 SSD Performance Expectations:

I was looking at some specs on one of my machines and decided to look at the difference for a SSD and spinner. Pretty interesting... The IOPS difference is more than I would have guessed.

Yes, and if you would've put in a NVMe enterprise SSD in the mix it would have been crazy. Expect 2000-3000MB/sec and 200-600 thousand IOPS - for a single drive.

@biggen said in RAID5 SSD Performance Expectations:

@Pete-S said in RAID5 SSD Performance Expectations:

@Pete-S said in RAID5 SSD Performance Expectations:

@scottalanmiller said in RAID5 SSD Performance Expectations:

@Pete-S said in RAID5 SSD Performance Expectations:

Having a drive failure will become such an odd failure like having a raid controller, a motherboard or a CPU fail. You'd just replace it and restore the entire thing from backup.

I think drives already fail less than RAID controllers. From working in giant environmnts, the thing that fails more than mobos or CPUs is RAM. That's the worst one as it does the most damage and is hard to mitigate.

The difference though is that mobo, controllers, PSUs, are stateless to the system but drives are stateful. So their failure has a different type of impact, regardless of frequency.

Well, the stateful-ness of the drives is not something we can count fully on, hence the saying "raid is not backup".

What I'm proposing is that when it becomes very unlikely that a drive fails we could rethink our strategy and go for single drives instead of raid arrays. In the very unlikely event that a failure did occur, we are restoring from backup, which we are prepared to do anyway.

With HDDs the failure rate is too high but with enterprise SSDs it's starting to get into the "will not fail" category.

As an example assume we have 4 servers with a RAID10 array of 4 x 2TB drives each. Annual failure rate of HDDs are a few percent, say 3% for arguments sake. With 16 drives in total, every year there is about 50% chance that a drive will fail. So over the lifespan of the servers it's very likely that we will see one or more drive failures.

Now assume the same 4 servers with a single enterprise 4TB NVMe drive in each. Annual failure rate is 0.4% (actual number a few years back). With 4 drives in total, every year there is less than 2% chance that any drive will fail. So over the lifespan of the server it's very unlikely that we will ever see a drive failure at all. Sure, if it does happen anyway, we are restoring from backup instead of rebuilding the array.

As long as you can justify the downtime in the event that a single drive failure takes an entire server down (albeit with a low statistical chance).

If that isn't a concern no use running RAID anyway.

That makes sense. But regardless of RAID or not, there are always things that can take the entire server down, for instance a motherboard failure. So that is something that is always there.

I think you can take the probability x downtime to get the average downtime. And that times the cost per hour if you want to put it in $$$.

So if something is 2% likely to happen and causes 10 hours of downtime, you get 0.2 hours (12 minutes) of downtime on average. If that downtime is going to cost $10K per hour then it's $2K.

If that downtime is unacceptable you need to have more servers or more reliable servers. 12 minutes of downtime per year is 99.997% availability. 10 hours of downtime per year is 99.8%.

I think he should go after the wifi/internet connection. So restrictions are made in the router and not on the devices. Use the mac address to put devices in different groups with different restrictions.

To keep the kids on track, allow internet only at certain times, block sites that are distracting during school hours, etc. That kind of stuff.

If they have a schedule and they know what restrictions are in place, they can plan their school work around it. For instance if youtube is only allowed one hour in the afternoon any projects that need youtube could be done then.

@beta said in Router/firewall recommendations for small branch office:

We have PA subscriptions for antivirus/IPS/URL filtering etc. and since we plan to have a VPN between the 2 sites, I'm not sure if it would make sense to get those subscriptions again if we bought a 220 instead of just routing all the traffic to HQ.

You'll put a lot more traffic over the HQ WAN by routing branch office traffic destined for the internet that way.
Ideally you'd just want traffic over the VPN that is destined for some resource on the HQ LAN. It will give you superior bandwidth utilization.

We have a customer who runs PA820s and they removed all their L3 routing in switches and routers and now route all their VLANs through the PA. You'll have more control over security that way. Doing the same at your branch office makes sense.

Since you have Palo Alto at HQ I would get the same brand for the branch office. Not because you absolutely have to, but because it's easier to manage and easier if you have a problem and need Palo Alto support to figure out the problem.

When it comes to URL filtering at the branch office there are other options, for instance Cloudflare Gateway.

Regarding VOIP I think it's better to just run the phones directly to the HQ PBX. 10 people is not enough to bother with a local PBX.

So in summary:

• A PA-220 at the branch office with whatever VLANs you need set up in it.
• Internet traffic goes to the internet.
• Traffic to HQ goes over the VPN link.
• IP phones connects directly to HQ over the VPN link.

I ran some test on a VM and created some groups and added some top-level directories for those groups.

Changed the group on each top-level directories and files below recursively with chown -R.

Set directories to permission 2770 and files to 0660 with chmod -R.

Changed smb.conf and added create mask=0660 and directory mask=2770.

Now new files and directories created on the share have the right permission and belongs to the right group automatically, simply depending on what group the top-level directory belongs to.

If you're not a member of a group, you will not even see the directories or files that belongs to that group.

I think this is a good interim solution without too much work. Then moving to onedrive or whatever can be done in the future on a department to department basis.

Only admin required for adding users is to add them to linux/samba and make sure they become members of the right groups.

It's also very simple to make a separate share out of the top-level folders if you wanted.

You need to add some drives as well but other than that you are set.

Put this baby in a 2U colo space and have complete remote control of every server over IPMI.

@manxam said in Building your own lab:

What node has access to the disks or is each node responsible for 3 drives?
Can you assign one node all the storage and the remaining 3 be just compute?

Each node is hardwired to the drives so in this particular case you have 3 x 3.5" bays for each node. The ones with 2.5" bays have 6x2.5" per node.

There are also SATA ports and USB ports inside for boot drives.