Containers on Bare Metal

scottalanmiller

LXD is what we use. Very fast, very mature, and good tools for it.

Emad R

Nice, do you try to do them with ceph storage or you simply go with the default zfs

scottalanmiller

@Emad-R said in Containers on Bare Metal:

@scottalanmiller

Nice, do you try to do them with ceph storage or you simply go with the default zfs

ZFS isn't a default on any system that I know. But definitely not CEPH, CEPH isn't very performant unless you do a lot of extra stuff (Starwind makes a CEPH acceleration product.) ZFS was only default for Solaris Zones, not LXD. Much of LXD doesn't have have ZFS as an option. We are normally on XFS.

Emad R

@scottalanmiller

https://lxd.readthedocs.io/en/latest/clustering/
https://lxd.readthedocs.io/en/latest/storage/

I think latest versions and especially with clustering recommends ZFS storage, which is nice cause now it is added easily as fuse fs

stacksofplates

@scottalanmiller said in Containers on Bare Metal:

LXD is what we use. Very fast, very mature, and good tools for it.

@Emad-R Yeah LXD has taken the OCI image idea and applied it to LXC. LXC was doing something kind of like that later on. When you did an lxc-create -t download it would look at a text file with links to tarballs to download. LXD has incorporated images from the beginning which has given them a lot of flexibility like updating and layering.

Emad R

@Emad-R

Very good read:

https://linuxhint.com/lxd-vs-docker/

scottalanmiller

@Emad-R said in Containers on Bare Metal:

@Emad-R

Very good read:

https://linuxhint.com/lxd-vs-docker/

That is a good way to break them down, I liked that.

StorageNinja

A few things...

Google and AWS don't bother running them on Baremetal. While some people do, they tend to be shops that like running lots of linux on bare-metal and for them, it's a OS/Platform choice rather than a Hypervisor vs. non-hypervisor choice. The majority of the containers in people's datacenters and in the cloud are in VMs.
VMware with the project pacific announcement at VMworld called out that they get better performance with their container runtime in a Virtual Machine, than bare metal Linux container hosts. (This makes sense, once you understand that the vSphere scheduler does a better job at packing with NUMA awareness than the Linux kernel. Kit explained this on my podcast last week if anyone cares to listen).
I run them on bare metal on my Pi4 cluster because I'm still waiting on drivers and EFI to be written for it so I can run a proper hypervisor on them.

Emad R

@StorageNinja

I would like to hear more about your pi4 cluster since the pi4 is fairly new, any links or hints or suggested products

StorageNinja

@Emad-R Eh, I got 6 of them with the maximum memory (4GB). Also looking to acquire some beefier ARM platforms that I can run experimental ESXi builds on. - https://shop.solid-run.com/product/SRM8040S00D16GE008S00CH/ has caught my eye, but there are a few other ARM packages that are also reasonably priced and have different capabilities (Jetson etc from Nvidia for CUDA etc). Was really hoping rancher would sort out a ARM install but egh, might end up running that on my Intel NUCs.

scottalanmiller

@StorageNinja said in Containers on Bare Metal:

Also looking to acquire some beefier ARM platforms that I can run experimental ESXi builds on. - https://shop.solid-run.com/product/SRM8040S00D16GE008S00CH/ has caught my eye

Now this looks really sweet. That's some cool stuff... both the hardware and ESXi on ARM. $459 is a little high for that CPU and only 16GB, but not horrible.