@NetworkNerd

1. Identify the Cause of High Disk IO and CPU Wait MariaDB Activity: Since mariadb is showing high IO during the problematic window, it's crucial to identify the queries causing this load. You can enable the slow query log in MariaDB to capture queries that are taking an unusually long time to execute. Scheduled Tasks: Check for any scheduled tasks (cron jobs) on the server that run around 5 AM CST. These could be system tasks, WordPress cron jobs, or database maintenance tasks. 2. Systemd-journald Failure The failure of systemd-journal-flush.service suggests that the journaling system is overwhelmed, likely due to the high IO load. Investigate the journal logs (journalctl) for any errors or warnings that occur around this time. 3. Review WordPress Plugins and Activities Plugin Behavior: Even though plugins like Updraft Plus are scheduled for different times, they might be triggering background tasks. Verify plugin behavior and logs. WordPress Cron: WordPress has its own cron system (wp-cron.php) that can sometimes trigger resource-intensive tasks. Review the WordPress cron events. 4. Server and Database Optimization Database Optimization: Run a check and optimization task on your MariaDB database. Over time, databases can become inefficient and slow. Upgrade Resources: An e2-micro instance is quite limited in resources. If this issue is related to resource constraints, consider upgrading the VM instance type. 5. Monitoring and Logs Enable Enhanced Monitoring: Tools like sar, iotop, or atop can provide in-depth system metrics. Make sure they are configured correctly. Access and Error Logs: Review NGINX, PHP-FPM, and MariaDB logs for any anomalies during the problematic time frame. 6. External Factors Traffic Spikes: Although Jetpack stats show low traffic, consider checking the access logs for unexpected traffic spikes, which might be bots or crawlers. Network Analysis: Use tools to monitor network activity. Unexpected external connections might be contributing to the load. 7. Testing and Isolation Isolate Components: Temporarily disable certain components or plugins during the problem window to see if the issue persists. Test in a Staging Environment: If possible, replicate the setup in a staging environment to test without affecting the live site.

@scottalanmiller Figured it out!
dpkg --purge pve-kernel-helper
Fixed it!

Alternative to manually install 3rd party repositories

There is an alternative to manually manage repositories and keys and that is to use extrepo

extrepo is a curated list of 3rd party repositories and keys and it's a debian package.
It's only been around a couple of years so I don't know how widely used it is yet.

Installation

To install it run

apt install extrepo Add repository

To add postgreSQL repository for example:

extrepo enable postgresql Disable repository

To disable a repository, for example:

extrepo disable postgresql Where do files go?

extrepo puts apt config files in /etc/apt/sources.list.d as you would manually but manages keys in it's own directory /var/lib/extrepo/keys

Repositories available

Currently these repositories are in there:

anydesk apertium-nightly apertium-release bareos belgium_eid_continuous brave_beta brave_nightly brave_release caddyserver consol debian_official dns-oarc docker-ce edge elbe eturnal eyrie fai feistermops gitlab_ce gitlab_ee gitlab_runner google_chrome google_cloud grafana grafana_beta grafana_enterprise grafana_enterprise_beta haproxy-2.8 i2pd janitor jellyfin jenkins jitsi-stable kea keybase kicksecure kicksecure_developers kicksecure_proposed kicksecure_testers lihas liquorix matrix mobian msteams neurodebian_software newrelic nginx node_12.x node_14.x node_16.x node_18.x notesalexp ooni openmodelica-contrib-nightly openmodelica-contrib-release openmodelica-contrib-stable openmodelica-nightly openmodelica-release openmodelica-stable openstack_antelope openstack_zed openvpn opera_stable opsi passbolt postgresql prosody proxmox-ceph-quincy proxmox-pve proxmox-pve8 r-project raspberrypi raspbian-addons realsense rspamd signal skype slack speedtest-cli spotify steam surface-linux sury syncevolution syncthing teamviewer_default teamviewer_preview torproject trinity vector vscode vscodium weechat whonix whonix_developers whonix_proposed whonix_testers winehq wire-desktop wire-internal-desktop wtf wtf-lts x2go x2go-extras x2go-lts x2go-nightly xpra xpra-beta yarnpkg zammad zulu-openjdk

@scottalanmiller said in Installing Mastodon 4.1.2 on Debian 11:

@stacksofplates said in Installing Mastodon 4.1.2 on Debian 11:

Their official image is here ghcr.io/mastodon/mastodon

Yeah, that one was flaky. But is it official? Why's it on a different account?

Yes it's their official. Ghcr is the GitHub packages repo. It's similar to GCR, ECR, etc. Other OCI image repositories are more popular now since Docker has imposed limits on Docker Hub.

@Obsolesce said in Allow Binaries on Linux to Run on Well Known Privileged Ports:

@Pete-S said in Allow Binaries on Linux to Run on Well Known Privileged Ports:

If you search for net_bind you would assume it would find both these post but it finds nothing.

It seems to only search "words", and that isn't a word or part of a word.

That does seem to be the case. It's not smart at all.

@Pete-S said in ProxMox: Set VM to AutoStart from Command Line CLI Start:

Nice but it's even easier with pure kvm as you don't have to go the roundabout way of using VMid.

virsh autostart somevm

Personally I find the number quite a bit easier.

It's an easy fix. Sometimes the directions for the upgrade don't account for the source location of the APT REPO for ProxMox. Check your /etc/apt files and see where your repo is configured. If you are going from Buster to Bullseye for example, make sure that you have this line somewhere and the error should go away...

deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

Thanks, we are up and running again!

I don't know if ubuntu has a similar cloud kernel.

Update: It looks like there is a linux-kvm kernel amongst others. Haven't tried it though.

BTW, in Debian/Ubuntu and other distros the different kernels are often referred to as kernel flavors. Good to know if you want to search for them.

@JaredBusch said in KVM networking with libvirt (virsh) questions:

I assume that Debian 11 uses NetworkManager? I don't have a clean Debian system running KVM to check.

If so nmcli and its related commands are your friend.

Thanks Jared.

Reading about nmcli and seeing your screenshots led me to understand that macvtap devices are only active when the VM is actually running.

Using ip link I can now see the macvtap device on the host. One for each VM connected.

7ad85ba6-1b7b-40a2-aa7a-d8e12988683f-image.png

@jaredbusch said in beyond bash shell scripting, what language should I use:

@scottalanmiller said in beyond bash shell scripting, what language should I use:

@jaredbusch said in beyond bash shell scripting, what language should I use:

@stacksofplates said in beyond bash shell scripting, what language should I use:

@jaredbusch said in beyond bash shell scripting, what language should I use:

@scottalanmiller said in beyond bash shell scripting, what language should I use:

Go is great as a language. But like Ruby, not installed generally. And fewer resources. If it was a greenfield new OS, yeah, Go for sure. But for practical reasons, Python I think.

As these are systems that I control, there is no reason Go cannot be installed.

Between your comments and prior ones from @stacksofplates I think I might try Go in order to learn it.

You normally wouldn't install it anyway as it's not a scripting language. You'd just compile your binary and ship that to your systems.

I completely misunderstood that about Go. Okay, I will do a bit of checking and decide what I want to do.

oh sorry, I figured you knew. That's why I never look at it, I don't want to deal with binaries in that way. But nothing wrong with that. Write it on your machine at home, compile, ship binary. Works just fine.

As fixed tasks, this is not a bad solution. So I will keep it in mind.

Right, no big deal in this case.

@stuartjordan I seem to recall this being one of the reasons I got away from KDE a while back. Some quick google work shows that it might be solved by using kio-fuse but YMMV.

minfree Defines the amount of free space left in the LVM volume group pve. With more than 128GB storage available the default is 16GB, else hdsize/8 will be used.

This is space to leave behind for snapshots. They suggest 1/8th the size of your hard drive.

@scottalanmiller said in New to Linux Administration: RHEL-Based or Debian-Based OS:

I agree, both is the obvious choice. But to truly answer the question, I'd focus on Ubuntu (not Debian) and then RHEL. Debian is great, but it is Ubuntu specifically that has the market.

Yes, almost the same is never the same as exactly the same.

@Dashrender

Gotta love Bob-

Just install one of the old supported OSes on another drive, reboot and run the firmware update. It's a self extracting executable.

With HP and Dell, it's their intention to support you by providing an additional layer of management on top of the actual manufacturers product. But you have to be in warranty to be able to leverage that support. If you intend to run equipment longer than the 5 year warranty, don't use Dell or HP or live without any firmware upgrades or support. If you use Dell or HP, replace hardware when it's out of warranty.

HP Gen8 servers are from 2012-2013 so 7-8 years old and not the best candidates for new installations.