Server Update

Almost 4 years after first building my virtualization server, it was finally time for a much needed upgrade.

The goal was to preemtively replace failing drives and increase the storage pool size, upgrade the Proxmox OS and all software to the latest version and add a GPU for Plex hardware transcoding.

Update Plan Link to heading

Pt 1: Storage Backup and Replacement Link to heading

Backup /dev/sdf, /dev/sda, /dev/sdb/, /dev/sdd to new 20TB & 18TB HDDs
Backup plex database, VM images & proxmox images
Comment out lines of drives to switch and mergerfs pool in /etc/fstab
Turn off VM autostart in proxmox settings and shutdown server
Open server and cleanup
Replace drives
Replace GPU
Turn on server
Run smart scan on all drives
Update /etc/fstab and NFS configs
Restart server
Test pool make sure everything is in its place
Turn on VMs and test apps
mergerfs balance
Restart server

Pt 2: Version Updates Link to heading

Update Proxmox version from 6.3-1 to 8.2
Update mergerfs version from 2.24.2 to 2.28.1

Pt 3: Setup GPU Link to heading

Replace GPU with NVIDIA GEFORCE 1070
Setup PCIE Passthrough on Proxmox to the Plex VM
Install NVIDIA drivers on the Plex VM
Enable and test hardware transcoding on Plex

Pt 1: Storage Backup and Replacement Link to heading

Before replacing the drives, I intially backed up the data on the drives I planned on replacing using this powered Sabrient HDD Enclosure following this process:

Put drive in Sabrient enclosure and turn it on.
Wait a bit for it to spin up, then connect it to server’s USB 3.0 port.
List drives to see its name: lsblk
Create partition and file system with fdisk:
```
fdisk /dev/sdX
```
- Delete partitions: d
- Create new partition table: g
- Create new partition with default settings: n
- Check the partition table before saving: p
- Write to disk (This is the point of no return): w
Create the ext4 file system replacing X with the drive letter:
```
mkfs.ext4 /dev/sdX1
```
Note the new drive Size, Serial Number and UUID in the table above, as well as the UUID of the drive it will be replacing.
Mount the drive:
```
mount -o rw /dev/sdX1 /mnt/ext
```
Copy files with rsync:
```
tmux
rsync -aAXv --progress /mnt/diskX/storage/ /mnt/ext/storage/
```
- tmux: Run in tmux to preserve session. If disconnected reconnect with tmux a.
- -a: Archive mode (preserves permissions, symbolic links, modification times, etc.).
- -A: Preserves ACLs (Access Control Lists).
- -X: Preserves extended attributes.
- -v: Verbose mode, shows the progress.
- --progress: Shows real-time progress of the copy.
Unmount drive and power off to safely eject drive:

umount /mnt/ext
udisksctl power-off -b /dev/sdX

If drive is busy, check to see what process is running with lsof +f -- /mnt/ext and kill each process with kill -9 <PID>. If that doesn’t work, restart server.

Shutdown drive by turning off HDD enclosure.
Disconnect HDD enclosure USB from server

Once all the drives were backed up I shutdown the server and replaced the drives.

Then I updated the proxmox /etc/fstab and restarted the system making sure all drives were showing up and the mergerfs pool was working.

Finally I ran mergerfs.balance to distribute the files across all the drives equally.

Pt 2: Version Updates Link to heading

Next I need to upgrade Proxmox two major versions from v6 to the latest v8.

This will also update all the packages to the latest version, including mergerfs.

First, I read the docs: Link to heading

Step 1: Prepare for the Upgrade

1.1. Back Up All VMs and Important Data Before starting the upgrade process, back up all VMs and important files:

Use vzdump to create backups of you VMs:

vzdump --all --compress lzo --storage <backup_storage>

Ensure that all VMs and critical data are backed up to an external storage location. Double-check that the backups are valid and restorable.

1.2. Back Up Configuration Files

Back up important configuration files:

tar czvf /root/proxmox_backup_etc.tar.gz /etc

Files to back up include:
- /etc/pve/
- /etc/network/interfaces
- /etc/resolv.conf
- Any custom configurations (like NFS or mergerFS settings).

1.3. Check for Disk Space and Update to Latest Proxmox VE 6.x

Ensure that you have at least 4GB of free disk space on the root partition:
```
df -h /
```
Update to the latest Proxmox VE 6.x before upgrading:
```
apt update
apt dist-upgrade
```
Reboot to apply the updated packages:
```
reboot
```

1.4. Run the pve6to7 Checklist

Run the pve6to7 script to check for potential issues:
```
pve6to7 --full
```
Follow the output and resolve any reported issues.

Step 2: Upgrade to Proxmox VE 7.x

2.1. Update Proxmox Repositories

Edit /etc/apt/sources.list to replace all buster entries with bullseye:
```
sed -i 's/buster/bullseye/g' /etc/apt/sources.list
```
Edit /etc/apt/sources.list.d/pve-enterprise.list or /etc/apt/sources.list.d/pve-no-subscription.list and update buster to bullseye for Proxmox repositories:
```
sed -i 's/buster/bullseye/g' /etc/apt/sources.list.d/pve-no-subscription.list
```

2.2. Update the Package Index and Upgrade to Debian Bullseye

Update package repositories:
```
apt update
```
Upgrade to Debian Bullseye and Proxmox VE 7:
```
apt dist-upgrade
```
This process may ask about replacing configuration files. Review changes carefully, and accept new versions where appropriate (especially for lvm.conf, ssh_config, etc.).

2.3. Reboot Into the Updated Kernel

After the upgrade finishes, reboot the system to use the updated Proxmox VE 7 kernel:
```
reboot
```

2.4. Check the Upgrade

After reboot, ensure that everything is working correctly by running:
```
pveversion
```
It should show version 7.x. Confirm that your VMs are running and that the storage pools and NFS mounts are correctly configured.

Step 3: Upgrade to Proxmox VE 8.x

3.1. Update Proxmox VE 7 Repositories

Update all Bullseye entries to Bookworm in /etc/apt/sources.list and /etc/apt/sources.list.d:

sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list
sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list.d/pve-install-repo.list

3.2. Update the Package Index and Upgrade to Debian Bookworm

Update package repositories:
```
apt update
```
Upgrade to Debian Bookworm and Proxmox VE 8:
```
apt dist-upgrade
```

3.3. Reboot Into the Updated Kernel

Reboot the system again to apply the new Proxmox VE 8 kernel:
```
reboot
```

3.4. Check the Upgrade

Verify the upgrade by running:
```
pveversion
```
Ensure it shows Proxmox VE 8.x.

Step 4: Post-Upgrade Tasks

4.1. Check VM Functionality

Verify that all VMs are running properly and that NFS mounts and mergerFS configurations are working correctly.
Check GPU passthrough for Plex to ensure the GTX 1070 is functioning as expected.

4.2. Re-check Configuration Files

Compare your backed-up configuration files (e.g., /etc/network/interfaces, /etc/pve) with the updated versions to ensure all settings are correctly applied.

4.3. Clear Browser Cache

Clear browser cache or force-reload the web UI to avoid issues with old UI files:
- Windows/Linux: Ctrl + Shift + R
- MacOS: ⌘ + Alt + R

3.4. Disable subscription popup:

Run these commands in the proxmox terminal:

echo "DPkg::Post-Invoke { \"dpkg -V proxmox-widget-toolkit | grep -q '/proxmoxlib\.js$'; if [ \$? -eq 1 ]; then { echo 'Removing subscription nag from UI...'; sed -i '/.*data\.status.*{/{s/\!//;s/active/NoMoreNagging/}' /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js; }; fi\"; };" >/etc/apt/apt.conf.d/no-nag-script
apt --reinstall install proxmox-widget-toolkit

Pt 3: Setup GPU Link to heading

Set Up PCIe Passthrough in Proxmox

Before using the GPU in the VM, you need to enable PCIe passthrough so the VM can directly access the GTX 1070.

a. Enable IOMMU in Proxmox:

Edit the GRUB configuration to enable IOMMU:
```
sudo nano /etc/default/grub
```
Find the line that starts with GRUB_CMDLINE_LINUX_DEFAULT and modify it to include the following:
```
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"
```
If you’re using an AMD CPU, replace intel_iommu=on with amd_iommu=on.
Update GRUB:
```
sudo update-grub
```
Reboot your server:
```
sudo reboot
```

b. Enable IOMMU and PCIe Passthrough in Proxmox:

Edit the kernel modules configuration:
```
sudo nano /etc/modules
```
Add the following lines to load the necessary modules for PCI passthrough:
```
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
```
Ensure your GPU is isolated by adding it to the blacklist:
```
sudo nano /etc/modprobe.d/blacklist.conf
```
Add the following lines to blacklist the GPU drivers on the host so the VM can use them:
```
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist rivafb
```
Bind the GPU to vfio-pci:

Find your GPU’s PCI ID:
```
lspci -nn | grep -i nvidia
```
Then, bind it by adding the PCI ID to /etc/modprobe.d/vfio.conf:
```
echo "options vfio-pci ids=XXXX:YYYY" | sudo tee /etc/modprobe.d/vfio.conf
```
Replace XXXX:YYYY with the PCI ID of your NVIDIA GPU.
Reboot the server again to apply all changes.

c. Add GPU to the VM:

In the Proxmox web interface, go to your Ubuntu Plex VM.
Edit the VM’s hardware settings, and add the PCI device corresponding to your GPU (this should appear once IOMMU is correctly set up).
Check the box for “All Functions” to ensure all GPU functions are passed to the VM.

Install NVIDIA Drivers on the Ubuntu Plex VM

Once the GPU is passed through, you need to install NVIDIA drivers in the Ubuntu Plex VM.

a. Use the ubuntu drivers tool to find the latest version of the NVIDIA driver for your system and install it:

sudo apt update
sudo apt install ubuntu-drivers-common
sudo ubuntu-drivers list --gpgpu
sudo ubuntu-drivers autoinstall
reboot

This will install the recommended drivers and CUDA support for your NVIDIA GPU.

b. Install NVIDIA utilities and nvidia-smi:

Install nvidia-smi to monitor the GPU usage:

sudo apt install nvidia-utils-470

You can check if the GPU is detected and functioning by running:

nvidia-smi

This should display information about your GTX 1070.

Enable Hardware Transcoding in Plex

To enable hardware transcoding, you need to configure Plex to use the GPU.

In the Plex web interface, navigate to Settings > Transcoder.
Enable the option “Use hardware acceleration when available”.
Enable “Use hardware-accelerated video encoding”.

Plex should now automatically use your NVIDIA GPU for transcoding tasks.

You can use nvidia-smi command to see utilization during a Plex hardware transcode.