I need a few more machines in my lab (who doesn’t?), and normally I’d just create more virtual machines on one of my big servers. But those servers are at capacity, and the prices on new servers (and especially RAM) is really high.

Also, those large servers tend to use a lot of energy and have heavy cooling requirements. This means lots of heat and noisy fans. I want to minimize the heat and noise I add. The room I have dedicated to servers is generally 5-10 degrees farenheit warmer than the rest of my house and I can hear the fan noise from adjacent rooms.

I’m wondering if, instead of a big server, having a fleet of very inexpensive small machines could work better?

I’ve experimented with running silent, low heat systems long ago when VIAs C3 and Intel Atom in Micro-ITX formats were en vogue. At the time, those systems disappointed me in terms of performance. Now, almost 2 decades later, I figure it’s worth another try.

In 2026, it seems like the way forward for inexpensive low power, low heat, low noise systems might be ARM processors. There’s a lot of variety out there, but Raspberry Pis seem to be the an immediate starting point since they have a well established ecosystem and a lot of community support.

At between USD$70-90 (Feb 2026 prices) for a Raspberry Pi 4 w/ 8GB of RAM, it may work out financially to be cheaper than buying a large server, but it also has to work and fulfill my requirements!

Companion Videos:

To go with these blog post, I’ve created some videos. A time of writing the video series is still in development, but will be updated here as they are released.

• Racking Raspberries Episode 1: Testing Out Uctronics Pi Rack Pro

Personal Requirements

I have several end-requirements:

1. Must be able to mount in a standard 19” rack.
2. Must be able to remotely manage the power state
3. Must be able to remotely manage the console
4. Must run Red Hat Enterprise Linux (and ideally CoreOS, too)
5. Must be able to provision over the network and ideally through Red Hat Satellite

The last two requirements are probably going to be the hardest to fulfill. Raspberry Pis are not actually supported by Red Hat, but there do appear to be work-a-rounds posted on the internet.

Hardware

For the first requirement, there are several Raspberry Pi rack mounting solutions available: from simple brackets to full, modular enclosures. I decided on purchasing a more complicated solution that could support Power-over-Ethernet (PoE) hats and 2.5” SATA drives: the UCTonics Pi Rack Pro.

This chassis allows for 4 Raspberry Pis per rack unit. Some of the more basic brackets allow for much higher density, but at the expense of supporting add-ons like the PoE hat. I think 4 Pis per Rack Unit is a good compromise for me.

The chassis is going for a “blade module” vibe, but it really falls short on that. Really, it’s just 4 simplistic trays with some add-on boards. The build quality is OK, but there are too many screws and there are some cable routing issues, notably around the HDMI cable. I wish that, like the SD Card, they had contrived some way to bring the HDMI port to the front of the unit. The LED panel is a neat addition, but the power button is kind of useless. Unfortunately, the LED screen and power button add-ons are not well supported in any Linux distributions and drivers are only as source code.

That being said, I’m reasonably happy with this enclosure, but I’m not sure the almost $300 price point is justified.

Still, this checks off requirement #1: Must be able to mount in a standard 19” rack.

Using standard Raspberry Pi PoE Hats, I’m able to power these machines via PoE. In addition, with the right switching hardware, I can remotely control the power to each port to provide remote power management.

This checks off requirement #2: Must be able to remotely manage the power state.

For being able to remotely view the console, there are various options including projects like PiKVM and products like NanoKVM. These make what was once a pricey endeavor, adding out-of-bounds-management to a computer, relatively inexpensive.

Despite that, I still don’t want to add a PiKVM or NanoKVM per Raspberry Pi. While these solutions are cheap individually, the cost still adds up when you have to buy them in bulk!

So, I decided to buy a TESmart 16-port Rack Mount KVM. This does not allow remote viewing over network, it is simply switching 16 different KVM inputs to a single output. While it doesn’t capture video output, it can be controlled remotely, either via network or RS-232. The thought here is to connect a SINGLE PiKVM, NanoKVM or similar to the KVM switch.

I can then hook up to 16 Raspberry Pis (4 UCTronics Chassis) to the KVM Switch and through a combination of the remote control on the KVM and PiKVM or NanoKVM, I can interact with all 16 Raspberry Pis consoles over the network, albeit not at the same time. I think this is a reasonable compromise.

Cost-wise, I think this makes sense. A NanoKVM is roughly USD$55, so sixteen of them would cost USD$880! That’s not including costs for cables and extra ports on an switch. By contrast, this KVM switch cost me about USD$300 and I’ve seen similar models for anywhere between USD$250-600.

For this to be truly seamless, I’ll likely have to modify the software of the PiKVM or NanoKVM also be able to control the KVM Switch. I’m not ready to take on that project yet, but there is a clear path forward. So, I consider this checking off my requirement #3 “Must be able to remotely manage the console”.

With the UCTronics chassis, PoE Hats, TESmart KVM Switch, and remote KVM solution (like PiKVM or NanoKVM), I think I have all my hardware requirements satisfied, and I can move on to the RHEL and Satellite.

Red Hat Enterprise Linux

Since I’m planning use these Raspberry Pis to replace x86_64 servers, I need to be able to run RHEL or RHEL adjacent Operating Systems like CentOS or Fedora.

This is a hurdle, because Raspberry Pis are not a supported hardware platform for RHEL!

That being said, it is possible to run RHEL on Raspberry Pi, but it requires some special handling. Most notably, RHEL for ARM requires that the hardware have UEFI firmware.

UEFI Background

UEFI originated with Intel Itanium (ia64). The processor architecture required new firmware and Intel/HP decided to create EFI (Extensible Firmware Interface) rather than leverage the Open Firmware standard used in many other workstation/server platforms. EFI was later ported to x86 and x86_64 as a replacement for the legacy IBM-compatible BIOS firmware and has become the new standard for that architecture in the past few years. Along the way EFI got renamed UEFI (Unified Extensible Firmware Interface) and got ported to ARM, but the uptake in the ARM ecosystem has been spotty at best. RHEL on ARM requires UEFI and all supported ARM machines include UEFI in non-volatile storage (ROM, Flash, et cetera).

Raspberry Pis do not include UEFI and have their own unique firmware. However, the community has ported an open-source UEFI implementations to the Raspberry Pi 4 and 5. The executable binaries can be stored on an SD card and then chain-loaded by the Raspberry Pi’s native firmware. Since the UEFI binaries are stored on volatile storage (i.e. an SD Card) rather than in a non-volatile ROM or Flash memory, I’m going to have to be careful when partitioning and installing an Operating System. A wrong move and I could overwrite or delete the partition holding the UEFI binaries, thus preventing the system from booting.

There are two current repositories for different versions of the Raspberry Pi:

• UEFI Images for Raspberry Pi 4
• UEFI Images for Raspberry Pi 5

NB at time of writing, Google does not bring the RPi 5 repo up in search…instead it links to an abandoned effort. The above link comes from a recent FreeBSD Wiki page. I’ve not tested the RPi 5 at all, YMMV.

Installing the firmware

Getting the UEFI firmware installed is a fairly straightforward exercise. I’m going to be using the SD card to hold the UEFI firmware and boot from it.

First, wipe any existing partition table and create a new GPT partition table on the SD card. GPT or Global Partition Table is a newer format that replaces the older DOS/MBR-style partition tables. A DOS/MBR-style partition table will work but seems to cause issues with the RHEL-installer.

Then, create a smallish partition of type EFI System. 100 MB should be sufficient.

## My device is mmcblk0, yours may be different.  Proceed with caution or you may have data loss!

# # Clear any existing partition table:
# wipefs -a /dev/mmcblk0
/dev/mmcblk0: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/mmcblk0: 8 bytes were erased at offset 0x76e47fe00 (gpt): 45 46 49 20 50 41 52 54
/dev/mmcblk0: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/mmcblk0: calling ioctl to re-read partition table: Success

# fdisk /dev/mmcblk0 

Welcome to fdisk (util-linux 2.41.3).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS (MBR) disklabel with disk identifier 0x4f27bc74.

###
### fdisk still defaults to DOS/MBR-style partition tables
### the 'g' command will force it into GPT mode.
###

Command (m for help): g
Created a new GPT disklabel (GUID: 714CC8E9-2EE4-4E1D-80B2-F088DE7F3D35).

Command (m for help): n
Partition number (1-128, default 1): 1
First sector (2048-62333918, default 2048): 2048
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-62333918, default 62332927): +100M

###
###  There may still be existing data and filesystem headers on partitions
###  they'll be erased later, but it's no harm to remove it now
###

Created a new partition 1 of type 'Linux filesystem' and of size 100 MiB.
Partition #1 contains a vfat signature.

Do you want to remove the signature? [Y]es/[N]o: y

The signature will be removed by a write command.

###
### Setting the partition type to EFI System (Type 1) is
### absolutely required.  Otherwise things won't boot!
###
Command (m for help): t
Selected partition 1
Partition type or alias (type L to list all): 01
Changed type of partition 'Linux filesystem' to 'EFI System'.

Command (m for help): p
Disk /dev/mmcblk0: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 714CC8E9-2EE4-4E1D-80B2-F088DE7F3D35

Device         Start    End Sectors  Size Type
/dev/mmcblk0p1  2048 206847  204800  100M EFI System

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Next, make a VFAT filesystem on that partition and extract the EFI archive at the root of that filesystem.

# mkfs.vfat /dev/mmcblk0p1 
mkfs.fat 4.2 (2021-01-31)
# mount /dev/mmcblk0p1 /mnt
# pushd /mnt
# unzip /path/to/RPi4_UEFI_Firmware_v1.50.zip 
Archive:  /path/to/RPi4_UEFI_Firmware_v1.50.zip
  inflating: RPI_EFI.fd              
  inflating: bcm2711-rpi-4-b.dtb     
  inflating: bcm2711-rpi-400.dtb     
  inflating: bcm2711-rpi-cm4.dtb     
  inflating: config.txt              
  inflating: fixup4.dat              
  inflating: start4.elf              
   creating: overlays/
  inflating: overlays/miniuart-bt.dtbo  
  inflating: overlays/upstream-pi4.dtbo  
  inflating: Readme.md               
   creating: firmware/
  inflating: firmware/LICENCE.txt    
   creating: firmware/brcm/
  inflating: firmware/brcm/brcmfmac43455-sdio.clm_blob  
  inflating: firmware/brcm/brcmfmac43455-sdio.bin  
  inflating: firmware/brcm/brcmfmac43455-sdio.txt  
  inflating: firmware/brcm/brcmfmac43455-sdio.Raspberry  
  inflating: firmware/Readme.txt     
# popd 
# umount /mnt
# sync

Now the SD card should be ready. The Pi should boot into the UEFI firmware:

Creating RHEL Installation Media

RHEL can be downloaded from Red Hat’s portal. Since I work for Red Hat, I have an employee subscription, but no cost RHEL subscriptions are available for selected use cases.

I always use the Boot ISO image. This ISO image doesn’t have packages and will need network connectivity to either Satellite, a repository server, or the Red Hat CDN to complete an install. The Full ISO images are now exceeding 10 GiB, and no longer fits on my USB flash drives.

In Fedora, the Fedora Media Writer can be used to write an ISO image to a USB drive.

There are so many tools out there, so use whatever is available/convenient. As a last resort, a simple dd command can be used:

# dd if=/path/to/rhel.iso of=/dev/myusbdrive

Just be sure to ensure the iso is the ARM64 (aarch64) version! I’ve accidentally wasted several hours of my life by not paying close attention and trying to use the x86_64 version.

Pre-requisite: Enable more than 3 GiB of RAM

In the RPi4 UEFI firmware, there is a default setting that restricts the usable amount of RAM to 3 GiB. This was to work around a bug present in some versions of the Linux kernel. This default seems, quite frankly, out of date since affected Linux kernel versions are several years old.

RHEL 9 has a relatively recent kernel, and does not suffer from this bug. So, we should disable this limitation and use the full amount of RAM.

I accidentally forgot to change this setting, and noticed a high incident of failures during installation because of ram disk corruption. Once I enabled the full amount of RAM, this issue disappeared completely.

To change this setting and enable the full amount of RAM:

1. On boot, press Esc to enter the UEFI settings menu:

1. Select Device Manager:

1. Select Rapsberry Pi Configuration:

1. Select Advanced Configuration:

1. Change Limit RAM to 3 GB to Disabled:

1. Press F10 to save:

1. Either hard reset the machine or navigate to the main menu and select Reset:

1. On subsequent boots, the UEFI setting menu should list 8GiB of RAM:

Booting/Installing RHEL

To boot from the USB thumb drive, access the EFI settings menu by pressing Esc.

Then navigate to the Boot Manager menu

And select the USB devices that has the installation media. In my case, it’s I used a SanDisk Cruzer USB drive, so it’s relatively easy to find.

After this RHEL boots and the installation process is basically the same as RHEL on x86_64!

ONE MAJOR EXCEPTION IS THAT YOU MUST PROTECT THE EFI SYSTEM PARTITION FROM BEING OVERWRITTEN!

If the RHEL installer reformats the EFI partition, or wipes the partition table completely, then the UEFI binaries that were installed will, of course, go away. This is the trouble with having the EFI binaries on an SD card as opposed in a ROM or on-board Flash memory.

Manual partitioning is required guarantee the EFI System partition is protected, but it was a bit tricky. There seem to be some bugs in Anaconda (the RHEL Installer) that cause it to freeze or incorrectly think the EFI partition is on LVM.

In the end, this is the partition table I settled on:

Again, it’s very important not to reformat the EFI System Partition!

Once the install is successful, Anaconda will generate a kickstart file that can be used to repeat the installation. This kickstart file can be used as a template for other machines, so that we can avoid using the installer GUI in the future.

After installation, the generated kickstart should be located in /root/anaconda-ks.cfg.

The kickstart is pretty standard, with the major customization being in the disk partitioning section:

# Generated using Blivet version 3.6.0
ignoredisk --only-use=mmcblk1,sda
# System bootloader configuration
bootloader --append="crashkernel=1G-4G:256M,4G-64G:320M,64G-:576M" --location=mbr --boot-drive=mmcblk1
# Partition clearing information
clearpart --none --initlabel
# Disk partitioning information
part /boot/efi --fstype="efi" --noformat --onpart=mmcblk1p1 --fsoptions="umask=0077,shortname=winnt"
part pv.2270 --fstype="lvmpv" --ondisk=sda --size=4095
part /boot --fstype="xfs" --ondisk=mmcblk1 --size=1024
part / --fstype="xfs" --ondisk=mmcblk1 --size=20480
volgroup rhel_rpi1 --pesize=4096 pv.2270
logvol swap --fstype="swap" --size=4092 --name=swap --vgname=rhel_rpi1

This is not the exact partition scheme I wanted, but it’s close enough, and is a good starting point for a more refined configuration.

Since I’m able to get a basic RHEL installation working, this checks off my requirement #4 “Must run Red Hat Enterprise Linux”!

Now, hopefully, after rebooting at the end of installation, RHEL comes up without any additional effort. This does seems to be the case with the latest version of the UEFI Firmware (1.50) and RHEL 9.7. Earlier versions of either RHEL or the UEFI firmware had issues with configuring a boot entry for RHEL. That configuration needed to be manually done after installation.

Configuring the EFI boot menu

This section is probably no longer needed, but may be useful in case the installer doesn’t create a boot entry, or if there is a desire to create a custom one for some reason.

First, enter the EFI settings menu by pressing Esc during the boot sequence:

Next, navigate to Boot Maintenance Manager:

Next, navigate to Boot Options:

Next, navigate to Add Boot Option:

Select the SD Card:

I still have my USB Drive plugged in at this point, so it shows up as a volume labeled ANACONDA. The SD Card has no volume label, and will have HD(1,GPT) somewhere in the middle if my instructions were followed.

At this point, we are now navigating the filesystem inside the EFI System volume.

Navigate to <EFI>:

Navigate to <redhat>:

Select shim.efi:

Add a description:

Select Commit Changes and Exit:

This newly created boot option will be set to the last possible boot option. This is likely not desired, but is easy to change using the Change Boot Order screen:

This screen can be a bit confusing. It shows the current boot order. The top-most item will be attempted first, and then if that fails, each subsequent item will be attempted.

To change press Enter, and then use the + and - keys to move items up and down until the desired order is displayed, and then hit Esc.

Now select Commit Changes and Exit to make save the boot order.

Now, on reboot, provided our Red Hat Enterprise Linux option is the default, the Raspberry Pi should boot directly into RHEL!

We can always manually select it by going into the Boot Manager:

and selecting Red Hat Enterprise Linux:

Provisioning over the Network with Satellite

So the installation procedure above works, but is a bit labor intensive and I’d rather not do it again, especially if I build this out to a full 16 Raspberry Pis!

At this junction, I can’t automate everything. Notably, installing the UEFI firmware and partitioning the SD Card will have to be done by hand.

However, I can install and reinstall RHEL over and over using a very similar process to how I provision x86_64 RHEL Machines.

The provisioning part of this setup is very likely not supported by Red Hat. However, other portions, like hosting ARM64 content in Satellite is. Since this is my home lab, I don’t care about support, but still caveat emptor.

This section assumes an already working Satellite provisioning setup for x86_64 servers!

Importing the RHEL Content into Satellite

The first step here, is getting our content into Satellite. This is relatively straightforward and the ARM architecture follows the standard for RHEL. We have a AppStream and BaseOS repositories, as well as Kickstart repositories.

Kickstart repositories are designed for provisioning and are tied to a specific RHEL release, e.g. 9.4, 10.1, et cetera.

The normal AppStream and BaseOS repositories are rolling releases, and are used post-provisioning for patching and installing new software.

These repositories need to be synced to the Satellite, added to a Content View, and then, optionally, that Content View promoted to a Lifecycle Environment.

All this setup, is really beyond the scope of this post, but below are the hammer commands to create a Content View. In my environment, I have one Lifecycle Environment, named Production, and the below commands will promote the new content view in to that Lifecycle Environment and create a corresponding Activation Key.

### Setting a shell variable for the organization id.  This will
### likely be one but will be dependent on your installation.

# ORG_ID=1

### Enable the various Repository sets for RHEL.  Not all of these are
### strictly needed.
###
### Note the Kickstart repositories need to be a specific version,
### 9.7, whereas most other repositories just use the rolling 9
### release.

# hammer repository-set enable --organization-id ${ORG_ID} --name "Red Hat Enterprise Linux 9 for ARM 64 - BaseOS (Kickstart)" --basearch aarch64 --releasever 9.7
# hammer repository-set enable --organization-id ${ORG_ID} --name "Red Hat Enterprise Linux 9 for ARM 64 - AppStream (Kickstart)" --basearch aarch64 --releasever 9.7
# hammer repository-set enable --organization-id ${ORG_ID} --name "Red Hat Enterprise Linux 9 for ARM 64 - AppStream (RPMs)" --basearch aarch64 --releasever 9
# hammer repository-set enable --organization-id ${ORG_ID} --name "Red Hat Enterprise Linux 9 for ARM 64 - BaseOS (RPMs)" --basearch aarch64 --releasever 9
# hammer repository-set enable --organization-id ${ORG_ID} --name "Red Hat Enterprise Linux 9 for ARM 64 - Supplementary (RPMs)" --basearch aarch64 --releasever 9
# hammer repository-set enable --organization-id ${ORG_ID} --name "Red Hat Satellite Client 6 for RHEL 9 aarch64 (RPMs)" --basearch aarch64


### Synchronize all this content down to the Satellite server.

# hammer product synchronize --organization-id ${ORG_ID} --name "Red Hat Enterprise Linux for ARM 64"

### Optionally add this product to the daily sync plan.  I think this
### sync plan is the default now for Satellite, but I'm not sure.

# hammer product set-sync-plan --organization-id ${ORG_ID} --name "Red Hat Enterprise Linux for ARM 64" --sync-plan "Daily Sync"


### Generate a list of repositories.  This command will generate a
### correct list ONLY if this is the first time adding any ARM64
### products/repositories to the Satellite.

# REPOS=$(hammer repository list --organization-id ${ORG_ID} | egrep 'aarch64' | awk '{print $1","}' | tr -d '\n' | sed -e 's/,$//')

### Create the Content View

# hammer content-view create --organization-id ${ORG_ID} --name "RHEL 9 ARM64" --repository-ids ${REPOS}

### Publish the Content View

# hammer content-view publish --organization-id ${ORG_ID} --name "RHEL 9 ARM64"

### Promote the Content View

# hammer content-view version promote  --content-view "RHEL 9 ARM64" --organization-id ${ORG_ID} --to-lifecycle-environment "Production"

### Create Activation Key.  Since I'm using Simple Content Access
### mode, I don't have to attach a subscription.

# hammer activation-key create --name RHEL9_ARM64 --organization-id ${ORG_ID} --content-view "RHEL 9 ARM64" --lifecycle-environment "Production"

After all those hammer commands, I have all the content-related parts of the Satellite configuration done.

Satellite Provisioning Flow

In Satellite, host provisioning is generally “network-driven”. What that means is that the host always boots from the network and Satellite serves different network boot files to influence the behavior of the host.

So, when Satellite is provisioning the host, i.e. it’s listed in build status in Satellite, the network boot files will be generated such that those file contain configuration that instructs the host to run the installer and use a Satellite generated kickstart file.

Once provisioning is finished, then those network boot files are modified to instruct the host to boot off local media (e.g. the SD Card or attached USB or SATA drive).

By managing hosts this way, I can re-provision RHEL by just clicking a button in Satellite and rebooting the host.

There are several challenges to getting this to work.

First Challenge: DHCP Booting

The first challenge is getting the DHCP server to send ARM-specific boot files to ARM servers.

By default, the configuration for Satellite’s built-in ISC-DHCP server only includes logic for handling x86_64 and x86 machines.

When a DHCP client requests and receives a lease, there are a large number of DHCP options that can be passed to augment the base protocol. These options are defined by RFC2939 and the IANA maintains a list of DHCP Options. Option 93, Client System Architecture, is the relevant option for booting. Option 93 is defined by RFC4578 and, again, the IANA maintains a current list of Processor Architecture Types. Interestingly, this list is for DHCPv6 and I could not find one for IPv4 DHCP, but this list appears to be accurate for both variants of DHCP.

The IANA Processor Architecture Types list defines 0x06 for x86/i386, 0x07 for x86_64/amd64, 0x09 for EBC (EFI byte code, a generic EFI type), and 0x0b for ARM64. These values are for traditional tftp-based netbooting, which I will be using. There are a separate set of codes for http-based netbooting.

In the default /etc/dhcp/dhcpd.conf file, there is an if/else block that controls this:

option architecture code 93 = unsigned integer 16 ;
if option architecture = 00:06 {
  filename "grub2/shim.efi";
} elsif option architecture = 00:07 {
  filename "grub2/shim.efi";
} elsif option architecture = 00:09 {
  filename "grub2/shim.efi";
} else {
  filename "pxelinux.0";
}

These filenames are based on a relative path to the TFTP root directory. In the case of RHEL/Satellite, the default is /var/lib/tftpboot

Based on this, clients with architecture code 0x0b fall through to the last else block and will be sent /var/lib/tftpboot/pxelinux.0, which is wrong. pxelinux.0 is an x86 executable for traditional BIOS-style PXE boot.

I’ll need to add a stanza for the ARM64 architecture. But before we can do that, we need to get the appropriate files and place them in /var/lib/tftpboot.

Second Challenge: Getting Grub2 and UEFI Shim for ARM64

For UEFI booting on i386 and x86_64, there are two main boot executables that are used:

1. shim.efi
2. grub$(arch).efi

shim.efi is a signed boot executable for use with computers requiring SecureBoot. These kinds of machines require a cryptographically signed executable the first boot executable.

grub$(arch).efi is a copy of the Grand Unified Boot Loader. GRUB is responsible for loading the Linux kernel and actually starting the Linux boot process.

shim.efi is loaded first, and then it can then load subsequent executables. It immediately tries find and load a corresponding GRUB executable, attempting a list of potential file names including grub.efi, grubx64.efi and others.

The UEFI firmware on Raspberry Pi’s do not require SecureBoot, so a equivalent shim.efi is not needed, but I’ll use it anyway just to keep things consistent. Since I’m hosting both x86_64 and ARM64 on the same Satellite server, the architecture code (aa64 for ARM, x64 for x86_64) needs to be included in the filename. Thus, the files for ARM will be named shimaa64.efi and grubaa64.efi.

Where does one obtain these files? They are contained in 3 packages, and are also installed by default in any running ARM installation. In the list below, grub2-efi-aa64-modules is not needed for initial boot and install, but will become important later.

1. shim-aa64
2. grub2-efi-aa64
3. grub2-efi-aa64-modules

However, if I try to install these on my Satellite Server:

# dnf install -y grub2-efi-aa64
Updating Subscription Management repositories.
Last metadata expiration check: 2:01:45 ago on Mon 08 Dec 2025 02:31:14 PM EST.
No match for argument: grub2-efi-aa64
Error: Unable to find a match: grub2-efi-aa64

These packages (except grub2-efi-aa64-modules) are not available in the x86_64 RHEL repositories.

So, that leaves a few methods to get them:

1. Download the RPMs from access.redhat.com and extract what we need
2. Get the RPMs from an installation ISO
3. Get the files from a running system.

Since I had already done a test install of RHEL from a USB Stick, I chose option #3. I’ll leave the two others as an exercise to the reader.

My test host has a hostname of rpi1.private.opequon.net. First, I ensure that all three of the above packages are installed and then use the following commands will synchronize all the appropriate files:

# # ON SATELLITE SERVER
# cd /var/lib/tftpboot/

# # Copy files from grub2-efi-aa64-modules
# rsync -avz --delete root@rpi1.private.opequon.net:/usr/lib/grub/arm64-efi .
# chown -R foreman-proxy:root arm64-efi

# # Copy files from grub2-efi-aa64 and shim-aa64
# scp root@rpi1.private.opequon.net:/boot/efi/EFI/redhat/*aa64*.efi .
root@rpi1.private.opequon.net's password: 
grubaa64.efi                                                                  100% 2622KB  20.0MB/s   00:00    
mmaa64.efi                                                                    100%  873KB   9.4MB/s   00:00    
shimaa64-redhat.efi                                                           100%  956KB  10.0MB/s   00:00    
shimaa64.efi                                                                  100%  956KB   9.9MB/s   00:00    
# chown foreman-proxy:root *aa64*.efi
# chmod 0644 *aa64*.efi
# ls -la *aa64*.efi
-rw-r--r--. 1 foreman-proxy root 2684536 Dec  8 16:43 grubaa64.efi
-rw-r--r--. 1 foreman-proxy root  893760 Dec  8 16:43 mmaa64.efi
-rw-r--r--. 1 foreman-proxy root  978528 Dec  8 16:43 shimaa64.efi
-rw-r--r--. 1 foreman-proxy root  978528 Dec  8 16:43 shimaa64-redhat.efi

This copies a few unnecessary files like mmaa64.efi, which is a memory tester, and shimaa64-redhat.efi which has the same contents as shimaa64.efi, but it’s not a big deal.

Now with these in place, we can update our /etc/dhcp/dhcpd.conf to include a reference to these executables:

option architecture code 93 = unsigned integer 16 ;
if option architecture = 00:06 {
  filename "grub2/shim.efi";
} elsif option architecture = 00:07 {
  filename "grub2/shim.efi";
} elsif option architecture = 00:09 {
  filename "grub2/shim.efi";
} elsif option architecture = 00:0b {
  filename "grub2/shimaa64.efi";
} else {
  filename "pxelinux.0";
}

Unfortunately, this configuration is NOT permanent! dhcpd.conf is under control of Satellite’s puppet modules and will get rewritten next time that satellite-maintain is run. However, fixing that is a tomorrow problem.

Third Challenge: Loading GRUB Modules

With the DHCP changes and efi files in place, I can run an installation. However, once the installation is complete, the system doesn’t come up on reboot! There are errors about chainloading.

Now this doesn’t mean that the installation wasn’t successful! If I were to navigate through the UEFI console and boot from the Red Hat Enterprise Linux Boot Menu entry, then the machine comes up successfully.

So, what’s going on here?

Recall that Satellite is controlling the boot process through network booting configuration files. At the end of the install process, there is a call-back to the Satellite server that indicates the installation has completed. This causes the Satellite server to change the net boot files to instruct the host to boot from local disk.

The DHCP server tells the host to load shimaa64.efi, which then loads grubaa64.efi. GRUB looks for configuration and, by default, it will look in the same place that the grubaa64.efi came from: the TFTP server.

On the TFTP server, we can find our GRUB configuration under /var/lib/tftpboot/grub2/

# cd /var/lib/tftpboot/grub2
# ls -la

###
### I've cut out a lot of files here for clarity/brevity
### 

-rw-r--r--.  1 foreman-proxy root              320 Jan 20  2021 grub.cfg
-rw-r--r--.  1 foreman-proxy foreman-proxy    1771 Jan 20 23:30 grub.cfg-01-d8-3a-dd-f2-ee-50
-rw-r--r--.  1 foreman-proxy foreman-proxy    1771 Jan 20 23:30 grub.cfg-d8:3a:dd:f2:ee:50

GRUB will go through different file patterns to find the correct one. In Satellite, the Ethernet MAC address is used to uniquely tie a specific file to GRUB. In the above example, there are two different variations of filenames for a machine with MAC address d8:3a:dd:f2:ee:50. I’m not sure which one GRUB, but Satellite generates them both.

After GRUB runs through all it’s filename patterns, it will default to grub.cfg, but in a Satellite provisioning situation, that should never happen.

These files tend to be fairly complicated scripts, and the ones provided by Satellite are provided as templates that are heavily generalized to work with RHEL, Fedora and several other distributions.

There are, of course different templates depending on what action is expected from the host. In this case, since the installation was completed, Satellite is expecting the host to boot from local boot, so the PXEGrub2 default local boot template was used.

While I’m troubleshooting, I can directly modify these file in /var/lib/tftpboot.

A simplified version looks like this:

# Loading modules for GPT Partitions, FAT Filesystem and Chainloading
insmod part_gpt
insmod fat
insmod chain

# Menu Entry to display
menuentry 'Chainload Grub2 EFI from ESP' --id local_chain_hd0 {
  
  # Search for the file to chainload
  unset chroot
  search --file --no-floppy --set=chroot /EFI/fedora/shim.efi

  # If this file is found, chainload it
  if [ -f ($chroot)/EFI/fedora/shim.efi ]; then
    chainloader ($chroot)/EFI/fedora/shim.efi
    echo "Found /EFI/fedora/shim.efi at $chroot, attempting to chainboot it..."
    sleep 2
    boot
  fi

“Chainloading” in this case is using the copy of GRUB we pulled down from the network to load another copy of GRUB (or if it were a different operating system, then loading that operating system’s bootloader). This second copy of grub then is used to load the RHEL installation on the local storage.

Chainloading (and the chainloader command) is NOT part of the basic grub.efi executable, at least so far as ARM is concerned. Therefore, that functionality needs to be loaded in at runtime from a module. This is the intent of the insmod chain command.

For some reason, the insmod commands fails to find the module to load when running on ARM. I’m not sure exactly how it works on x86_64. Debugging here is difficult. The chainload module is either part of the base grub.efi for x86_64 or the insmod is somehow able to figure out how to load it. I’m not sure.

Regardless, I was able to figure out how to get insmod to work correctly and load the ARM version of modules from Satellite’s tftp server! This requires changing the insmod command to include some hints on where to find the actual module.

From:

insmod chain

change to:

insmod (tftp)grub2/arm64-efi/chain.mod

Similar changes can be made to the other insmod statements, but chain.mod was the one causing the issue.

These hints tell GRUB to load this module from the TFTP server from a specific directory. Now that it can successfully get this module, the boot can continue!

Fourth Challenge: Post-Install Template

Modifying GRUB files in the /var/lib/tftpboot directory works, but Satellite owns these files and will rewrite these for various reasons. I need to modify the Satellite template used to generate this file.

Satellite uses the template referenced by the localboot template parameter to generate the post installation version of this GRUB configuration. Unfortunately, this parameter is not exposed in the UI except for at a global level. Since this variant is ONLY applicable to ARM, I don’t want to override this template FOR EVERY MACHINE.

To solve this, I can use Satellite Host Groups functionality to create a Host Group just for Raspberry Pi hardware. The Host Group will have a parameter set to reference the updated templates for localboot.

Putting it all Together: Satellite Configuration

Working through those challenges, I can now have all the information I need to configure Satellite to provision a host.

1. Create a new templates for ARM including a new localboot template and a partition table
2. Create a hostgroup using the Content View, Partition Table, and Local Boot template
3. Ensure my Rapsberry Pis are set to boot from the network first
4. Create a host in that host group to provision
5. Reboot and watch it run!

Putting it all Together: Satellite Templates

To complete the configuration, I had to create two templates in Satellite.

1. Chainload Template

I cloned pxegrub2_chainload template into pxegrub2_chainload_RPI and modified it to include insmod changes.

<%#
kind: snippet
name: pxegrub2_chainload
model: ProvisioningTemplate
snippet: true
description: |
  In Foreman's typical PXE workflow, managed hosts are configured to always boot from network and inventory build flag dictates if they should boot into installer (build is on) or boot from local drive (build is off). This template is used to chainload from EFI ESP for systems which booted from network. It is not as straightforward as in BIOS and EFI boot file must be found on an ESP partition.

  This will only be needed when provisioned hosts are set to boot from network, typically EFI firmware implementations overrides boot order after new OS installation. This behavior can be set in EFI, or "efi_bootentry" host parameter can be set to "previous" to override boot order back to previous (network) setting. See efibootmgr_netboot snippet for more info.
-%>
<%
  paths = [
    '/EFI/redhat/shimaa64.efi',
    '/EFI/redhat/grubaa64.efi'
  ]
  config_paths = [
    '/EFI/fedora/grub.cfg',
    '/EFI/redhat/grub.cfg',
    '/EFI/centos/grub.cfg',
    '/EFI/rocky/grub.cfg',
    '/EFI/almalinux/grub.cfg',
    '/EFI/debian/grub.cfg',
    '/EFI/ubuntu/grub.cfg',
    '/EFI/sles/grub.cfg',
    '/EFI/opensuse/grub.cfg',
  ]
-%>

insmod (tftp)grub2/arm64-efi/part_gpt.mod
insmod (tftp)grub2/arm64-efi/fat.mod
insmod (tftp)grub2/arm64-efi/chain.mod


<%=
  default_connectefi_option = 'scsi'
  connectefi_option = @host ? host_param('grub2-connectefi', default_connectefi_option) : default_connectefi_option
  connectefi_option = nil if connectefi_option == 'false'
  "connectefi #{connectefi_option}" if connectefi_option
%>


menuentry 'Chainload Grub2 EFI from ESP' --id local_chain_hd0 {
  echo "Chainloading Grub2 EFI from ESP, enabled devices for booting:"
  ls
<%
  paths.each do |path|
-%>
  echo "Trying <%= path %> "
  unset chroot
  # add --efidisk-only when using Software RAID
  search --file --no-floppy --set=chroot <%= path %>
  if [ -f ($chroot)<%= path %> ]; then
    chainloader ($chroot)<%= path %>
    echo "Found <%= path %> at $chroot, attempting to chainboot it..."
    sleep 2
    boot
  fi
<%
  end
-%>
  echo "Partition with known EFI file not found, you may want to drop to grub shell"
  echo "and investigate available files updating 'pxegrub2_chainload' template and"
  echo "the list of known filepaths for probing. Available devices are:"
  echo
  ls
  echo
  echo "If you cannot see the HDD, make sure the drive is marked as bootable in EFI and"
  echo "not hidden. Boot order must be the following:"
  echo "1) NETWORK"
  echo "2) HDD"
  echo
  echo "The system will poweroff in 2 minutes or press ESC to poweroff immediately."
  sleep -i 120
  halt
}

1. local boot template

The pxegrub2_chainload_RPI template above is a snippet. The original one was used by the PXEGrub2 default local boot template.

I cloned the PXEGrub2 default local boot template into PXEGrub2 default local boot clone RPI and modified to use my newly created pxegrub2_chainload_RPI template.

<%#
kind: PXEGrub2
name: PXEGrub2 default local boot
model: ProvisioningTemplate
description: |
  The template to render Grub2 bootloader configuration for provisioned hosts,
  that still boot from the network.
  Hosts are instructed to boot from the first local medium.
  Do not associate or change the name.
-%>
set default=<%= global_setting("default_pxe_item_local", "local") %>
set timeout=20
echo Default PXE local template entry is set to '<%= global_setting("default_pxe_item_local", "local") %>'

<%= snippet "pxegrub2_chainload_RPI" %>

1. Partition table

Using the kickstart file generated when I installed with a USB drive, I created a partition table template named RPi SD-Card and SATA Drive Default.

In order for the Partition Table to show up in Create Host or Create Host Group screens, it must be associated with an individual Operating System version.

Since I added the RHEL 9.7 Kickstart, I need to add this template specifically to the Red Hat 9.7/RHEL 9.7 Operating System.

This is found under the Hosts->Provisioning Setup->Operating System menu item:

In Partition Table tab for the specific RHEL Version (in our case RedHat 9.7):

Make sure the partition table template is in the selected items column!

The contents of the partition table template are:

# Default RPi to use only attached SD Card

# Generated using Blivet version 3.6.0
ignoredisk --only-use=mmcblk1,sda
# System bootloader configuration
bootloader --append="crashkernel=1G-4G:256M,4G-64G:320M,64G-:576M" --location=mbr --boot-drive=mmcblk1
# Partition clearing information
zerombr
clearpart --linux 
# Disk partitioning information
part /boot/efi --fstype="efi" --noformat --onpart=mmcblk1p1 --fsoptions="umask=0077,shortname=winnt"
part /boot --fstype="xfs" --ondisk=mmcblk1 --size=1024
part / --fstype="xfs" --ondisk=mmcblk1 --size=20480 --grow

part pv.1 --fstype="lvmpv" --ondisk=sda --grow
volgroup rhel_rpi --pesize=4096 pv.1
logvol swap --fstype="swap" --size=4092 --name=swap --vgname=rhel_rpi

I decided on putting /boot and / on the SD Card as normal partitions, rather than Logical Volumes. Then on the SATA drive, I put that drive’s entire contents into a volume group called rhel_rpi. In the above table, I only put a single logical volume, one for swap. However, in future, I’ll likely add one for /var and other locations.

I’m not sure if this is the most optimal or not, but it does seem to cause the least issues on provisioning/re-provisioning.

Normally, I would just have a kickstart that would wipe ALL the drives and go from there. This is not possible here, because the EFI system partition must be preserved.

I had three scenarios I needed to cover:

1. Only the EFI partition on the SD Card and no partition table on the SATA drive. This is the “brand new” state.

2. Multiple partitions on the SD Card, and no partition table on the SATA drive. This is unlikely to happen realistically, but could, if I replaced the SATA drive.

3. Multiple partitions on the SD Card, and partitions on the SATA drive. This is the normal re-provisioning states.

In the kickstart directive, the clearpart --linux was very useful. It clears ONLY Linux related partitions, so it leaves the EFI System partition unmodified and intact, but will delete any other partitions.

This solves scenario #3 above, which is likely the most common, but partitioning would still fail on #1 and #2 because there wasn’t a partition table on the SATA drive.

I tried multiple arguments to clearpart to solve #1 and #2, but ended up realizing that zerombr was what I actually needed. zerombr initializes a partition table only on disk that need it. So it effectively ignores the SD Card, since it will always have a partition table, but will, if needed, put a partition table on the SATA drive.

Putting it all together: Host Group

With those templates and provisioning templates, I can now create the Host Group:

In the host group tab, I set the Lifecycle Environment, Content View, and Deploy On. In Content Source, yavanna.private.opequon.net is my Satellite host.

In the Operating System tab, I select the RHEL Release that corresponds with our Kickstart repository, update the Partition Table to the template I created, and then set the PXE loader to Grub2 UEFI.

In the parameters tab, we add kt_activation_keys to the Activation Key that was created and local_boot_PXEGrub2 to the template we created for local boot.

Putting it all togther: Changing Boot Order

For this to work, a host must have UEFI PXEv4 set as the default boot option. This is the default unless one has installed via a USB drive or manually modified the boot order.

Putting it all together: Creating Hosts

Now that I’ve got everything in place, I can create hosts.

Navigate to Create Host in the Satellite Menu:

Fill out a hostname and select the Host Group for the Raspberry Pis:

The Host Group contains almost all the basic parameters are needed to provision this host. So, there’s nothing much else to enter:

The Network Interface still needs to be defined:

Like so:

With the Host defined in Satellite, I can boot the Raspberry Pi and it should start to PXE boot:

And then continue onto GRUB:

And after some time, it should continue to installing:

Summary

After doing all this configuration in Satellite, I now have a very similar process to provisioning these Raspberry Pis as I do my x86_64 machines.

This technically checks off my requirement #5 “Must be able to provision over the network and ideally through Red Hat Satellite”.

However, I fear this setup is a bit fragile. I know that the DHCPd configuration will be overwritten next time I upgrade Satellite. I’ve started to look into patching Satellite, but it may be easier to run a separate DHCP server that is not entirely managed by Satellite.

In the same vein, I’m not sure if the UEFI binaries put in /var/lib/tftpboot will stay in place or not. Upgrading Satellite may wipe them out, but I’m not 100% sure yet. Regardless, those were setup by hand and not by any supported way. So they wont be updated and could potentially become out of date.

That being said, I’m pleased with how this came together, especially for something unsupported.

Cost Calculations

Prices are going to fluctuate, so this section will likely be out of date very quickly. This analysis is current as of Jan 2026.

Each Raspberry Pi was configured identically:

The UCTronics Chassis cost $269.99, I divided by 4 to get the per Pi price of $21.87.

The TESMart KVM switch cost $349.99. I divided that by 16 to get a per Pi price of $6.25.

I haven’t bought a NanoKVM or equivalent for this system yet, but I put in a price of $100 or $6.25 per Pi.

Maxing this configuration out with 16 total Pis yields an estimated cost of $3873.12.

This is not including the price of a PoE-compatible switch!

Yikes!

If one is comparing with NEW x86_64 servers or even refurbished recent models (e.g. a HPE Gen 10+), then this is cheaper. Gathering quotes on Server Monkey, a refurbished HPE DL360 Gen 10 Plus chassis (without processors, RAM, or disk) starts at $3,100.

However, the if I go back to slightly older models, a HPE Gen 10 DL360 w/ a 16 Core Processor, 128 GiB RAM, and 4 TB SATA SSD can be priced out for upper $2,000s to low $3,000s. I feel pretty confidant that a single server configured like that could run at least 16 virtual machines if not more and be the similar, if not more performant.

On the other hand, 16 independant hosts have a lower blast-radius for failure compared to a single machine. So that price for the HPE server is maybe low. I’d want more than 1 drive for redundancy, since a single drive failure would wipe out ALL my virtual machines. And, I’d probably want more than 128 GiB of RAM to account for operating system overhead.

Server Monkey is also not the cheapest vendor. I would likely be able to find a better deal on eBay or somewhere else, eventually. This is how I’ve acquired most of my server hardware anyhow, but it requires patience and diligence.

That being said, there’s also some flexibility in the price of the Pi setup. I think I overpaid for HDMI cables and SD Cards….and I certainly feel the UCTronics chassis is over priced.

I also may be able to get more value, though not necessarily a lower price, through buying more capable single board computers, like Raspberry Pi 5s or Orange Pis. There’s also potential experiments I could do with even more unusual architectures like RISC V.

I’ll leave it up to the reader to determine if this is valuable or not, but it’s certainly no slam dunk.

Problems with RHEL

I encountered during this two bugs in RHEL itself that hampered progress.

The first occurred between RHEL 9.2 and 9.4, where a bug caused the frame buffer to fail. So the host would boot successfully, but I’d have no video from the console.

This bug was fixed somewhere between 9.4 and 9.6.

The second bug involved the UEFI shim and other EFI modules. The ones from 9.7 work great. However, after 9.7 they were updated and booting fails when you have more than 3 GiB of RAM enabled.

I can copy the UEFI boot programs from 9.7 and they work as expected. I’m sure I could track down why in the RPM, but I haven’t yet.

In the interim, I’ll likely have to black-list or version restrict the UEFI packages. Until I figure out which specific package, I can set the package_upgrade parameter to false on the Host or the Host Group in Satellite, which will prevent a dnf update -y from running during the kickstart install. Hopefully this is fixed when 9.8 is released.

Conclusions and next steps

I am fairly happy with how this turned out…all my requirements were fulfilled. However, I’m not sure of the value here. Whether a setup like this is more cost effective than used x86_64 hardware is not clear, but I’m leaning towards no.

That being said, initial cost isn’t the only factor. I do believe this setup is less noisy just from my own perception, though real measurements are necessary. I’ve also yet to do power consumption tests, which may prove that TCO over a long period is better.

There are also some serious problems with this setup:

First, this isn’t really supported by Red Hat. Both the Satellite procedure as well as just running RHEL on Raspberry Pis. I don’t care too much as this is just lab hardware, but it is annoying that anything could break with any update.

Second, while modifications to files like /etc/dhcp/dhcpd.conf are functional, these files are controlled by Puppet modules. Thus, the changes will be overwritten when doing patching, upgrading, or reconfiguration via satellite-maintain.

This is solvable, but may lead me to running a DHCP server separate from Satellite.

Third, I’ve not been able to get RHEL 10 working…yet. While still current for quite some time, RHEL 9 is no longer the latest release. I like to stay current if I can.

As for next steps, I’d like to do a few different things:

1. Perform some power usage comparisions between this total setup and some of my x86_64 servers.

2. Integrate a NanoKVM or similar with the KVM switch. This will require some custom development work, but would be nice to be able to switch between Raspberry Pis in the same interface.

3. Integrate Power Controls into Satellite. I can control things via my PoE Switch, but it would be nice to take advantage of the Satellite UI for that.

4. Test this out with other RHEL variants, like CoreOS for OpenShift Nodes. I don’t think these would be powerful enough to run as Control/Master nodes of K8s cluster…I suspect that the disk latency is not good enough to run etcd, but it would be interesting to see these Raspberry Pis be made part of a multi-architecture OpenShift cluster.

5. Test out Raspberry Pi 5’s and potentially other single-board computers.

We’ll see if I have time for any of it!

I needed my pods to create files with the group write bit set. Essentially, I needed a umask of 0002, but my pods were running with umask of 0022.

I was afraid I was going to have start twiddling around with umasks for my Tekton tasks, but instead found that I could solve my issue using Linux Access Control Lists (ACLs).

ACLs are not often used on Linux, but can be quite powerful. For this article, the relevant feature is the ability to set default permissions for created files. These ignore the current processes umask and act as a directory specific umask.

The Setup

My PersistentVolumes were NFSv4 shares exported from a RHEL8 host, so I would need to set these ACLs on directories exported from that server.

Here is the normal behavior:

nfs-server#  cd /path/to/nfs_share

nfs-server# umask
0022      <-  Files should be created rwxr-x-r-x 

nfs-server# getfacl .  
# file: .
# owner: root
# group: root
# flags: -s-
user::rwx
group::rwx
other::r-x


nfs-server# touch without_default_perms_acl
# ls -l
total 0
-rw-r--r--.  1 root root   0 Jan 19 11:44 without_default_perms_acl

Now, let me set default permissions via an ACL:

nfs-server# cd /path/to/nfs_share

# Let's set our default ACLs for group and other.

nfs-server# setfacl -m d:g::rwX . 
nfs-server# setfacl -m d:o::r-X .

nfs-server# getfacl .
# file: .
# owner: root
# group: root
# flags: -s-
user::rwx
group::rwx
other::r-x
default:user::rwx
default:group::rwx
default:other::r-x

nfs-server# umask
0022      <-  Our umask remains unchanged

nfs-server# touch with_default_perms_acl
nfs-server# ls -l 
total 0
-rw-rw-r--. 1 root root 0 Jan 19 11:50 with_default_perms_acl
-rw-r--r--. 1 root root 0 Jan 19 11:44 without_default_perms_acl

As you can see, the file created AFTER setting the default ACLs ignores the umask and instead uses the default set in the ACL! New subdirectories will also inherit these ACLs.

When this directory is exported by NFSv4, this ACL is honored by the client.

For more information, see acl(5) and setfacl(1).

A word of warning, not all NFS servers will support this, and I’m fairly certain you must be using NFSv4 for this to work. I’ve tested it with a RHEL8 nfs-server, but YMMV for other nfs servers.

WHY would I need to do this?

OpenShift, unique among K8s distributions, semi-randomizes the UID and GID of it’s pods. This is a security measure that adds barriers for attackers.

When two pods need to share a PV, we need to carefully construct permissions so that they share a common group and that group has write permissions on all files.

Normally this is done by:

1. Changing the group of the root of the NFS share to 0 (root) or a defined common group

2. Setting the setgid-bit, so that all new files and directories inherit that group.

3. Setting the umask to 0002, so that group read-write is set on all files.

Step #1 and #2 above are fairly straight forward, but setting the umask on every pod in every scenario can be difficult, especially if you’re dealing with pre-created container images and many images (like in a tekton pipeline)

Using default ACLs, instead of relying on umask, allows me to make one change, on the server side, that is automatically followed by all pods.

I’ve been recently updating my fileserver to RHEL 8 and working with the original array I created in 2015 and documented in my post: RAID 6 and LVM. In the original version of that post, I did not specify a name.

I’ve found that naming my RAID volumes has a great benefit when reinstalling the system via Kickstart, which I’ll write about more in detail later.

So, my RAID volumes are unnamed….how do I give them a name?

First, let’s look at our RAID volume using the mdadm --detail command:

# mdadm --detail /dev/md200
/dev/md200:
           Version : 1.2
     Creation Time : Thu Sep 24 17:47:54 2015
        Raid Level : raid6
        Array Size : 3906521088 (3.64 TiB 4.00 TB)
     Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
      Raid Devices : 6
     Total Devices : 6
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Thu Jun  9 03:11:45 2022
             State : clean 
    Active Devices : 6
   Working Devices : 6
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : 200
              UUID : 6d90c19a:9812e659:6854747b:a89f1280
            Events : 34710

    Number   Major   Minor   RaidDevice State
       0       8       97        0      active sync   /dev/sdg1
       1       8       17        1      active sync   /dev/sdb1
       2       8       49        2      active sync   /dev/sdd1
       3       8      113        3      active sync   /dev/sdh1
       4       8       65        4      active sync   /dev/sde1
       5       8       81        5      active sync   /dev/sdf1

Above, there is a name field, but it’s not really a human name, just 200. This is really just the /dev/md### number.

Renaming is a relatively simple operation, but it can’t be done live.

In my setup, I have a LVM Volume Group called galadriel. I want to rename my RAID device to also be galadriel to match.

1. Run mdadm --detail and save the output somewhere for reference.

2. Unmount all volumes related to either the RAID device or volume group using the RAID.

3. Stop the volume group. -a n deactivates the volume group.

    vgchange -a n galadriel
   

4. Stop the md device using mdadm

    sudo mdadm --stop /dev/md200
   

5. Reassemble with new name. This reassembles the drive with the --name. The device names can be gleaned from the output of `mdadm –detail.

    sudo mdadm --assemble --update=name --name=galadriel /dev/md200 /dev/sdg1 /dev/sdb1 /dev/sdd1 /dev/sdh1 /dev/sde1 /dev/sdf1
   

6. Reactivate the volume group

    vgchange -a y galadriel
   

7. Check that everthing is OK, by looking at /dev/mdstat, comparing new output of mdadm --detail, vgs, and lvs.

    # cat /proc/mdstat
    md200 : active raid6 sdf1[1] sdh1[2] sdd1[3] sda1[4] sdc1[0] sdb1[5]
          3906521088 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU]
          bitmap: 0/8 pages [0KB], 65536KB chunk
   

8. It may be required to update /etc/mdadm.conf. In RHEL7+, the UUID is used, so only the dev path may need to be updated:

   # cat /etc/mdadm.conf
       
      ### Snippet of relevant line:
   ARRAY /dev/md/galadriel level=raid6 num-devices=6 UUID=6d90c19a:9812e659:6854747b:a89f1280
   

This works great for the first user, i.e. the one that is specified in the installer. However, other users are not able to log in after installation.

To make reading easier, I’ve put firstuser and seconduser as my two users. I’ve also clipped out hostnames and timestamps from journalctl output.

If attempting to log in as one of these other users via ssh, the following messages are logged in the journal:

sshd[928089]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=172.31.0.50 user=seconduser
sshd[928089]: pam_sss(sshd:account): Access denied for user seconduser: 6 (Permission denied)
sshd[928089]: Failed password for seconduser from 172.31.0.50 port 42840 ssh2
sshd[928089]: fatal: Access denied for user seconduser by PAM account configuration [preauth]

Failed password for seconduser makes this seem like a password issue, but it’s not. An actual bad password looks like this:

krb5_child[979500]: Preauthentication failed
krb5_child[979500]: Preauthentication failed
sshd[979494]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=172.31.0.50 user=seconduser
sshd[979494]: pam_sss(sshd:auth): received for user seconduser: 7 (Authentication failure)
sshd[979494]: Failed password for seconduser from 172.31.0.50 port 39754 ssh2

The real error, is actually coming from PAM and we can see the difference from my scenario vs the true wrong password. If we look at the first example, the real error is:

pam_sss(sshd:account): Access denied for user seconduser: 6 (Permission denied)

compare to the wrong password scenario, where the error is this:

pam_sss(sshd:auth): received for user seconduser: 7 (Authentication failure)

From this, we can see that a true wrong password fails in the auth modules of PAM (as is expected), but my error happens in the account modules.

So, one of the PAM account modules is denying my user.

Looking at /etc/pam.d/password-auth, which is a symlink to /etc/authselect/password-auth, we see:

account     required                                     pam_unix.so
account     sufficient                                   pam_localuser.so
account     sufficient                                   pam_usertype.so issystem
account     [default=bad success=ok user_unknown=ignore] pam_sss.so
account     required                                     pam_permit.so

Don’t really see anything that would allow one user, but deny an other.

pam_sss.so is the module that interfaces with FreeIPA via sssd. It sources it’s configuration in /etc/sssd/sssd.conf, and there we find our culprit:

From sssd.conf:

[domain/mydomain.example.com]
simple_allow_users = $, firstuser

sssd-simple(5), provides very simple user and group allow/deny lists. In this case, if a user is not listed in simple_allow_users, then they are not allowed to login.

What this means is, only firstuser is allowed to log in. Adding seconduser to the list will allow them to log in, and so forth.

This seems like a bug. At the least it’s very unexpected behavior!

This post is part of a collection of blog posts related to OpenShift and FreeIPA (aka idM).

• OpenShift Certificate from IPA on RHEL 8
• OpenShift authentication with IPA
• OpenShift Group Syncing with IPA (Not yet published)
• Automated Certificate Management with IPA and cert-manager (Not yet published)

Introduction

In the previous post, OpenShift Certificate from IPA on RHEL 8, I explained how to create certificates for an OpenShift cluster using FreeIPA/RHEL idM.

Now, I want to be able to log into my OpenShift cluster using my FreeIPA credentials.

OpenShift Background

OpenShift has the concept of an IdentityProvider which connects an source of identity verification, like FreeIPA’s LDAP server, to OpenShift.

OpenShift 4.x uses an object called an OAuth to configure identity providers like LDAP. For OpenShift 3.x, this configuration is similar, but locaed in /etc/origin/master/master-config.yaml

Let’s look at an example OAuth manifest:

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
  - ldap:
      attributes:
        email:
        - mail
        id:
        - dn
        name:
        - cn
        preferredUsername:
        - uid
      bindDN: uid=openshift,cn=sysaccounts,cn=etc,dc=private,dc=opequon,dc=net
      bindPassword:
        name: ldap-secret 
      ca:
        name: opequon-custom-ca-oauth
      insecure: false
      url: ldaps://ipa.private.opequon.net/cn=users,cn=accounts,dc=private,dc=opequon,dc=net?uid
    mappingMethod: claim
    name: ldapidp
    type: LDAP

This is an example of a LDAP Identity provider, there are other flavors if desired. Notice OAuth.spec.identityProviders looks for an array so it is possible to specify multiple providers.

We have a few things that need filled out here:

• name
• bindDN
• bindPassword
• ca
• url
• attributes
• mappingMethod

Name

This is the display name for this IdentityProvider. When logging on, if multiple IdentityProviders are configured, the user will see this name to choose from. In the case of only having a single provider, this is almost never seen.

Whatever value for name is, it should be something meaningful to the humans that interact with this OpenShift cluster.

bindDN and bindPassword

The username and password for accessing the cluster. bindPassword is normally sourced from a Kubernetes Secret.

ca

The certificate chain for the LDAP server. Almost every LDAP server uses a certificate signed by a non-public certificate authority (i.e. not included in the default RHEL certificate authority bundle), therefore this is almost always required.

Since FreeIPA acts as a certificate authority, this will need to be specified in the final configuration.

This is normally sourced from a ConfigMap.

url

This is the URL of the LDAP server with a query string, as defined by RFC4516 (which obsoletes, RFC2254). The query string is, along with the attributes, the most important part fo the configuration. If it is wrong, then no one can log in, and it is significantly easier to get wrong than the other fields.

The RFCs are rather dense reading, so it’s worthwhile to break down the parts of a query string:

schema://hostname:port/base_dn?attributes?scope?filter
                       ^                             ^
                       +-----------------------------+
                              Query String

After the schema, hostname, and port, there are four fields:

• base_dn
• attributes
• scope
• filter

LDAP organizes it’s data into a tree-like structure. base_dn specifies where in the tree to start searching. This can be visually seen by tools like Apache Directory Studio.

attributes specify what attributes from the found entities to return from the search. An entity is a node on the tree that represents something, like a user account or a group. These, of course, can have many attributes. Some common examples are mail for e-mail address, sn for surname, givenName, loginShell, homeDirectory.

For OpenShift, this should be a attribute that is going to be unique to each entity in the entities returned by the query, like uid which represents a Linux username.

Again, a tool like Apache Directory Studio can be very helpful exploring the LDAP structure.

scope determines if or how the search recurses through the tree. The options are:

• base
• one
• sub

The default is sub

base restricts itself to ONLY the base_dn and is not really useful in this situation.

one searches only the first level below the base_dn.

sub fully recurses the tree below the base_dn.

The LDAP Wiki has a good page explaining LDAP Search Scopes.

Finally, filter is an LDAP Filter that allows refinement of the resultset even further. This is a really deep topic, but a simple example of this is (objectclass=person). This filter will only return entities that are of objectclass person.

If no filter is specified, then (objectclass=*) is the default. This filter is true for all entities.

The LDAP Wiki has a lot of good examples of LDAP Queries.

With this information, we can create a URL for IPA. Fortunately, for the default install of FreeIPA, we can have a fairly simple query string.

From the example:

ldaps://ipa.private.opequon.net/cn=users,cn=accounts,dc=private,dc=opequon,dc=net?uid

This example omits some parameters. If we were to fill in the default explicitly, it would look like this:

ldaps://ipa.private.opequon.net/cn=users,cn=accounts,dc=private,dc=opequon,dc=net?uid?sub?(objectclass=*)

In FreeIPA, all users are always listed under cn=users,cn=accounts,dc=mydomain. This differs some other LDAP systems (specifically Active Directory). This makes our search really simple, as we can just pull every entity out of this base_dn and be assured that all users will be able to log into OpenShift.

An Aside: Restricting Access

In this example, the LDAP URL is constructed so that all users can log in to OpenShift.

In many situations, there is a desire to lock this down to allow, for example, only users from a certain group to log in or other similar restriction.

While this is certainly possible, with the correct LDAP filter. I would posit that this is an anti-pattern, and that access control is much better co-ordinated via Groups and RBAC policies.

I have several reasons for this:

1. It’s imperative that the LDAP query returns quickly. This query is run every time a user logs in. If the query takes a long time to run, then the user experience will suffer.

2. Changing the OAuth object can potentially cause loss of service, especially if updated with incorrect values.

If you configure a very complicated query (especially if using nested group search on a complicated LDAP tree), then #1 is very possible.

And, secondly, if you specify a complicated query it increases the chances of having to change the query in the future, which increases the changes for mistakes when updating the OAuth object.

Therefore, my opinion is to let everyone log in. It’s a simple query and is unlikely to every change, unless there are massive change to the LDAP structure.

In that situation, controlling access is done via Groups and Role-Based Access Control. This is easier to implement, less risky to change, and much more flexible. A future blog post will cover this topic in detaill.

attributes

When a user logs into OpenShift using an IdentityProvider, a set of proxy Kubernetes object is created to represent that user. These objects are used predominately in role based access control and group membership, as well as in creation metadata for certain other objects.

The attributes field configures the IdentityProvider to map attributes in the LDAP entity to fields in these proxy objects.

There are four attributes, let’s go over them quickly:

• name
• email
• id
• preferredUsername

name is a human readable name of the user. In Free IPA, this is in the cn attribute of the LDAP entity.

email is, obviously, the user’s email address. In Free IPA, this is in the mail attribute of the LDAP entity.

id is the most important of these attribute, as it is meant to map to a LDAP field that uniquely identifies the user. As a consequence, this attribute should never change during the life of the LDAP entity. If it does change then the link between the user entity in LDAP and the User object in OpenShift will be broken, and the user will no longer be able to log in.

In Free IPA, the id attribute should be mapped to the dn attribute of the LDAP entity.

preferredUsername is optional, but useful. By default, the IdentityProvider will use id as the username for that user in OpenShift. Unfortunately, dn is a big long, ugly LDAP Distinguished Name, like this: uid=cfh,cn=users,cn=accounts,dc=private,dc=opequon,dc=net.

preferredUsername, if specified, causes a different attribute to be used for username. Generally, I want to use the same username as I use for logging into Linux systems. In Free IPA, this is in the uid attribute of the LDAP entity.

mappingMethod

When multiple IdentityProviders are configured in a single cluster, then the mappingMethod is set to determine how username conflicts are handled.

In the case where you only have one IdentityProvider configured, then claim is the right value.

Putting it all together

Pre-requisites

Service Account

I need a read-only service account in my FreeIPA LDAP. This is not a regular user. I don’t want it to be able to log into any hosts or even the IPA console. I only want it to be able to do queries against FreeIPA’s LDAP.

To do this, create a LDAP System Account.

All these examples use the base domain of my system (dc=private,dc=opequon,dc=net). You’ll obviously need to swap this out with the specifics of your FreeIPA installation.

On the FreeIPA server:

[root@ipa ~]# ldapmodify -x -D 'cn=Directory Manager' -W
Enter LDAP Password: 
# Paste in the below
dn: uid=openshift,cn=sysaccounts,cn=etc,dc=private,dc=opequon,dc=net  <- CHANGE OPENSHIFT
changetype: add                                                          TO DESIRED NAME OF
objectclass: account                                                     SERVICE ACCOUNT
objectclass: simplesecurityobject
uid: openshift          <-  THIS SHOULD MATCH THE DN ABOVE
userPassword: changeMe  <-  THIS IS YOUR SERVICE ACCOUNT PASSWORD
passwordExpirationTime: 20380119031407Z <- 2038 EFFECTIVELY THE END OF TIME
nsIdleTimeout: 0

^D  <- ACTUALLY TYPE Control-D

If successful, you should see the following output:

adding new entry "uid=openshift,cn=sysaccounts,cn=etc,dc=private,dc=opequon,dc=net"

You can then test this service account using a command similar to the following:

ldapsearch  -x -D 'uid=openshift,cn=sysaccounts,cn=etc,dc=private,dc=opequon,dc=net' -W

This will spit out every object in the LDAP database. Of course you can filter, see ldapsearch(1).

For both the ldapmodify and ldapsearch command, the arguments mean the following:

• -x - use simple authentication (instead of a Kerberos ticket)
• -D - the distinguished name of the user
• -W - prompt for password

Certificate Authority

This file is normally found at /etc/ipa/ca.crt on any machine connected to FreeIPA.

Grab this file, so it can put it into a config map later, or alernatively run oc commands from a Linux machine that is connected to FreeIPA.

OpenShift Configuration

Creating Secrets and ConfigMaps

The OAuth object we are creating requires a Secret for the LDAP service account password.

This secret must be in the openshift-config Project, but can be named anything so long as that name is used in the later OAuth object.

To do so via the command line:

# oc project openshift-config 
# oc create secret generic ldap-secret --from-literal=bindPassword=changeMe

This will create a Secret like the below. Alternatively, this manifest can be applied directly using oc create, just be sure to substitute the value of data.bindPassword with the Base64 encoded string of your actual password.

apiVersion: v1
kind: Secret
metadata:
  name: ldap-secret
  namespace: openshift-config
type: Opaque
data:
  bindPassword: Y2hhbmdlTWU=

In addition to the Secret, the FreeIPA certificate authority chain must be in a ConfigMap object in the openshift-config namespace. This file is normally found at /etc/ipa/ca.crt on any machine connected to FreeIPA.

To create this ConfigMap via the command line:

oc project
oc create cm custom-ca-oauth --from-file=ca.crt=/etc/ipa/ca.crt -n openshift-config

Alternatively, this manifest can be applied directly using oc create.

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca-oauth
  namespace: openshift-config
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    REPLACE ME
    -----END CERTIFICATE-----

Creating OAuth Object

With the LDAP service account created and the Secrets and ConfigMaps in place, it’s time to create to update the OAuth object.

In OpenShift 4, an OAuth object will exist by default regardless of if there are any IdentityProviders configured. We could use oc replace to overwrite this object or just edit the object using either oc edit or the web console.

Regardless of method chosen, the resulting object definition should look like this:

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
  - ldap:
      attributes:
        email:
        - mail
        id:
        - dn
        name:
        - cn
        preferredUsername:
        - uid
      bindDN: uid=openshift,cn=sysaccounts,cn=etc,dc=private,dc=opequon,dc=net
      bindPassword:
        name: ldap-secret       # MUST MATCH OUR SECRET FOR LDAP
      ca:
        name: custom-ca-oauth   # MUST MATCH OUR CA CONFIGMAP
      insecure: false
      url: ldaps://ipa.private.opequon.net/cn=users,cn=accounts,dc=private,dc=opequon,dc=net?uid
    mappingMethod: claim
    name: ldapidp
    type: LDAP

bindDN and url must, of course, be updated to match your environment. The names of the Secret and Certificate Authority ConfigMap must match the names of those created in the previous section.

Once this is applied, the authentication ClusterOperator will apply the configuration you can check the status by looking at the ClusterOperator objects (short name for these is co)

# oc get co authentication
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.10.12   True        False         False      24h     

While being configured, the PROGRESSING column will flip to false. This configuration should only take a few minutes to apply.

If it the DEGRADED column flips to true for a long period (or if AVAILABLE becomes false), then use oc describe co authentication to see events related to the problem.

Testing Logging in

After the authentication ClusterOperator applies the configuration, on next log in attempt, the user will be presented with an option to log in using FreeIPA as the provider. The name of this provider is specified in the OAuth object spec.identityprovider.ldap.name field. In our example, this is ldapipa.

Once logged in, we should be able to see our User object.

# oc get users
NAME   UID                                    FULL NAME    IDENTITIES
cfh    826d995a-2045-42dd-a4fa-81d338ebbefe   Clark Hale   ldapidp:dWlkPWNmaCxjbj11c2Vycyxjbj1hY2NvdW50cyxkYz1wcml2YXRlLGRjPW9wZXF1b24sZGM9bmV0

Additionally, we can see the Identity object

[root@meriadoc opequon_labs_ocp_playbooks]# oc get identity
NAME                                                                                   IDP NAME   IDP USER NAME                                                                  USER NAME   USER UID
ldapidp:dWlkPWNmaCxjbj11c2Vycyxjbj1hY2NvdW50cyxkYz1wcml2YXRlLGRjPW9wZXF1b24sZGM9bmV0   ldapidp    dWlkPWNmaCxjbj11c2Vycyxjbj1hY2NvdW50cyxkYz1wcml2YXRlLGRjPW9wZXF1b24sZGM9bmV0   cfh         826d995a-2045-42dd-a4fa-81d338ebbefe

We can see that these long strings are actually the value of the id attribute, which we set to dn:

[root@meriadoc opequon_labs_ocp_playbooks]# echo "dWlkPWNmaCxjbj11c2Vycyxjbj1hY2NvdW50cyxkYz1wcml2YXRlLGRjPW9wZXF1b24sZGM9bmV0" | base64 -d
uid=cfh,cn=users,cn=accounts,dc=private,dc=opequon,dc=net

Next Steps

Now that users can log in, the next step is to synchronize groups from FreeIPA and then assigning Roles to those users.

• OpenShift Certificate from IPA on RHEL 8
• OpenShift authentication with IPA
• OpenShift Group Syncing with IPA (Not yet published)
• Automated Certificate Management with IPA and cert-manager (Not yet published)

    - name: Remove Annotation Blah
      k8s:
       api_key: ""
       state: present
       definition:
         apiVersion: v1
         kind: Node
         metadata:
           name: varda.private.opequon.net
           annotations:
             blah: NULL

The above snippet removes the blah annotation from this node object.

Certainly not intuitive…took me a good day or two of searching when I need this last year.

The AmigaDOS is an interesting piece of history. It was based upon TRIPOS, which was initially released in 1978. UNIX was released initially in 1973ish, so it’s unclear to me how much influence UNIX had on TRIPOS and subsequently AmigaDOS.

I have a suspicion TRIPOS was relatively free of UNIX influence, based on the absence of . and .. for relative file access.

For the uninitiated, . and .. are special files that occur in every UNIX directory. . refers to the current directory and .. refers to the parent directory. The root directory, i.e. /, is the only special case. Since it has no parent, root’s .. refers to itself.

What this means is, if I have a tree like this:

$ tree
test
├── a
│   └── file1
├── b
│   └── sub
│       ├── file2
│       └── subsub
└── c
    └── sea_file

I can easily refer to files in OTHER directories than my current working directory.

### My current working directory is test.
$ pwd
/test

### To refer to a file under test I can use '.' to refer to '/path/to/test'
$ cat ./a/file1
      ^
      +------------- The full for file is /test/.
                     but it refers to /test

### If I change my working directory to b/sub
$ cd b/sub

### I can still easily refer to files in other directories:

$ cat ../../a/file1
      ^  ^
      |  +----- .. of /test/b/, which refers to /test/
      +-------- .. of /test/b/sub which refers to /test/b/

This concept is so essential in UNIX/Linux that was at a bit of a loss when I started using the AmigaDOS shell again.

In UNIX, I’m very used to doing this:

cp -r /path/to/some/dir/* . 

To copy everything in /path/to/some/dir/ to the current directory.

In AmigaDOS, I reflexively ran a similar command (note: #? is equivalent to *):

5.Ram Disk:> copy work:test/#? .
   .   [created]
   a..copied.
   b..copied.

This created a directory called . and copied my files into it! NOT WHAT I WANTED!

5.Ram Disk:> dir
     . (dir)

This leads me to wonder….how does one do relative pathing in the Amiga Shell? If I used to know years ago, I’ve forgotten.

There are basically three characters that end up providing similar functionality to . and .. in UNIX:

• :
• /
• ""

Rather than having a unified file tree, AmigaOS has volumes each with their own name (similar to CP/M or VMS and derivative operating systems).

Volumes names come from three sources:

• File system labels, e.g. Workbench:, the normal volume name of the bootable AmigaOS partition.

• Device names, e.g. DF0: for the first floppy disk.

• Assigns which map to directories in volumes to volume names, e.g. C: which normally goes to SYS:C and SYS: which maps to the boot volume.

There’s only one name space for volumes, so multiple names could map to the exact same storage location. However, generally, the AmigaDOS Shell will show the file system label.

For example, the first floppy disk has a device name of DF0:, which never changes no matter which disk is inserted. A disk will have a file system label like Workbench: and if that disk is booted from, SYS: is also assigned to the disk. All three of these volume names point to the same storage.

Additionally, Assigns do not work the same when they point to sub directories of other volumes. If I do this:

5.Ram Disk:> cd c:
5.Workbench:C>

Notice my current working directory changes to what C: references, rather than have it appear as if it were an actual volume.

With that in mind, let’s return to my example and assume I have a volume name test: with the following tree:

test:
├── a
│   └── file1
├── b
│   └── sub
│       ├── file2
│       └── subsub
└── c
    └── sea_file

So long as my current working directory is somewhere under test:, I can can list the files in the root of test: like this:

5.TEST:c> dir :
     a (dir)
     b (dir)
     c (dir)

If my current working directory is test:b/sub, I can copy test:c/sea_file to there like so:

5.TEST:b/sub> dir
  file2   subsub
5.TEST:b/sub> copy //c/sea_file ""
5.TEST:b/sub> dir
  file2     sea_file  subsub

In this case, "" acts just like . in UNIX, and // acts just like ../../.

This doesn’t seem as elegant as UNIX, but it is functional. However, I’m sometimes unable to tell if I actually find UNIX design to be good or if I’m just used to it.

Originally, I was going to write that I couldn’t find this information in the AmigaOS documentation, but as I was finishing up this blog post, I found basically my exact description under the Command Line Characters section (sorry it’s not a direct link). No mention of “relative path” references, though…so it is a bit hard to find.

Recently, I’ve had to work pretty extensively with various cloud providers.

Many companies I’ve worked with fully integrate their cloud provider accounts into their existing on-premise network. For this blog, I’m going to call this a cloud “enclave”.

The hosts and services in the cloud enclave are, for all intents and purposes, equal to resources in the “on-premise” data center. VMs in the cloud get IP addresses that are routable on the existing, “on-premise” networks and DNS is configured such that cloud and “on-premise” names roll up under the same domain and are universally resolvable. Most cloud workloads are only accessible via the private network.

There are significant advantages to this “enclave” concept.

1. Since services in the cloud are not publically accessible, the security “threat-level” is similar to private, on-premise services.

2. There are not two classes of network/DNS services. Everything is basically on one big happy network.

3. (Arguably) things are more portable, since there’s not reliance on a cloud provider specific way to expose services.

Creating an “enclave” and bridging it to one’s “on-premise” network is a significantly more complicated activity than most tutorials describe. For a home lab, this may seem like overkill, but I consider this a useful exercise as it more closely resembles the setup I’ve seen in many companies.

This blog post describes the configuring a cloud enclave with Azure. I had three major goals with this:

My cloud enclave must

1. be routable from my existing home network.
2. integrate with my existing DNS solution
3. be automated so it could be created/destroyed on-demand
4. be minimally disruptive to my existing setup.

Existing Home Lab

First, let’s consider my “On-Premise” home network. I run many services that would be considered essential in a typical corporate network:

• DNS
• LDAP
• Kerberos
• Certificate Management

These services are provided by Red Hat Identity Management (IdM), which is a version of FreeIPA bundled with RHEL.

Additionally, I also have DHCP managed by Red Hat Satellite.

My IP address topology is relatively simple. I have a flat /24 as my main network, although I’ve dabbled with splitting out separate subnets and VLANs for things like baseboard management controllers and storage.

For my enclave, I only need to extend DNS and the routable network into the cloud. DHCP will be handled separately by the cloud, and the other services, e.g. LDAP, and Kerberos, will become accessible by virtue of extending DNS and the routable network.

Designing my IP Space

I have used 172.31.0.0/24 as my home network IP range for many years.

When provisioning private cloud networks, I could choose any IP range in the RFC1918 space, so long as it doesn’t overlap with my “on-premise” network.

I decided to extend my routable network to 172.31.0.0/16, which can be divided into 8 /19 networks, each having approximately 8192 IP addresses.

With this layout, my “on-premise” network can remain the same as it’s always been, since 172.31.0.0/24 is a subnet of 172.31.0.0/19. Additionally, I’ll have the potential to use other subnets of 172.31.0.0/19 in case I ever choose to deviate from a flat network.

With a /19, it’s highly unlikely that I’ll use all of the IP addresses in a given allocation. In my situation, this is ideal, since I will never have to worry about running out of IP addresses. In an “Enterprise” situation, I would be more granular in specifiying things (IPv6 can’t come soon enough!).

Designing DNS

Until now, I’ve had one flat DNS zone for all hostnames in my network: private.opequon.net.

I could continue with one flat zone, but through my experimentation, this seems to be more trouble than it is worth as every cloud host will have to register its hostname with my FreeIPA/IdM server.

Instead, it seems worthwhile to leverage each cloud providers internal DNS and delegate a zone to each cloud provider. This way a VM started in Azure can auto register its hostname, without any on-going effort on my part.

I can then bridge together the zones of my existing “on-premise” network and each cloud zone using DNS forwarders. My existing “on-premise” zone can remain untouched.

What I’ve come up with for my network is:

For Azure, there is a problem with this set up and reverse DNS, but I’ll address that in the “Problems and Improvements” section.

Critical Components

With my DNS and IP ranges set, we can move on to the critical components of this “enclave”. While there are many different “bits” that need to be configured, the two critical components that make this work are

1. Site-to-Site VPN
2. DNS Fowarding

Site-to-Site VPN w/ Libreswan

Azure offers a Site-to-Site VPN object that uses IPSec tunnels to create a VPN.

For the “on-premise” side of the tunnel, I use the Libreswan implementation that’s bundled with RHEL.

Visually, my configuration looks like this:

Azure does not directly support Libreswan, but does offer an Openswan option. Libreswan is a fork of Openswan and their configuration format has diverged somewhat, but it’s a good starting point.

For reference on configuring my RHEL gateway host, Using LibreSwan with Azure VPN Gateway was the only useful blog post I’ve been able to find.

DNS Forwarding

At time of writing, Azure does not have a first-class DNS forwarder service. Therefore, I have to build this component. Fortunately, it is straightforward to build a DNS forwarder using a tiny VM and DNSMasq.

The solution looks like this:

In Azure, I create a private zone (azure.private.opequon.net) and link it to my VNET. With that, hosts inside the VNET can query the Azure DNS servers and resolve hostnames in my private zone.

However, hosts outside of the VNET, like my on-premise network, cannot access Azure DNS!

Having the DNS Forwarder VM allows me to, essentially, proxy DNS queries between my on-premise FreeIPA and Azure DNS.

So that my Azure VMs also can resolve on-premise names. The DNS Forwarder VM also sends DNS queries back to FreeIPA for certain zones. To ensure all Azure VMs have this by default, the VNETs primary DNS server is changed from the default Azure DNS to my DNS Forwarder.

Let’s set everything up!

Manual Steps

Most things are covered by the automation, but a few items I had to manually configure due to lack of available automation modules and/or APIs.

Both of these items are one time setup, and neither incure a cloud cost.

Router Setup

My Libreswan endpoint is within my network and thus is behind my router. Therefore, I need to configure my router to forward certain ports to my Libreswan host.

If you Libreswan endpoint is at the edge of your network (i.e. it holds a public IP address), then this step is not necessary.

FreeIPA Forward Zones

At time of writing, there are no Ansible modules allow configuration of Forward Zones in FreeIPA/IdM. I may just have missed it, not sure.

For these examples to make sense, here is my relevant configuration:

• DNS Forwarder VM IP address: 172.31.225.4
• FreeIPA/IdM Server Hostname: ipa.private.opequon.net
• Azure subdomain: azure.private.opequon.net

First, I create a DNS Forward Zone in FreeIPA with a forward policy of ‘only’. This ensures that queries for azure.private.opequon.net are forwarded to the DNS Forwarder VM.

[root@ipa ~]# ipa dnsforwardzone-add --forward-policy=only --forwarder=172.31.225.4 azure.private.opequon.net.
Server will check DNS forwarder(s).
This may take some time, please wait ...
ipa: WARNING: DNS server 172.31.225.4: query 'azure.private.opequon.net. SOA': The DNS operation timed out after 10.0006103515625 seconds.
  Zone name: azure.private.opequon.net.
  Active zone: TRUE
  Zone forwarders: 172.31.225.4
  Forward policy: only

The warning about DNS operation timed out will occur if you’re performing this step BEFORE provisioning the entire enclave with the rest of the automation. Ignore it.

Next, to make the forward zone effective, I need a “glue record” to include it in my primary zone. This is just a simple NS record that points back to my IPA server.

[root@ipa ~]# ipa dnsrecord-add private.opequon.net azure --ns-rec=ipa.private.opequon.net.
  Record name: azure
  NS record: ipa.private.opequon.net.

Once the rest of the enclave is configured, this IdM configuration should immediately (or pretty close to immediately) work.

Automation Code

All the code described here is on GitHub

The automation I have for this is written in a combination of Terraform and Ansible.

Two main scripts call the pieces in unison:

• build.sh - Creates all cloud infrastructure in Azure and setups Libreswan and the DNS Forwarder
• destroy.sh - Tears everything down.

The build.sh and destroy.sh scripts assumes that the Site-to-Site VPN tunnel is being configured on the same host that is running the script. Some modification of the Ansible will be required if you want to run it from a different host.

Sometimes the Azure APIs are very slow and cause the Terraform code to timeout! Luckily, all this code is idempotent, so if that happens, it can just be run again.

Terraform Files

I’ve attempted to format and organize the Terraform files in a way that is easy to understand, rather than strictly to normal.

The main objects are in the following files:

• main.tf - Objects for the site-to-site VPN
• subnets.tf - Subnets and Network Security Groups
• dns.tf - DNS Private Zone
• dns_forwarder.tf - DNS Forwarder VM

Then there are more normal Terraform files, that have what you’d expect.

• outputs.tf
• providers.tf
• variables.tf
• versions.tf

main.tf

main.tf contains the Site-to-Site VPN resources, and the bare minimum pre-requisites.

azurerm_resource_group defines the resource group. Every Azure resource by this automation will live in this resource group.

azurerm_virtual_network defines our “Azure Virtual Network”, aka VNet, to which we will add subnets. The VNet has variables for the overarching address space as well as DNS servers for everything. In this automation, I’m overriding the default Azure DNS with the statically assigned IP address of our DNS Forwarder.

azurerm_subnet creates the subnet required for the Site-to-Site VPN. This is here, instead of in subnets.tf, because it’s required for the VPN connection. Also, it must be named GatewaySubnet, otherwise dependant resources throw an error.

azurerm_local_network_gateway provides details for our “Local” network, meaning the “On-Premise” side of the Site-to-Site VPN tunnel.

azurerm_public_ip provides a public IP address for the Azure side of the Site-to-Site VPN tunnel.

azurerm_virtual_network_gateway provides the Azure-side of the Site-to-Site VPN tunnel. It links together the Public IP address and the Azure GatewaySubnet.

azurerm_virtual_network_gateway_connection creates the VPN. It links together the azurerm_virtual_network_gateway, azurerm_local_network_gateway, and the VPN Shared Key and creates the VPN configuration on the Azure side. Once this resource has finished provisioning, it is possible to start LibreSwan on the on-premise side and activate the connection.

subnets.tf

subnets.tf contains all azurerm_subnet definitions, except for the GatewaySubnet.

In the public repo, I have just one subnet for my “main” network. In my personal lab, I add separate subnets for OpenShift and other things I run in the cloud.

dns.tf

dns.tf does some minimal DNS work.

azurerm_private_dns_zone actually creates our private DNS zone in Azure.

azurerm_private_dns_zone_virtual_network_link then links that DNS zone to our VNet. I have the registration_enabled flag set to true, so all Virtual Machines connected to the VNet will automatically have their hostnames registered in our private zone.

dns_forwarder.tf

dns_forwarder.tf defines the Virtual Machine that will be our DNS forwarder.

azurerm_network_interface defines it’s network interface and, importantly, its static IP within the VNet.

azurerm_linux_virtual_machine is the virtual machine definition. Currently uses a RHEL 8 VM, but this could be changed to almost any RHEL-like VM and still work.

azurerm_network_security_group is a simple network security group for the instance.

Ansible Playbooks

There are three Ansible playbooks. The build.sh and destroy.sh scripts pass the outputs of the Terraform build into these, so they have no separate inventory or variable files.

• azure_dns_forwarder.yaml - Configures the DNS forwarder
• azure_ipsec.yaml - Configures on localhost, the on-premise side of the VPN tunnel
• azure_ipsec_remove.yaml - Destroyes the on-premise side of the VPN tunnel

Sample terraform.tfvars

A sample terraform.tfvars is included.

# What Azure Region to use for all resources
azure_region_name="eastus"

# Resource group for all resources
resource_group_name="ENCLAVE-EAST"

# Default name for most resources
default_resource_name="ENCLAVE-EAST"

# Address Space and DNS servers for VNET
vnet_address_space=["172.31.224.0/19"]
vnet_dns_servers=["172.31.225.4"]

# Subnet for the Azure side of the Site-to-Site VPN
gateway_subnet=["172.31.224.0/24"]

# Names for Local Network Gateway and resources related to
# the on-premise side of the Site-to-Site VPN
local_network_gateway_name="ONPREM"
on_premise_name="ONPREM"

# On-Premise network range and public IP.
on_premise_private_network_ranges=["172.31.0.0/19"]
on_premise_public_ip_address="108.56.139.185"

# DNS Zone for Azure Private DNS
private_dns_zone_name="azure.private.opequon.net"

# Static Private IP address of DNS Forwarder.
dns_forwarder_ip_address="172.31.225.4"

# Name for thePublic IP address for the Azure side of the VPN
vpn_azure_public_ip_name="VPN-EAST"

# Name and range of main subnet in Azure
subnet_main_name = "EAST-MAIN"
subnet_main_cidr_ranges = ["172.31.225.0/24"]

# DNS information for DNS Forwarder VM
on_premise_dns_zones = ["private.opequon.net", "0.31.172.in-addr.arpa"]
on_premise_dns_server = "172.31.0.101"
# The Azure Private DNS server is a static IP address
# See:  https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances#considerations
azure_private_dns_server = "168.63.129.16"
# The reverse DNS doesn't work, currently, for on-prem resolving Azure PTR records.
vnet_dns_reverse_zones = ["225.31.172.in-addr.arpa"]

# Shared Key for VPN
# DO NOT CHECK THIS INTO GIT, it's only here to show what the variable name is!
vpn_shared_key="mysecretkey"

Problems and Improvements

There are still problems with this. Maybe I will solve and update when I have the time.

Lack of security groups

This configuration is pretty wide open internally. I’m doing very little segregation of Azure subnets that would be common in an Enterprise organziation.

I still feel pretty safe with this configuration, because it’s only really accessible from my home network, but if using this as a pattern for something more serious. It’s worth considering additional network segmentation.

Reverse DNS

Try as I might, I cannot get Reverse DNS forwarding to Azure private DNS working with FreeIPA/IdM.

When I set up a forward zone in FreeIPA/IdM, it expects that the destination DNS to have a SOA record equivalent to the forward zone. So, if my forward zone is 225.31.172.in-addr.arpa, then the DNS server I’m forwarding to is expected to have an SOA record for 225.31.172.in-addr.arpa.

Azure DNS appears to dump all PTR records into a big in-addr.arpa zone. So, there are no appropriate SOA records and FreeIPA/IdM refuses to set up the forward zone.

Other DNS implementations, like dnsmasq, can cope with this. I suspect because they textually parsing the DNS query at the server before sending it on, but I’m not entirely sure. A bit more research is required.

At any rate, Reverse DNS does not automatically work with this solution, which is a problem.

Plain text VPN key

This example just has the shared VPN key as plain text. For a home lab, this is mostly acceptable, although still not ideal. For an enterprise solution, this VPN key should be vaulted and highly protected.

Routing weirdness

In this example, the IPSec tunnel is not at the edge of the network, but is instead inside my private network. My consumer router forwards certain ports to the IPSec host in order to make the connection work.

When I’m on the road (a rarity these days), I use the VPN client/server provided by my consumer router, and there are routing difficulties when trying to use resources in the enclave when I’m connected to my router’s VPN.

My regular desktop operating system and Jekyll has been packaged by Fedora for some years. However, most plugins, including jekyll-redirect-from, are NOT packaged by Fedora.

In the past, this has led me to have to muck about with other package tools, like gem or pip. I don’t like it. They put stuff in weird places, duplicate system provided dependencies and just cause a mess.

No…call me stubborn, but I’d just like a nice simple RPM packages. Thank you very much.

Enter Fedora Copr. Copr is a front-end to Fedora’s package build system that allows one to build custom RPM packages and host them in a personal, publicly-accesible, yum repositories.

Copr has been around for a while, and I play around building some Emacs packages about 4 years ago, but it’s become substantially easier.

In just a few commands, I can have the Jekyll plugin I want, or really most Gems, converted to an RPM and available for me to install.

Setup

First, in order to use Copr, a Fedora Account is required. Signing up for one is free and relatively easy.

Next, I’d highly recommend getting the copr-cli package installed on your Fedora machine. It’s part of the default Fedora repositories, so it should be as simple as dnf install copr-cli.

Almost all of the tasks could be done through the Copr Web UI, but in this post, I’m using copr-cli.

Next, populate the Copr token by visiting https://copr.fedorainfracloud.org/api/

Put the output there into ~/.config/copr. It should look something like this:

[copr-cli]
login = gibberish
username = your_user_name
token = gibberish
copr_url = https://copr.fedorainfracloud.org
# expiration date: 2022-07-09

copr whoami should now return your Fedora Account name:

[chale@work-wired ~]$ copr whoami
xlark

Creating a project

The first step is to create a project. This acts as your yum repository. It’s possible to create multiple projects, if desired.

For example, I have an emacs project for Emacs related packages, and I created a jekyll-gems project for all the Jekyll Gems I intend to use:

To create a project named jekyll-gems, run the following command:

[chale@work-wired ~]$ copr-cli create --chroot fedora-35-x86_64 jekyll-gems
New project was successfully created.

The --chroot fedora-35-x86_64 specificies the package build environment. There are many different options. Fedora 35 seemed reasonable to me, since I’m currently using it, although rawhide may be a more future proof selection.

Building the Package

Now that we have a project, building a package from a Gem file in just one command:

[chale@work-wired ~]$ copr-cli buildgem --gem jekyll-redirect-from jekyll-gems
Build was added to jekyll-gems:
  https://copr.fedorainfracloud.org/coprs/build/3144944
Created builds: 3144944
Watching build(s): (this may be safely interrupted)
  17:17:28 Build 3144944: pending
  17:17:58 Build 3144944: running
  17:20:28 Build 3144944: succeeded

This builds an RPM from the Gem, jekyll-redirect-from into the project jekyll-gems.

Since, jekyll-redirect-from is hosted on RubyGems, Copr pulls the Gem file directly from there without any additional information required.

After the build has succeeded, the package is ready to install!

Installing the Package

First enable your Copr repository. The name should always be in the form of “Fedora Account/Project Name”:

[chale@work-wired ~]$ sudo dnf copr enable xlark/jekyll-gems 
Enabling a Copr repository. Please note that this repository is not part
of the main distribution, and quality may vary.

The Fedora Project does not exercise any power over the contents of
this repository beyond the rules outlined in the Copr FAQ at
<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,
and packages are not held to any quality or security level.

Please do not file bug reports about these packages in Fedora
Bugzilla. In case of problems, contact the owner of this repository.

Do you really want to enable copr.fedorainfracloud.org/xlark/jekyll-gems? [y/N]: y
Repository successfully enabled.

After that, we can just install the package like normal! By Fedora standards, all RPMs for Gems are always prefixed by “rubygem-“

[chale@work-wired ~]$ sudo dnf install rubygem-jekyll-redirect-from
Copr repo for jekyll-gems owned by xlark                                                              15 kB/s | 2.6 kB     00:00    
Dependencies resolved.
=====================================================================================================================================
 Package                            Architecture Version                Repository                                              Size
=====================================================================================================================================
Installing:
 rubygem-jekyll-redirect-from       noarch       0.16.0-1.fc35          copr:copr.fedorainfracloud.org:xlark:jekyll-gems        14 k

Transaction Summary
=====================================================================================================================================
Install  1 Package

Total download size: 14 k
Installed size: 9.8 k
Is this ok [y/N]: 

Sharing your project

Projects and their corresponding repositories created in Copr are public. Anyone running Fedora can enable your repository using dnf copr enable.

Others can benefit from whatever you package using Copr, so if you package something useful, feel free to share with others!

• Resurrecting the VAXen, part 1
• Resurrecting the VAXen, part 2: SCSI2SD
• Resurrecting the VAXen, part 3: Hardware Repairs

Start and stop is the nature of a lot of personal projects. I started to get my VAXen up and running several years ago and then, had to stop for other life reasons.

My condo was remodeled and I’m having to reassemble my lab, so it seems like a good time to restart the project.

First thing first was testing all the VAXen on taking them out of storage. All booted and got to the boot monitor, except for my VAXstation 4000/90. For the entire time I’ve owned it, this poor VAX has never booted and I never dedicated the time to diagnosing it. This is my most powerful VAX, so I’ve always wanted to restore it to a working state.

Turns out, there were two major issues with this VAX:

• A Damaged SIMM socket, presumably preventing memory
• A dead clock battery

VAXen have diagnostic lights to assist in boot up diagnostics. This VAX had all eight lights lit steady on bootup, and would not progress from there.

According to the VAXArchive, this means “Power is applied, but no instruction is executed”; essentially, no instructions have been executed by the CPU. During normal startup, this is the first light combination displayed, but a healthy VAX quickly progresses to other light sequences.

Staying in this state, per my research, generally indicates a memory issue or an issue with the real time clock.

First, I experimented with the memory. Maybe I had a bad module? Luckily, my VAXstation 4000/60 is functional and takes the same SIMMs as the 4000/90.

To my knowledge there are 2 sizes of SIMM modules available for VAXstation 4000s: 4MB and 16MB. These are fairly easy to distinguish based on the labels affixed to them. AA mean 4MB, CA means 16MB

For the 4000/90, these need to be installed in sets of four. The 4000/90 has 8 slots and no on-board RAM, where as the 4000/60 has 6 slots and 8MB of onboard RAM.

The 4000/60 and the 4000/90 are different in how SIMMs must be installed and the differences are not immediately intuitive. Fortunately, the service manual is available in the MANX Archive.

For the 4000/60, matching pairs must be installed with the lower capacity SIMMs in lower numbered banks.

For the 4000/90, matching sets of 4 must be installed in a staggered pattern.

All of the RAM from the 4000/90 tested good in the 4000/60, so all of the SIMM modules were good.

My 4000/90 is “fully loaded”, and it has always had all SIMM spots occupied. So, knowing that the SIMMs were good, I loaded only one set of four.

On power up, I got the boot monitor!

This is the first time this machine has booted in the decade I’ve owned it!

The culprit here was a damaged SIMM socket, it’s very difficult to get a picture of this, but I’ve tried.

I’ve ordered, what may be a replacement on Digi-Key.

Now, this solved some of the problems, but it was not always reliably booting. If I started it completely cold, it would still be stuck with all diagnostic LEDs illuminated. However, a quick power cycle after a few minutes of running would reliably get the machine to the boot monitor.

This has to be the real time clock, as I’ve seen reports that a dead real time clock will cause this symptom. The battery is, I presume, the original and does not hold a charge. Interestingly enough, the 4000/60 does not seem to be affected by this, even though it’s a similar, but not identical, system board.

Most DEC products of this era use the DS1287 RTC chip. This chip contains both the circuitry for the clock, as well as a built in battery. These have been out of production for years, and many people have resorted to hacking them to replace the built in battery.

I can’t be arsed to do something like that, especially because there are plug-compatible parts stil being produced. The DS12887+ and DS12887A+ are produced by Maxim Integrated Products, and I’ve used them before in PCs and other systems that use the DS1287. I figured I might as well replace ALL of my DS1287’s so I’ve ordered several from Digi-Key At over $10 a pop, they are not as economical as coin cells, but it beats having to do some time consuming modification to the either the DS1287 chip or the board.

The last remaining issue is that the SCSI2SD card that I’m using with the VAXes always throws an error on boot. I can later issue boot dka0 from the system monitor, but this error prevents the VAX from auto-booting.

?? 001  10      SCSI  0048

Currently, I can’t seem to find any information for this error, but that’s something for another day.