To BE or not to BE? Using Qemu to run a Big Endian Debian system

An issue was reported on a Big Endian system. And, if memory serves right, we had Big Endian bugs a few times in the past.

For readability, we'll refer to Big Endian as BE and Little Endian as LE.

The reason we didn't catch the bug in the first place is, despite the fact that we have extensive testing on multiple OSes, using different compilers and across different CPUs, all our test systems are LE.

Endianness is, basically, the way bytes are organized in memory. We started a long time ago with BE, then got LE systems. Some of them are Bi-endian and can do either BE and LE. Wikipedia has more details if you'd like to read about it. 

There is a number of CPUs that can run in BE: SPARC, MIPS, PowerPC, ARM and a few others. Like our x86 CPUs, our favorite ARM boards all run in LE but they can also run Big Endian. It was probably easier to run them in LE, less maintenance to do which means developers can focus on the important things: stability and improving hardware support

We could blindly fix the bug in our tests, but being able to test it ourselves would be better and possibly easier. That could possibly open the door to a new buildbot.

Finding a Linux (or BSD) that support it is not easy. Other option for a usable, recent Linux supporting BE is Gentoo, CLFS and possibly Arch. Embedded OS such as OpenWrt is apparently another possibility but it is limited in terms of packages. Unfortunately, Debian dropped support for PowerPC (BE) 2 years ago.

FYI, if you are looking for cheap hardware for a native Big Endian system, look for a Power Mac G5 (or G4) and install FreeBSD powerpc.

That's where qemu is great, as you can see in a previous post. The advantage of doing it in software versus getting physical hardware is that it we can run it along the rest of the buildbots with existing hardware and thus we avoid having to spend extra to power dedicated hardware, its maintenance and rack space.

Initial set-up

First, we need to install qemu and its utilities. We'll use the MIPS architecture in this case, on an Ubuntu 22.04 64 bit host. If you want to try the PPC architecture, this post is a good starting point.

apt install qemu-system-mips qemu-utils

Now, we need to get the appropriate kernel and initrd to do a netboot. You might need to adjust the URL to download the initrd and kernel in the future.

Notes: 

• The mips architecture is BE. If we wanted to do LE, we would go with mipsel or mips64el. Same goes for PowerPC, however Debian only offers PPC in LE (ppc64el). All ARM on Debian is LE.
• Buster is the last Debian release that offers MIPS Big Endian images. Bullseye no longer does.
  

wget http://ftp.debian.org/debian/dists/buster/main/installer-mips/current/images/malta/netboot/initrd.gz
wget http://ftp.debian.org/debian/dists/buster/main/installer-mips/current/images/malta/netboot/vmlinux-4.19.0-21-4kc-malta

Now, let's create a qcow2 disk image:

user@ubuntu:~$ qemu-img create -f qcow2 disk.img 25G
Formatting 'disk.img', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=26843545600 lazy_refcounts=off refcount_bits=16

Where:

• -f qcow2 specifies the format, qcow2
• disk.img is the resulting file
• 25G is the maximum size of the disk inside the image. That size is not allocated immediately, the file will keep growing as changes are done (additions/deletion).

We are allocating much more than we actually need so we have room to install other packages later on. A base installation would be fine with 2Gb.

Installation

Now, let's install Debian on the guest. Bear in mind that the installation will take about an hour on a recent CPU:

qemu-system-mips -hda disk.img -kernel vmlinux-4.19.0-21-4kc-malta -initrd initrd.gz -nographic -m 256m

Where

• -hda points to the disk image
• -kernel points to the kernel we downloaded
• -initrd points to the netboot install initrd
• -nographic will not open a graphic interface and display output in the current console
• -m 256m gives 256Mb of memory to the guest. 32 bit kernels are limited to 256Mb of RAM (if unspecified, default for qemu is 128Mb). 64 Bit MIPS qemu can get up to 2047Mb but there isn't any Debian for that. If you were to use more than 256Mb, adding "mem=256m@0x0 mem=XXXm@0x90000000" to -append (where XXX is the amount in -m minus 256Mb) might be needed

Note: mips64 would have been preferable because it can support more RAM but Debian doesn't offer that architecture.

It will start in the console. Simply follow the instructions like you would install a regular Debian system. The only two important choices that were made here, were to install all files in a single partition (and use the simple guided process when partitioning) and not install any X system due to the low memory.

At the end of the installation, a warning windows will be displayed mentioning there is no bootloader installed. It is expected, so it's fine.

A few moments later, Debian will let you know the installation is done. It will not shutdown but reboot (and restart with the netinstall if we let it). Interrupt the process by closing the terminal window (or hit Ctrl + a, c then input the command 'quit' to stop it).

.


The initrd we downloaded earlier is for netinstall only it won't work to boot our system. We'll need to grab the one generated during the installation. For this, we'll mount the qcow2 image we just installed, disk.img. One way to mount it is to use the tools included with qemu.

We first need to load nbd module:

sudo modprobe nbd

Then we connect the image to /dev/nbd0 and mount its first partition (because we installed all the files in a single partition) somewhere on our host:

mkdir qcow
sudo qemu-nbd --connect=/dev/nbd0 disk.img
sudo mount /dev/nbd0p1 qcow

From there, we'll copy the initrd from /boot then unmount it and disconnect the image:

cp qcow/boot/initrd.img-4.19.0-21-4kc-malta .
sudo umount qcow
sudo qemu-nbd --disconnect /dev/nbd0
rmdir qcow

Now, let's update our above command line to run our newly installed system:

sudo qemu-system-mips -hda disk.img -kernel vmlinux-4.19.0-21-4kc-malta -initrd initrd.img-4.19.0-21-4kc-malta -append "root=/dev/sda1" -nographic -m 256m -net user,hostfwd=tcp::1022-:22 -net nic

We also added forwarding to access SSH on the system. Refer to the previous blog post for more details about it.

After booting, we're greeted with a familiar prompt:

Debian GNU/Linux 10 debian ttyS0

debian login:

After logging in with the credentials we configured during the installation, running lscpu will give the following result:

root@debian:~# lscpu
Architecture:          mips
Byte Order:            Big Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
BogoMIPS:              1228.80

Which confirms it's a BE system. There are a number of other ways to determine that.

If updating the system will bring a new kernel, and we'll simply have to follow the same procedure as described above when we copied the initrd. In this case, get both the new kernel and its corresponding initrd then adjust the qemu command line once again for the next time we boot it (update both -kernel an -initrd entries).

Note: Compiling Aircrack-ng will take about an hour with a recent CPU. Just running tests (make check) will take longer the the compilation itself.

Trimming

The qcow2 image will keep growing even if we remove packages or delete files. Reclaiming free space is just a matter of zero'ing the disk space left in the guest then recompressing the image on the host after powering it off.

The first step is to fill the disk with zero's in the guest using dd then deleting the file. Make sure you have enough disk space on the host before you do that:

dd if=/dev/zero of=zerofile
rm -f zerofile

Deleting the file is very important or you may end up with an unbootable system. If that happens, just mount the qcow2 image like shown above and delete the file.

When done, shut the guest down. If you look at the file, disk.img, on the host, it will take the amount of space we initialized it with, 25Gb. Now, recompress it:

mv disk.img disk.img.bak
qemu-img convert -O qcow2 disk.img.bak disk.img

After installation and updates, it took approximately 1.8Gb and recompressed, 1.6Gb, saving roughly 200Mb.


Now, enjoy your new MIPS Big Endian system. Compiling aircrack-ng inside is exactly the same procedure as you would do on a regular x86 Debian system.