Monday, October 15, 2018

To BE or not to BE? Using Qemu to run a Big Endian Debian system

An issue was reported on a Big Endian system. And, if memory serves right, we had Big Endian bugs a few times in the past.

For readability, we'll refer to Big Endian as BE and Little Endian as LE.

The reason we didn't catch the bug in the first place is, despite the fact that we have extensive testing on multiple OSes, using different compilers and across different CPUs, all our test systems are LE.

Endianness is, basically, the way bytes are organized in memory. We started a long time ago with BE, then got LE systems. Some of them are Bi-endian and can do either BE and LE. Wikipedia has more details if you'd like to read about it

There is a number of CPUs that can run in BE: SPARC, MIPS, PowerPC, ARM and a few others. Like our x86 CPUs, our favorite ARM boards all run in LE but they can also run Big Endian. It was probably easier to run them in LE, less maintenance to do which means developers can focus on the important things: stability and improving hardware support

We could blindly fix the bug in our tests, but being able to test it ourselves would be better and possibly easier. That could possibly open the door to a new buildbot.

Finding a Linux (or BSD) that support it is not easy. Other option for a usable, recent Linux supporting BE is Gentoo, CLFS and possibly Arch. Embedded OS such as OpenWrt is apparently another possibility but it is limited in terms of packages. Unfortunately, Debian dropped support for PowerPC (BE) 2 years ago.

FYI, if you are looking for cheap hardware for a native Big Endian system, look for a Power Mac G5 (or G4) and install FreeBSD powerpc.

That's where qemu is great, as you can see in a previous post. The advantage of doing it in software versus getting physical hardware is that it we can run it along the rest of the buildbots with existing hardware and thus we avoid having to spend extra to power dedicated hardware, its maintenance and rack space.

Initial set-up

First, we need to install qemu and its utilities. We'll use the MIPS architecture in this case, on an Ubuntu 18.04 64 bit host. If you want to try the PPC architecture, this post is a good starting point.

apt install qemu-system-mips qemu-utils

Now, we need to get the appropriate kernel and initrd to do a netboot. You might need to adjust the URL to download the initrd and kernel in the future.

Note: the mips architecture is BE. If we wanted to do LE, we would go with mipsel or mips64el. Same goes for PowerPC, however Debian only offers PPC in LE (ppc64el). All ARM on Debian is LE.

wget http://ftp.debian.org/debian/dists/stretch/main/installer-mips/current/images/malta/netboot/initrd.gz
wget http://ftp.debian.org/debian/dists/stretch/main/installer-mips/current/images/malta/netboot/vmlinux-4.9.0-7-4kc-malta


Now, let's create a qcow2 disk image:

user@ubuntu:~$ qemu-img create -f qcow2 disk.img 25G
Formatting 'disk.img', fmt=qcow2 size=26843545600 cluster_size=65536 lazy_refcounts=off refcount_bits=16


Where:
  • -f qcow2 specifies the format, qcow2
  • disk.img is the resulting file
  • 25G is the maximum size of the disk inside the image. That size is not allocated immediately, the file will keep growing as changes are done (additions/deletion).
We are allocating much more than we actually need. A base installation would be fine with 2Gb.

Installation

Now, let's install Debian on the guest. Bear in mind that the installation will most likely take a few hours:

qemu-system-mips -hda disk.img -kernel vmlinux-4.9.0-7-4kc-malta -initrd initrd.gz -nographic -m 256m

Where
  • -hda points to the disk image
  • -kernel points to the kernel we downloaded
  • -initrd points to the netboot install initrd
  • -nographic will not open a graphic interface and display output in the current console
  • -m 256m gives 256Mb of memory to the guest. 32 bit kernels are limited to 256Mb of RAM (if unspecified, default for qemu is 128Mb). 64 Bit MIPS qemu can get up to 2047Mb but there isn't any Debian for that. If you were to use more than 256Mb, adding "mem=256m@0x0 mem=XXXm@0x90000000" to -append (where XXX is the amount in -m minus 256Mb) might be needed

Note: mips64 would have been preferable because it can support more RAM but Debian doesn't offer that architecture.

It will start in the console. Simply follow the instructions like you would install a regular Debian system. The only two important choices that were made here, were to install all files in a single partition (and use the simple guided process when partitioning) and not install any X system due to the low memory.

At the end of the installation, a warning windows will be displayed mentioning there is no bootloader installed. It is expected, so it's fine.













A few moments later, Debian will let you know the installation is done. It will not shutdown but reboot (and restart with the netinstall if we let it). Interrupt the process by closing the terminal window (or hit Ctrl + a, c then input the command 'quit' to stop it).

.


The initrd we downloaded earlier is for netinstall only it won't work to boot our system. We'll need to grab the one generated during the installation. For this, we'll mount the qcow2 image we just installed, disk.img. One way to mount it is to use the tools included with Qemu.

We first need to load nbd module:

sudo modprobe nbd

Then we connect the image to /dev/nbd0 and mount its first partition (because we installed all the files in a single partition) somewhere on our host:

mkdir qcow
sudo qemu-nbd --connect=/dev/nbd0 disk.img
sudo mount /dev/nbd0p1 qcow


From there, we'll copy the initrd from /boot then unmount it and disconnect the image:

cp qcow/boot/initrd.img-4.9.0-7-4kc-malta .
sudo umount qcow
sudo qemu-nbd --disconnect /dev/nbd0

rmdir qcow

Now, let's update our above command line to run our newly installed system:

sudo qemu-system-mips -hda disk.img -kernel vmlinux-4.9.0-7-4kc-malta -initrd initrd.img-4.9.0-7-4kc-malta -append "root=/dev/sda1" -nographic -m 256m -net user,hostfwd=tcp::1022-:22 -net nic

We also added forwarding to access SSH on the system. Refer to the previous blog post for more details about it.

After booting, we're greeted with a familiar prompt:

Debian GNU/Linux 9 debian ttyS0

debian login:


After logging in with the credentials we configured during the installation, running lscpu will give the following result:

root@debian:~# lscpu
Architecture:          mips
Byte Order:            Big Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
BogoMIPS:              1161.21


Which confirms it's a BE system. There are a number of other ways to determine that.

Updating the system will bring a new kernel, and we'll simply have to follow the same procedure as described above when we copied the initrd. In this case, get both the new kernel and its corresponding initrd then adjust the qemu command line once again for the next time we boot it (update both -kernel an -initrd entries):

sudo qemu-system-mips -hda disk.img -kernel vmlinux-4.9.0-8-4kc-malta -initrd initrd.img-4.9.0-8-4kc-malta -append "root=/dev/sda1" -nographic -m 256m -net user,hostfwd=tcp::1022-:22 -net nic

Here is what the updated system looks like (uname -a):

Linux debian 4.9.0-8-4kc-malta #1 Debian 4.9.110-3+deb9u5 (2018-09-30) mips GNU/Linux

Trimming

The qcow2 image will keep growing even if we remove packages or delete files. Reclaiming free space is just a matter of zero'ing the disk space left in the guest then recompressing the image on the host after powering it off.

The first step is to fill the disk with zero's in the guest using dd then deleting the file. Make sure you have enough disk space on the host before you do that:

dd if=/dev/zero of=zerofile
rm -f zerofile


Deleting the file is very important or you may end up with an unbootable system. If that happens, just mount the qcow2 image like shown above and delete the file.

When done, shut the guest down. If you look at the file, disk.img, on the host, it will take the amount of space we initialized it with, 25Gb. Now, recompress it:

mv disk.img disk.img.bak
qemu-img convert -O qcow2 disk.img.bak disk.img


After installation and updates, it took approximately 1.8Gb and recompressed, 1.6Gb, saving roughly 200Mb.


Now, enjoy your new MIPS Big Endian system. Compiling aircrack-ng inside is exactly the same procedure as you would do on a regular x86 Debian system.

Saturday, October 6, 2018

Aircrack-ng packages

As mentioned in our 1.4 release blog post, we are now providing packages (repositories) for a "few" Linux distributions (and sometimes multiple version of each of them):
  • Debian
  • Ubuntu
  • Mint
  • SLES
  • OpenSuse
  • Fedora
  • RHEL
  • CentOS
  • Amazon Linux
  • Elementary OS
TL;DR: if all you want is to install the package, head over here.

Rationale

Packages are an easy and convenient way to install software compared to compiling it.

All you have to worry about, after installing a package, is to make sure your system is up to date and there is no need to worry about each individual piece of software anymore. Today's distributions often even check automatically and notify if updates are available.

Another reason is that most Linux distributions often have old versions of Aircrack-ng in their repository and sometimes are a few years old.

It can be a problem for us when providing support. We often end up telling people to uninstall and recompile the latest release or try the current development code where the bug they're experiencing is fixed.

If you're a software developer, it's not too hard to figure out how to compile a piece of software, as long as the software is current and is documented. However, in some cases, it can get complicated.
And if you're not a developer, it is a daunting task.

We recently decided to tackle this issue and provide recent versions via packages, and for multiple OSs.

Why not a snap or a flatpak?

There are more than just those two possibilities as you can see in this post and all of those software are still not widely adopted yet. We may, in the future, offer snaps or flatpaks.

Buildbots

For CI/CD, we have been using buildbots on top of Travis CI and AppVeyor to automatically build Aircrack-ng on multiple platforms and multiple distros. It happens to every commit done to the master branch in our GitHub repository

We recently added a packages building bot to the buildbots for all the distributions mentioned on top.

Building packages and dependencies


Creating packages for that many distributions is not easy and if we did it the same way package managers do, we would have to keep spending a considerable amount of time and resources. So, the decision was made that the best route was to statically compile latest version of the dependencies into Aircrack-ng.

Basically, all the dependencies needed for each binary are built into each of them and what that means practically is that you can take the executable and just copy and paste it on another distro, no matter what packages are installed on that distro and how old or outdated it is, it would just work.

It wasn't an easy feat.

There is a drawback, the binaries are larger than if they were coming from the distribution itself (or if you were compiling them yourself with the default options), because they are dynamically linked to its dependencies, which are sometimes shared with other software.

There is also a huge advantage. As mentioned above, we can support multiple OS and multiple versions of each of these OS easily and  as a bonus, you will always have the latest version of the dependencies which are, most of the time, newer than what your distribution is providing. Added bonus: it sometimes fix bugs found in the library available in the distro.

 

Repositories

Maintaining repositories to distribute the packages was another issue, we could have gotten an Ubuntu PPA repository, our own repository for Debian, the different derivatives of SuSe and RedHat but maintaining repositories for different distributions is a time and resources consuming task. So, we went with PackageCloud.io to handle it.

They provides instructions on how to add each of the repositories, either manually or automatically via their script. Afterward, it is just a matter of installing or updating Aircrack-ng via your package manager.

As mentioned, two flavors are available:
  • release: Any release published on our website, starting from this release, 1.4
  • git: for the most adventurous, built from each commit in our GitHub repository, with the latest features and bug fixes. While our repository is fairly stable, it may sometimes have bugs
Providing packages also means our package is a drop-in replacement for the existing Aircrack-ng package available from your distro and it will still be working with any package that requires it as a dependency.

Windows

Windows doesn't have any package manager we can leverage. Development binaries for Windows, built from our GitHub repository, are available on AppVeyor. On that link, select the last target, pkg, then click on the Artifacts tab.

Sauce

To make our life easier so we can focus on the development, it is, like all the CI/CD, automated thanks to our buildbots. The magic sauce is in packages.yaml in build/pipelines.


Finally


If your distribution is providing an up to date package of Aircrack-ng, we recommend to use it instead of our packages. That is, unless you are using the git packages.

If your distribution isn't in the list of supported ones but uses DEB or RPM packages, you can try overriding the distribution in PackageCloud installation script

If you have any question about it, please head to our forum.