As a developer who gets paid to work on improving ARM support in various open source projects, including the Linux kernel, I am used to things like cross compiling, accessing development boards over serial wires and other stuff that is quite common in the embedded world. However, as a LEG engineer, I actually work on systems that are much more powerful, and involve firmware layers and other system software components that are typically associated with a desktop or server PC, and not with a NAS box or a mobile phone. So why am I still using my x86 box to do the actual work?
The reality is that the desktop PC market is not a very appealing market to try and conquer with a new CPU architecture, and conquering the appealing ones is already proving to be hard work. So if the ARM development community wants ARM based workstations, it appears we will have to take matters into our own hands.
Please, do try this at home!
Due to my involvement with the UEFI port of the Celloboard (which is due to ship any day now), I was given a AMD Overdrive B1 development board last year, which is based on the same AMD Seattle SoC (aka Opteron A1100), but has a ATX form factor, a standard ATX power supply connector, two [working] PCIe slots, and onboard SATA (14 ports!) and networking, all of which are fully supported in the upstream Linux kernel.
So what would I need to turn this into a desktop system that is good enough for my day to day work?
The fan
The most annoying thing about switching from embedded/mobile dev boards to ‘server’ dev boards is the bloody fans!! To anyone reading this who is in charge of putting together such systems: a development board is quite likely to spend most of its lifetime within earshot of a developer, rather than in a data center rack. So could we please have quieter fans?!?
</rant>
OK, so the first thing I did was replace the fan with a less noisy one. Do note that the AMD Seattle SoC uses a custom design for the heatsink, so this replacement fan will fit Cello and Overdrive, but not other arm64 based dev boards.
The case
Due to the ATX form factor and ATX power supply connector, there are lots of nice cases to choose from. I chose the smallest one I could find that would still fit a full size ATX board, so I ended up with the Antec Minuet 350, which takes low-profile PCIe cards.
The peripherals
My Overdrive board came with RAM installed, and has networking and SATA built in. So what’s lacking in terms of connectivity for use as a workstation is graphics and USB.
The AMD Seattle SoC has one peculiarity compared to x86 that complicates matters a little here: the RAM is mapped at physical address 0x80_0000_0000 (yes, that’s 9 zeroes), which means there is no 32-bit addressable RAM for PCI DMA. This is something that we could work around using the SMMU (IOMMU in ARM speak), but this is currently not implemented in the UEFI firmware or the Linux kernel, and so we need PCI peripherals that are capable of 64-bit DMA addressing.
For USB, I ended up selecting the SilverStone SST-EC04-P, which ships with a low-profile bracket, and has an onboard connector that can be used to wire up the two USB ports on the front of the case.
For graphics, I looked for a passively cooled, not too recent (for driver support, see below) card with HDMI output, and ended up with the Geforce 210 based MSI N-210, which has a nice, big heatsink (and no fan) and ships with a low profile bracket as well.
Kernel support
The lack of 32-bit addressable RAM for PCI DMA breaks assumptions in quite a few kernel drivers. For the Realtek 8169 Gig-E chip on the CelloBoard, we upstreamed patches that enable 64-bit DMA addressing by default on PCIe versions of the chip.
Much in the same way, I had to fix the nouveau and the ALSA drivers for the Geforce 210. Note that the proprietary, closed source NVidia driver is only available for x86, and so cards that are well supported by the open nouveau driver are strongly preferred.
All these patches have been in mainline since v4.10.
Userland support
‘Userland’ is the word kernel hackers use to refer to everything that executes outside of the kernel. My userland of choice is the Gnome3 desktop, which works quite well on the upcoming Ubuntu version (17.04), but older releases suffer from an annoying SpiderMonkey bug, which is caused by the incorrect assumption on the part of the SpiderMonkey developers that pointers never use more than 47 bits, and that bits 48 and up can be used for whatever you like, as long as you clear them again when trying to dereference the pointer value.
However, the arm64 kernel can be configured to use only 39 bits for virtual addressing, which still leaves plenty of address space and sidesteps the SpiderMonkey bug. This way, older Ubuntu versions are usable as well. I am currently using 16.10.
Firmware support
Now this is where it gets interesting. And I am not just saying that because I work on firmware.
So far, we have enabled everything we need to run an ordinary Ubuntu desktop environment on the Overdrive board. But interacting with the UEFI firmware at boot time still requires a serial cable, and a PC on the other end.
The problem here is driver support. Unlike SATA and USB, which are usually supported by class drivers, network interfaces and graphics cards require UEFI drivers that are specific to the particular chip. For the network interface on my Overdrive, this is a solved problem, given that it is integrated with the SoC, and supported by a driver that AMD have contributed. However, for plug-in peripherals such as my Geforce 210, the driver problem is usually addressed by putting a driver in a so-called option ROM on the card, and exposing it to the firmware in a way that is standardized by the PCI spec.
EFI Byte Code
Back when Intel was conquering the world with Itanium, they foresaw the problem that is affecting the ARM ecosystem today: an x86 driver can only run on a x86 CPU, and an ARM driver can only run on an ARM CPU, but option ROMs do not have unlimited space. Intel solved this by inventing an instruction set called EBC (for EFI byte code), and adding an interpreter for it to the UEFI reference code base. In theory, this allows expansion card vendors to recompile their code using an EBC compiler, and flash it into the option ROMs, so that the cards can be used on any architecture.
In reality, though, EBC is not widely used, is not promoted anymore by Intel (now that Itanium is dead), and even if expansion card vendors could get their hands on the compiler (which is not offered for sale anymore), recompiling source code that is riddled with x86 architecture (or rather, PC platform) based assumptions is not guaranteed to produce a driver that works on other architectures, especially ones with weakly ordered memory that does not start at address 0x0. For graphics cards in particular, the situation is even worse, given that many cards ship with a legacy VBIOS ROM (which requires legacy PC-BIOS compatibility in the x86 UEFI firmware) rather than a UEFI driver built for x86.
And indeed, it turned out that my nice low profile passively cooled Geforce 210 card did not have a UEFI driver in the option ROM, but only a legacy VBIOS driver.
X86 emulation in UEFI
Fortunately, I have another GeForce 210 card that does have a UEFI driver in its option ROM. So I dumped the ROM and extracted the driver, only to find out – not entirely unexpectedly, given the above – that it was a x86 driver, not a EBC driver, and so it is not supported on UEFI for 64-bit ARM.
So when Alexander Graf (of Suse) approached me at Linaro Connect two weeks ago, to discuss the possibilities of running x86 option ROMs on an emulator inside UEFI, I was skeptical at first, but after some more thought and discussion, I decided it was worth a try. Over the past ten days, we have collaborated online, and managed to implement an X86 emulator inside UEFI, based on an old version of QEMU (which is still LGPL licensed) combined with the more recent AArch64 support (whose copyright is owned by HiSilicon)
While this does not solve the problem of crappy drivers that make PC platform based assumptions, it works quite reliably for some network drivers we have tried, and even performs a lot better than EBC (which is a straight interpreter rather than a JIT).
And of course, it allows me to boot my Overdrive B1 in graphical mode.