Jonas Schäfer

ROCKPro64 as Multimedia System and NAS: A long journey

Abstract

The ROCKPro64 is a single-board computer based on the RK3399 hexacore ARM system-on-chip. It supports PCIex4, hardware decoding of many modern video codecs and gigabit ethernet. In this article, I describe my journey to convert the device into a NAS-multimedia-system hybrid, solving challenges such as DRM incompatibilities and performance issues.

TL;DR:

  • Use a kernel from debian-backports
  • If you do any kind of realtime/media playback/high throughput: Disable the deepest sleep state and potentially frequency scaling.
  • If you want to use mpd: patch it against the unaligned access or get a more recent release.
  • If you want to use kodi with libwidevinecdm: install an arm64 kernel with armhf userland and patch your glibc.

Introduction

When I set up our home theater, I initially wanted it to be able to work without any "real" computers. I didn't want to have a plug a laptop in some HDMI cable in order to watch a movie or a series.

Unfortunately, it didn't work out that way. When I installed kodi on the Raspberry Pi 3B, I quickly realized that the digital rights management (DRM) logic required by some streaming services is too heavy for the poor little Pi. 1080p content would not play fluently or overheat the device within minutes (ever heard how short the audio buffer of the Pi is? that's how you'll find out).

So in the past months, full-blown x86 laptops have served as input to the HDMI channel. That is neither efficient nor convenient, given that many modern SoCs come with hardware-enabled decoding of common video codecs (even though I doubt that this can be usefully used for DRM content) and that ARM platforms may be much less power-consuming than x86 systems. And while it's possible to run Kodi on the laptop to gain its remote-controllability and integration with infrared remotes for instance (and thanks to the IR forwarder I built earlier, I can actually send IR commands via the network to the laptop), even the bi-weekly fights with xscreensaver or DPMS over when it's appropriate to turn off an output are enough.

Then someone in some chat mentioned the ROCKPro64. The ROCKPro64 is a single-board computer based on the Rockchip RK3399 system-on-chip. It features an ARM hexacore, a Mali GPU and a Hantro video processing unit. It can decode VP8, VP9 and H.264 in hardware, has gigabit ethernet and a PCIe x4 slot.

Previously, my network attached storage needs are fulfilled by a 2015 intel box which I bought in a haste when my desktop PC died (I needed somewhere to plug the disks to get to the data). It consumes 35 W in idle (with disks in standby!), so it goes without saying that it's not always online. Needless to say, that is bad for backups.

Looking at the hardware specs, the ROCKPro64 looked like it could fullfil both needs: With its hardware decoding and the hexacore (which can be clocked up to 1.8 GHz on the more powerful cores) it should easily be able to render even full HD DRM content. The PCIe x4 slot should be sufficient to connect a SATA controller which can serve the RAID 1 from the NAS.

This way, I could get rid of the Intel box, have an always-on NAS, and a more convenient home theater. Splendid, right?

Well, there are some pitfalls.

Note

Before we get into the nitty-gritty details, one word of warning: I will generally not repeat things you can find elsewhere here. In particular, I'll just write "apply this patch and rebuild" instead of giving you a step-by-step guide on how to actually do that. The reason is that this article contains enough information which will (hopefully!) be out of date in a year or two, even without me providing step-by-step guides for processes better documented elsewhere.

Shipping and other meta

The first thing which annoyed me was that I only found out after the fact that PINE64 ships from Asia. I don't like the thought of having that piece of hardware express-shipped via airplane. If I had known earlier, I would've gone with a local retailer, even though that would've meant a 20%+ markup on the price.

Other than that, the transaction and shipping went just fine. They don't seem to handle VAT correctly, so you will have fun with customs, though.

Installing Debian

As I mentioned, I want to do more with this than just media. Otherwise, I would probably have gone for LibreELEC or a similar Just-enough OS (JeOS) for this purpose. Hence, I will be using Debian, my favourite OS and the same system I use on all my other computers (except smartphones...).

There are ready-made installer images for the ROCKPro64 available right from Debian, which is nice.

At first, I was confused about how to boot this; I am used to images which contain an OS then self-inflate to the entire medium (like for the Raspberry Pi). I did not buy the eMMC module, so the only thing the board would boot off is the micro SD slot. So I had to boot the installer off the medium I was going to install on (which is actually possible, because the installer loads itself into memory on boot).

It took me a few attempts to get a bootable arm64 Debian system, however. To get a bootable system, I:

  • Did not delete the installer partition (that turned out to be irrelevant, but also useful because it offers a way to boot into a shell when things don't work)
  • Created a 2 GiB ext2 /boot with bootable flag
  • Created a 25 GiB ext4 / without any special flags.

I did not enable swap (also probably irrelevant) for the sake of the SD card.

There are a few peculiarities about this:

  • This is not a UEFI system, even though some pages seem to claim it is. There were no efivars in sysfs or procfs, so even if U-Boot (which is the bootloader which they ship in the SPI flash) is capable of providing an UEFI environment, it somehow doesn't provide it to the Debian installer.
  • U-Boot is massively underdocumented. We'll get to the challenges related to U-Boot later, when we have to re-install the system while bypassing some of the debian-installer steps.
  • U-Boot looks for boot.scr script files (or, as we'll see later, extlinux/extlinux.conf). The first of which it finds, it'll execute and boot. In the standard deployment of U-Boot on the ROCKPro64, it only searches on the eMMC and on the μSD.

Note

In between, I actually re-flashed the SPI because I thought some intermittent issues I had were related to an outdated U-Boot version. An image which flashes U-Boot into the SPI flash when booted can be found on GitHub.

The first thing I tested was obviously Kodi. Kodi and DRM-protected content. Which did not work, because libwidevinecdm, the library handling the DRM-protection, is not available for arm64.

Hence, I needed an armhf (32-bit) userland, just like on the Raspberry Pi. Now Debian supports something called MultiArch, where you can have things from different CPU architectures installed simultaneously.

Unfortunately, that didn't work out for Kodi, because Kodi depends on python3 (not just libpython3-*) and that binary and executable package is not multiarch capable.

Installing Debian with arm64 kernel and armhf userland

Now for the fun question: how do you install Debian with arm64 kernel but armhf userland?

Note

What this article describes is how I did it, not how I would do it again. The easiest way is probably using debootstrap and qemu on another system in order to generate a root filesystem you can then just dd onto the SD card. I was not in the mood of wrangling with qemu-user, even though the internet™ claims it's easy enough. I preferred to wrangle with a system I (thought I) understood, which would be the Debian installer (henceforth abbreviated as d-i, because I can't be bothered to type out "Debian installer" all the time).

What I did was this:

  • Boot the installer and go through the steps until it starts with "Installing the base system". This is where d-i calls the debootstrap utility.
  • Enter a shell using Ctrl+Alt+F2 and hitting enter.
  • Use ps and cat /proc/../cmdline to find the complete command line of the debootstrap call.
  • Kill debootstrap. Switch back to the main TTY (Ctrl+Alt+F1) to confirm that d-i noticed that debootstrap is gone. If it has not, also kill the parent of debootstrap. Iterate until an error message pops up.
  • Clean out the /target directory (but be careful not to delete mountpoints).
  • Call debootstrap again, but with --arch=armhf
  • Wait for debootstrap to finish

Now d-i is obviously a bit unhappy, but you can skip past that unhappiness. You'll now have to go through the steps manually (it will throw you back into the list of steps all the time). It will sometimes try to do the base system installation again, but notice that the chroot isn't empty. You must then refuse its question to continue, at which point it'll continue with the step you actually asked for. This may not work for the bootloader installation.

Once the installation is over, you enter your shell again. You need to chroot into the /target (make sure that /proc, /sys and /dev are bind-mounted!) and then run dpkg --add-architecture=arm64, followed by apt-get update and apt-get install linux-image-arm64 linux-base, to get the kernel installed.

Edit /etc/kernel-img.conf and add image_dest = /boot. Find the kernel version you just installed by looking into /boot and run linux-update-symlinks install $version $image_path, where you replace $version with that version and $image_path with the absolute filename of the vmlinuz file associated with that kernel.

You should now have /boot/vmlinuz as a symlink pointing at the kernel image.

Create the /boot/extlinux directory and inside that an extlinux.conf file with the following contents:

default l0
menu title Debian
prompt 0
timeout 5

label l0
menu label Debian (armhf)
linux /vmlinuz
initrd /initrd.img
fdt /dtb
append root=UUID="9d348833-9ec0-4bc9-b2b4-520fc4076740" apparmor=0

Make sure to substitute the UUID in there with the UUID of your SD card partition. You can find it by running e.g. blkid /dev/mmcblk1*.

Note

Normally, there is the flash-kernel package which is supposed to create appropriate configuration in /boot for U-Boot to understand. In my mixed arm64/armhf setup, I was unable to get it to work, though. It would generate boot.scr files which U-Boot then refused to run (SCRIPT FAILED).

Note

Nontheless, you should definitely install flash-kernel if d-i did not install it for you. It is required in order to update the devicetrees in /boot.

You can now reboot the system and should hopefully end up in your freshly installed Debian.

Setting up the Multimedia System

The kernel version in the stable archive is currently 5.10.x, and many features are not enabled during build time. That has been fixed in the more recent kernel versions, so you'll likely have to install a kernel from Debian backports if you want to use any of the more advanced features (such as video acceleration). I hear that 5.17.x is the least you want.

Installing the Music Player Daemon (mpd)

You can install and configure mpd as usual. However, because of a bug which has been fixed in mpd upstream, it is possible that you may not be able to index your library. The symptom is that mpd gets killed with the SIGBUS signal, triggered by an unaligned memory access.

Your options are:

  • Build your own patched mpd package
  • Check if by now 0.23.7 has reached the Debian repository or complain to the maintainer if not

As I was already deep in debugging this, I chose the first path (built my own package). The patch is simple:

diff --git a/src/tag/ApeLoader.cxx b/src/tag/ApeLoader.cxx
index 596ac820..3ede6a1d 100644
--- a/src/tag/ApeLoader.cxx
+++ b/src/tag/ApeLoader.cxx
@@ -72,11 +72,15 @@ try {
    /* read tags */
    unsigned n = FromLE32(footer.count);
    const char *p = buffer.get();
+   uint32_t u32buffer;
    while (n-- && remaining > 10) {
-           size_t size = FromLE32(*(const uint32_t *)p);
+           // fix alignment before dereferencing as uint32_t
+           memcpy(&u32buffer, p, sizeof(uint32_t));
+           size_t size = FromLE32(u32buffer);
            p += 4;
            remaining -= 4;
-           unsigned long flags = FromLE32(*(const uint32_t *)p);
+           memcpy(&u32buffer, p, sizeof(uint32_t));
+           unsigned long flags = FromLE32(u32buffer);
            p += 4;
            remaining -= 4;

You can just add that to the patch series of the mpd package and rebuild it locally.

Because it is already fixed upstream, I anticipate the fix becoming available soon-ish in Debian. If I get around to reverting the mpd version to one from the archives, I may even file a bug to make the maintainers aware.

HDMI issues and Enabling S/PDIF output

Now this is a nasty one. The board features multiple audio outputs. There are at least:

  • I2S pins exposed on the Pi-compatible header
  • HDMI (also via I2S internally)
  • Analog (via I2S)
  • S/PDIF

Unfortunately, there are only 3 direct memory access (DMA) blocks available for these. So you will have to disable one of them in order to make S/PDIF work.

If you have an Audio/Video Receiver (AVR) device, you may want to use HDMI directly even for audio. Unfortunately, it seems impossible to output audio with the ROCKPro64 if no display is connected and turned on in the HDMI chain. I mean this quite literally; there was simply no audio output (without any error indication) if I connected the ROCKPro64 to my Denon AVR until I also connected a display on the other end of the Denon AVR and turned it on (this took me a while to figure out…).

Note

I am really annoyed by that, so if you know a fix, let me know. I am tempted to pull HDMI pin 19 high to see if that helps, but haven't gathered the motivation yet to hack a cable. See the footer below the post for contact info.

So if you want headless music playback of considerable quality with just the things available on the board, you'll have to enable S/PDIF.

The enablement of S/PDIF is controlled via the so-called devicetree. You can read more about devicetrees on the Linux and the Devicetree page of the kernel documentation or on devicetree.org.

The current devicetree used by your board is available in compiled form at /boot/dtb (TODO: verify). To edit it, we first need to decompile it using dtc -I dtb -O dts < /boot/dtb > ~/full.dts. That will emit a few warnings, but that's ok (I hope).

Devicetree source files are organized in blocks (much like C code), delimited by curly braces.

We now have to find the blocks which match spdif. Use text search for that. You should find a status = "disabled" in those blocks. Replace that with status = "okay". Do not replace any other status = "disabled" entries!

Now because of the DMA issue mentioned above, you'll have to pick one of the i2s blocks (search for i2s@ to find them) and change the status = "okay" in it to status = "disabled". I used the first one, which corresponds (I think) to the I2S pin headers I don't plan to use anyway.

If there is no status = "okay" in the block, you may add a status = "disabled".

The best way to install your new device tree is to compile it and put it in /etc/flash-kernel/dtbs/rk3399-rockpro64.dtb. This can be done in a single step:

dtc -I dts -O dtb < full.dts > /etc/flash-kernel/dtbs/rk3399-rockpro64.dtb

Followed by:

cp --backup /etc/flash-kernel/dtbs/rk3399-rockpro64.dtb /boot/dtbs/$(uname -r)/rockchip/rk3399-rockpro64.dtb

(By using cp instead of flash-kernel, you have a .bak file to recover the original devicetree in case you made a mistake which renders your system unbootable.)

By putting the file in /etc/flash-kernel/dtbs, you ensure that you will get your modified device tree on subsequent kernel updates. The flipside of this method is that you will not benefit from a kernel update which ships with an improved device tree. As things like the cooling fan levels are also configured inside the device tree, you may want to diff the upstream device tree against your local device tree regularly. You can find the original dtb file in /usr/lib/linux-image-$(uname -r)/rockchip/rk3399-rockpro64.dtb.

After a reboot, you should see the SPDIF card in aplay -l.

Note

I eventually also disabled the other (analog) I2S output.

Installing kodi

You should probably install kodi from backports straight away. If you then attempt to play DRM-protected content, you'll find that kodi crashes.

The reason is that libwidevinecdm uses a new feature called RELR, which needs support by the runtime dynamic linker used. The linker is provided by the glibc source package and unfortunately, officially support for RELR is not even in upstream yet.

In order to fix that I took the patch against libc from LibreELEC, because it seemed smaller and easier to backport. Unfortunately, it did not apply cleanly to the libc from debian. This is the patch backported to the current glibc 2.31-13+deb11u4.

Building libc took several hours on the ROCKPro64 and it is a good opportunity to test your cooling rig :-).

Once you installed the patched libc, and possibly rebooted, it is possible to play DRM-protected content with Kodi. At least with Netflix, however and in contrast to the Raspberry Pi, 1080p content is not delivered to Kodi. I'll still have to figure that one out.

Setting up the NAS system

This is pretty straightforward. I just got a PCIe x4 SATA controller (no raid functionality) plugged it in and it worked out of the box. The one I have is based around the JMicron JMB585 chipset, if that matters to anyone.

There was one catch, though (fixed below): When using rsync to copy data off the disks attached to the ROCKPro64, it would only reach 65 MiB/s at most. Given that the disks are spinning disks with more than 170 MiB/s sequential read performance and in a RAID1 configuration, this was certainly unexpected.

Fixing the Throughput

There were two issues with the build at this point:

  1. Music played back with mpd had gaps.
  2. rsync did not deliver full gigabit line rate, even through rsyncd (i.e. without ssh).

I debugged both issues at length. In fact, I spent several days, including the use of an oscilloscope, just to understand what was going on with the music playback. I may make a detailed post debugging this later on, because it was a pretty weird journey.

Either way, both of these issues share a common root cause: The cluster-sleep cpu idle state.

I don't know for sure what that is about, but I think it boils down to this: When a CPU has nothing to do, in addition to reducing the clock frequency, it may also be put in a sleep state, which reduces power consumption at the cost of latency. A CPU may have multiple of those for different levels of "idleness" (however that is measured).

As it turns out, the deepest sleep state on the ROCKPro64 (cluster-sleep) is too deep.

Apparently, waking up from that state takes long enough that it both:

  1. Prevents mpd (and other audio things, actually) from providing data to the sound driver in time, or the sound driver from sending that data to the hardware in time.
  2. Introduces sufficient latency inside rsync itself to limit the bandwidth.

To disable the sleep state (until the next reboot, anyway) as well as cpufreq scaling, run:

echo 1 | tee /sys/devices/system/cpu/cpu*/cpuidle/state2/disable
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

(Disabling only one of those did not help reliably.)

Now you may fear that this will make your board burn immediately, but in fact it did not have any influence on the CPU temperature I measured (though I still have to check the power consumption).

FWIW, the above change is also required to get reliable fluent media playback with kodi.

Bottom line

The ROCKPro64 is an affordable base for a power-efficient NAS system, if combined with a PCIe-to-SATA converter.

It is also suitable for music playback and playback of local media, but you'll probably have issues with streaming services due to resolution restrictions because of insufficient widevine certification levels.

There are issues, as detailed above, but they can be circumvented and will eventually be fixed upstream (except the power management thing I suppose).

In contrast to the Raspberry Pi and the surrounding ecosystem, the ROCKPro64 definitely has a worse "out of the box" experience at this point. However, if you put the work in, you are rewarded with a reliable and performant system.

I got the ROCKPro64 in July of this year and since then, it has served as NAS and music player mostly. Due to the lack of Full HD on Netflix, I haven't used it for kodi much and that use case is still served using x86 laptops.

It was still worth it, even if only because having reliable backups is really nice.

Happy hacking!