May 3, 2020

Installing solyste on a Turris Omnia

Four years ago, CZ.NIC launched a campaign on Indiegogo to create what they called "The open-source center of your home". The idea sounded kinda interesting to me. I was not really thrilled about the software, I was excited about the hardware. As you can see, the listed features were quite good. Therefore, I decided to pre-order a board. My primary idea was to port my personal Linux distribution called solyste, to it.

Unfortunately, embedded hardware often use custom U-Boot and also heavily patched Linux kernel. No surprise, it was the same here. I dislike having to build custom kernels from obscure and sometimes unmaintained trees, because it is not a long-term solution. What can we do when manufacturers decide to (silently) drop the hardware ? We become stuck with old, buggy and flawed software. For those reasons, I had to wait for patches in upstream / mainline Linux repo.
As I write these lines, front LEDs are not really supported by 5.4.x and we can't control them. It seems to work with OpenWrt and LEDE, though (I didn't test by myself).

Upstreaming support took a long time, but eventually it came later. Even if a few bits are still missing, I knew I could try to boot on the board. Due to the Covid-19 crisis, I had to stay at home like many people across the globe. After all those years, it was the occasion to work on solyste and Omnia, in my spare time.

Kernel

Booting a kernel on the default system was not really hard. I had to compile it with mvebu_v7_defconfig and it went surprisingly fine. The device tree file is mandatory too. The kernel (without loadable modules) will be meticulously tuned later. Make sure you can access U-Boot prompt with serial connection preliminarily. Then, using TFTP I could "send" it to the board, with the following commands :

tftpboot 0x01000000 zImage
tftpboot 0x02000000 armada-385-turris-omnia.dtb
bootz 0x01000000 - 0x02000000

You only need to make sure that your kernel supports Btrfs, to be able to mount the first partition (and to avoid a kernel panic). That's what the vanilla system uses.

Userland

Few years ago, I embraced musl libc for my own projects and I never looked back. Rich, by any chance, if you read these lines, you are truly awesome.
The repository musl-cross-make provides a makefile to build the musl cross-compiler. That's what I use in solyste (before, I was also building the cross-compiler with my own scripts). Boosted by the previous success, I wrote the proper code to cross-compile the softwares for armv7 architecture.

Embedded MMC

It was is the scary part. Managing the eMMC is sometimes extremely difficult. An innocent tiny mistake can brick your device for good. What a harsh reward. Turris Omnia has 8Go NAND flash. It would be very sad to not use all that space. Anyway, at the moment we simply CAN'T boot from both USB3, because of U-Boot version. mSATA SSD or mPCIe to USB boot is possible, but it requires you to buy something else, one more time.

Luckily, in addition to the default operating system, a rescue system is installed in SPI flash. We can exploit it to install solyste. To switch to rescue, follow this page. It is based on a busybox and a few other tools. There is a script called rescue.sh in /bin directory to reflash the router, as described on this page. We won't use it because it supports btrfs only.

/ # ls bin/
[           date        head        mktemp      rmdir       tr
[[          dd          hostid      mount       sed         traceroute
ash         df          hostname    mv          seq         true
awk         dirname     id          nc          setsid      umount
basename    dmesg       ipcalc      netmsg      sh          uname
bunzip2     du          kill        nice        sha256sum   uniq
busybox     echo        killall     nslookup    sleep       unxz
bzcat       egrep       less        pgrep       sort        uptime
bzip2       expr        ln          pidof       strings     vi
cat         false       lock        ping        stty        wc
chgrp       fgrep       logger      ping6       sync        wget
chmod       find        ls          printf      tail        which
chown       free        lspci       ps          tar         xargs
clear       fsync       lsusb       pwd         tee         xz
cmp         ftpget      md5sum      readlink    test        yes
cp          grep        mkdir       rescue.sh   tftp        zcat
cttyhack    gunzip      mkfifo      reset       time
cut         gzip        mknod       rm          touch

fdisk binary is present in /sbin but we still lack e2fsprogs to create ext4 filesystem. ext4 is perfect for me (my systems are read-only anyway). Not a problem here as e2fsprogs is included in base group in solyste. That means I can easily build a static mke2fs binary. That's neat, isn't it ?

$ file mke2fs
mke2fs: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, with debug_info, not stripped

That mke2fs binary could be also very useful in the future. I decided to copy it in /sbin. Maybe we should ask CZ.NIC to do it, but I feel it would be too much hassle, as they are focusing on Btrfs. Now, we create the filesystem :

/ # fdisk /dev/mmcblk0
/ # mke2fs -t ext4 /dev/mmcblk0p1

We are (very) good so far.

solyste is finally ready to be deployed in eMMC. Let's extract the archive stored on an external USB drive, with tar :

/ # mount -t ext4 /dev/mmcblk0p1 /mnt
/ # tar xvf SOLYSTE-OMNIA.tar.gz -C /mnt
/ # chown -R 0:0 /mnt/*
/ # sync
/ # umount /mnt

Press the reset button to reboot. Before we edit U-Boot environment variables, we verify if the ext4 partition is correctly listed :

=> ext4ls mmc 0:1 /
<DIR>       4096 .
<DIR>       4096 ..
<DIR>      16384 lost+found
<DIR>       4096 bin
<DIR>          0 boot
<DIR>          0 dev
<DIR>          0 etc
<DIR>          0 mnt
<DIR>          0 proc
<DIR>          0 root
<DIR>          0 sys
<DIR>          0 tmp
<DIR>          0 var

Hum, it does not seem to be very good… The directories were simply empty and the files inside them were missing :

=> ext4ls mmc 0:1 /boot
** Can not find directory. **

At this point, I thought there was a problem with the eMMC, even if I didn't do something wrong. But it was still correctly detected :

=> mmcinfo
Device: mv_sdh
Manufacturer ID: 90
OEM: 14a
Name: H8G4a
Tran Speed: 52000000
Rd Block Len: 512
MMC version 4.0
High Capacity: Yes
Capacity: 7.3 GiB
Bus Width: 8-bit
Erase Group Size: 512 KiB

Thanks to TFTP, I could boot on my kernel and the main partition was totally fine. Something was bad with U-Boot. Maybe the ext4 implementation is too simple.
In embedded world, filesystems in 64-bit mode can lead to erratic behavior. I decided to boot once again in rescue mode to create the ext4 filesystem, in 32-bit this time. Do you remember when I told you earlier that mke2fs could be necessary ? So, here we go again :

/ # mke2fs -t ext4 -O ^64bit /dev/mmcblk0p1

Then, after extracting the archive and rebooting, the ext4ls command was more realistic :

=> ext4ls mmc 0:1 /
<DIR>       4096 .
<DIR>       4096 ..
<DIR>      16384 lost+found
<DIR>       4096 bin
<DIR>       4096 boot
<DIR>       4096 dev
<DIR>       4096 etc
<DIR>       4096 mnt
<DIR>       4096 proc
<DIR>       4096 root
<DIR>       4096 sys
<DIR>       4096 tmp
<DIR>       4096 var
=> ext4ls mmc 0:1 /boot
<DIR>       4096 .
<DIR>       4096 ..
           17506 armada-385-turris-omnia.dtb
         3793992 zImage

VICTORY ! As the main ext4 filesystem is completely detected by U-Boot, we can now setup the boot itself.

U-Boot variables

The default environment variables can be listed with printenv. Here the default bootargs and mmcboot variables. We will modify them :

bootargs=earlyprintk console=ttyS0,115200 rootfstype=btrfs rootdelay=2 root=b301 rootflags=subvol=@,commit=5 rw
mmcboot=setenv bootargs "$bootargs cfg80211.freg=$regdomain"; btrload mmc 0 0x01000000 boot/zImage @; btrload mmc 0 0x02000000 boot/dtb @; bootz 0x01000000 - 0x02000000

First, we will replace bootargs with setenv. I tried the following, but I had many kernel panics :

setenv bootargs rootfstype=ext4 root=/dev/mmcblk0p1 rw console=ttyS0,115200

The kernel was booting too fast and the eMMC was not discovered in time. It's also quite common. Let's add rootdelay. Even default variable has it. I appended quiet too.

setenv bootargs rootfstype=ext4 root=/dev/mmcblk0p1 rootdelay=2 rw console=ttyS0,115200 quiet

mmcboot variable is similar to tftpboot command. Defining cfg80211.freg makes no sense here, because my board do not have WiFi. Secondly, btrload is for btrfs. The ext4 counterpart is ext4load. With those parameters, we can compose the variable :

setenv mmcboot "ext4load mmc 0:1 0x01000000 boot/zImage ; ext4load mmc 0:1 0x02000000 boot/armada-385-turris-omnia.dtb ; bootz 0x01000000 - 0x02000000"

Do not get fooled by my words, I had to try four or five times to find the correct value. Once bootargs and mmcboot variables are setup, we save them with saveenv command and… we are done ! You can either push the reset button on the back or directly launch run mmcboot command.

RAM issues

Usually, when the Omnia was unplugged from electrical source or after a reset, the DDR3 RAM will fail to initialize. Here the output :

Memory config in EEPROM: 0x01
ddr3_tip_pbs_rx failure CS #0
Title: I/F# , Tj, Calibration_n0, Calibration_p0, Calibration_n1, Calibration_p1, Calibration_n2, Calibration_p2,CS0 ,
VWTx, VWRx, WL_tot, WL_ADLL, WL_PH, RL_Tot, RL_ADLL, RL_PH, RL_Smp, Cen_tx, Cen_rx, Vref, DQVref,               PBSTx-Pad0,PBSTx-Pad1,PBSTx-Pad2,PBSTx-Pad3,PBSTx-Pad4,PBSTx-Pad5,PBSTx-Pad6,PBSTx-Pa
d7,PBSTx-Pad8,PBSTx-Pad9,PBSTx-Pad10,           PBSRx-Pad0,PBSRx-Pad1,PBSRx-Pad2,PBSRx-Pad3,PBSRx-Pad4,PBSRx-Pad5,PBSRx-Pad6,PBSRx-Pad7,PBSRx-Pad8,PBSRx-Pad9,PBSRx-Pad10,
Data: 0,72,19,15,19,15,20,20,CS0 ,
0,0,10,10,0,423,7,1,6,23,10,4,0,                63,63,63,63,31,31,63,63,63,63,63,               0,0,0,0,0,0,0,0,0,0,0,          0,0,0,0,0,0,0,0,0,0,0,
0,0,6,6,0,422,6,1,6,19,10,4,0,          63,63,63,63,31,31,63,63,63,63,63,               0,0,0,0,0,0,0,0,0,0,0,          0,4,2,8,0,0,3,8,8,10,0,
0,0,6,6,0,421,5,1,6,19,10,4,0,          63,63,63,63,31,31,63,63,63,63,63,               0,0,0,0,0,0,0,0,0,0,0,          2,0,0,6,0,0,0,2,6,3,0,
0,0,10,10,0,423,7,1,6,23,10,4,0,                63,63,63,63,31,31,63,63,63,63,63,               0,0,0,0,0,0,0,0,0,0,0,          1,2,1,5,0,0,2,0,4,1,0,

Run_alg: tuning failed 0
DDR3 run algorithm - FAILED 0x1
DDR3 Training Sequence - FAILED

I share it to you in case you noticed it on your board. I do not know if my unit is defective or not. I just hope it will not happen when it's deployed, in a remote location. I would be unable to access it.

solyste on Turris Omnia

After the previous steps, solyste is now perfectly booting. The board is ready to be used. Needless to say, I'm quite happy. Maybe it's time to brag a little bit…

Kernel :

solyste armv7l on 5.4.33-SOLYSTE (/dev/ttyS0)

solyste-omnia login: root
Password:

Welcome to your solyste box!

[solyste-omnia] /root # cat /proc/version
Linux version 5.4.33-SOLYSTE (ypnose@solyste) (gcc version 9.2.0 (GCC)) #1 SMP Sat May 2 13:46:17 CEST 2020
[solyste-omnia] /root # uname -a
Linux solyste-omnia 5.4.33-SOLYSTE #1 SMP Sat May 2 13:46:17 CEST 2020 armv7l GNU/Linux
[solyste-omnia] /root # du -h /boot/zImage
3.6M    /boot/zImage
[solyste-omnia] /root # zcat /proc/config.gz | grep CONFIG_MODULES
# CONFIG_MODULES is not set

cpuinfo :

[solyste-omnia] /root # cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 1 (v7l)
BogoMIPS        : 1600.00
Features        : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x4
CPU part        : 0xc09
CPU revision    : 1

processor       : 1
model name      : ARMv7 Processor rev 1 (v7l)
BogoMIPS        : 1600.00
Features        : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x4
CPU part        : 0xc09
CPU revision    : 1

Hardware        : Marvell Armada 380/385 (Device Tree)
Revision        : 0000
Serial          : 0000000000000000

Daemons / services :

[solyste-omnia] /root # perpls -G
[+ +++ +++]  crond          uptime: 107s/107s  pids: 147/146
[+ +++ ---]  getty-ttyS0    uptime: 107s/-s  pids: 145/-
[+ +++ ---]  klogd          uptime: 107s/-s  pids: 155/-
[- --- ---]  monit          (service not activated)
[- --- ---]  openntpd       (service not activated)
[+ +++ +++]  sshd           uptime: 107s/107s  pids: 152/150
[+ +++ ---]  syslogd        uptime: 107s/-s  pids: 144/-
[- --- ---]  unbound        (service not activated)
[- --- ---]  vnstatd        (service not activated)
[+ +++ ---]  watchdog       uptime: 107s/-s  pids: 148/-
[solyste-omnia] /root # pstree
init---rc.intro---perpboot-+-perpd-+-crond
                           |       |-klogd
                           |       |-mksh---pstree
                           |       |-sshd
                           |       |-syslogd
                           |       |-2*[tinylog]
                           |       `-watchdog
                           `-tinylog
[solyste-omnia] /root # print $KSH_VERSION
@(#)MIRBSD KSH R59 2020/04/14

The end

Even if I plan to write an article explaining why solyste is different and how it works, you can visit my "Projects" page in the meantime. Have a nice day and take care of you.