Author Topic: Instructions: How to manually upgrade Gemian to v4 (without wiping your data)  (Read 410 times)

shuntcap

  • Jr. Member
  • **
  • Posts: 79
    • View Profile
How to manually upgrade Cosmo Linux from v3 to v4 without wiping your data
One serious drawback to running Linux on the Cosmo is that with each new Linux release, Planet's upgrade method wipes your entire Linux partition.  For those who actually use Linux on the Cosmo, this is highly undesirable.  While previous upgrades (v1->v2 and v2->v3) could be performed manually through apt, v4 breaks this due to implementing a replacement Android device driver compatibilty layer.  However, is it still possible to to manually upgrade from v3 to v4!  This guide will show you how.

You will need:
1) TWRP installed on the Cosmo (https://support.planetcom.co.uk/index.php/Linux_for_Cosmo#TWRP_installation)
2) adb installed on a workstation
3) Enough hard drive space on your workstation to back up the full size of your Cosmo Linux partition
4) The Planet USB-C/Ethernet hub (Wi-Fi will be temporarily lost) and wired (not wireless) Internet connectivity in your home or office (https://store.planetcom.co.uk/collections/planet-accessories/products/gemini-usb-c-hub)
5) Your full concentration and steady nerves

Disclaimer: You proceed at your own risk.  This is a complicated manual upgrade and many things can go wrong if you are not careful.  I am not responsible, nor is Planet Computers, nor is OESF, nor is Adam Boardman, for anything that goes wrong with your equipment while following this guide.


Step 1: Back up your Linux partition
Stop!  Do NOT skip this step!  Before we do anything, back up your Linux partition just in case something goes wrong during the upgrade.  If you have nothing worth saving on it, don't even bother reading this guide.  You're better off just upgrading with Planet's method.

To back up your Linux partition onto your workstation, first boot TWRP.  On the Mount screen, tap Cancel.  Swipe right to allow modifications.  Plug a USB-C cable from your workstation into the Cosmo's left port.  Make sure you have adb installed on your workstation, then perform these commands:
Code: [Select]
adb pull /dev/block/by-name/linux cosmo_linux_partition_backup
adb pull /dev/block/by-name/DEBIAN_KDE cosmo_linux_boot_backup

This will take some time.  Your Linux partition, including personal data and customizations, will be backed up to a file on your workstation called cosmo_linux_partition_backup.  On a Linux workstation, if you wish, you can mount this file as a loopback device and view the contents:

Code: [Select]
sudo mkdir -p /mnt/cosmo_linux
sudo mount -o loop cosmo_linux_partition_backup /mnt/cosmo_linux
ls /mnt/cosmo_linux

You will see your Linux filesystem data.

The second file backed up is the Linux boot partition.  We back this up to be absolutely certain we can restore your entire working Linux setup as it is right now.  This ensures that we have the kernel version that matches your Linux installation.

See Step 6 for restoration instructions in case anything goes wrong during the upgrade.


Step 2: Set up Ethernet
During the upgrade process, we will lose both Wi-Fi connectivity and the X server graphical interface due to removing the Android device driver compatibility layer which has changed in v4.  These will be restored by the time we finish.  Since MediaTek only provides closed-source Android device drivers for their hardware platforms (beyond basic things like keyboard, LEDs, and LCD initialization), Gemian Linux has to rely on interfacing with Android libraries to provide accelerated graphics, sound, GPS, camera, Wi-Fi, and other critical device support.

We therefore need to rely on wired Ethernet connectivity instead, both for logging into the Cosmo remotely to issue commands, and for the Cosmo's Internet connection.  Planet's USB-C hub is required for this.  You will need a working LAN with a WAN gateway for Internet access.

We will need to determine what IP address the Cosmo receives by DHCP when the USB-C/Ethernet hub is inserted.  During the rest of the upgrade, we will need to remotely log into the Cosmo because the X server will crash.

There are two ways of doing this.  The first is the easiest.  The second is mostly informational.


Method 1
Plug the USB-C/Ethernet hub into the Cosmo's left USB port, then plug an Ethernet cable into the jack, ensuring that the other end is connected to a router with Internet access or a switch behind such a router.  You should see the activity LEDs on the hub's Ethernet jack light up if your cable is live.  Your router should be configured to serve DHCP addresses (most consumer and small office routers are configured this way by default).

We can also power the Cosmo through the hub, but there is a special plug-in order to follow or you won't have Ethernet.  First, disconnect everything.  Now connect the hub to the Cosmo.  Connect USB-C power to the hub.  Finally, connect the Ethernet cable.  (There is no RNDIS interface in this configuration.)  As a side note, this configuration should be of particular interest to anyone wishing to dock the Cosmo and use a monitor, mouse, keyboard, and even Ethernet, and still supply power to the Cosmo for unlimited runtime.  Once HDMI out is working over the right USB port, this will be possible!  I have done it under Android.

On the Cosmo, open a terminal emulator (xterm, for example).  Type this command:
Code: [Select]
ifconfig eth0 | grep inet
You should see output similar to this:
Code: [Select]
        inet 192.168.1.103  netmask 255.255.255.0  broadcast 192.168.1.255
Make note of the inet address (192.168.1.103 in this example; yours will differ), then from your workstation (which should be on the same LAN), type this:
Code: [Select]
ssh cosmo@192.168.1.103
You are now remotely logged into the Cosmo and can issue commands.  We will use this interface for the upgrade.

Now disable the Wi-Fi interface using NetworkManager or ConnMan.  I don't use either (they are completely disabled), so I won't tell you how to do that.

Our goal is to make sure Internet connectivity is achieved through the USB-C/Ethernet hub, not through Wi-Fi.  Wi-Fi will be lost during the upgrade (but restored at the end).  Test your Internet connection now, then proceed to Step 3.


Method 2
The Cosmo presents an RNDIS interface on the left USB port.  RNDIS is a way of emulating an Ethernet connection over a USB cable.  The Cosmo's IP address on this interface is always 10.15.19.82.  Plug a USB-C cable from your workstation into that left port.  On your workstation, log into the Cosmo:
Code: [Select]
ssh cosmo@10.15.19.82
Take down the Wi-Fi interface using whatever connection manager you have installed.  I do not use NetworkManager or ConnMan, so I just take it down by hand:
Code: [Select]
sudo ifconfig wlan0 down
Now we need to determine what IP address the Cosmo will pull for the USB-C/Ethernet hub.  I've been unable to get the USB-C/Ethernet hub to work in the right-hand USB port under Linux (although it does receive power), and likewise RNDIS only runs on the left-hand port.  So we cannot maintain both an RNDIS and a hub-based Ethernet connection at the same time.  We will need to swap the plugs.  First, execute this command:
Code: [Select]
sleep 30; ifconfig eth0 > eth0_info
Next, within 30 seconds (time it with a watch), remove your USB-C cable, then plug the USB-C/Ethernet hub into the left side.  Now plug in the Ethernet cable.  You should see the activity LEDs on the hub's Ethernet jack light up if your cable is live.  Wait for the 30 seconds to expire, during which the Cosmo should pull an IP address from your router via DHCP.  Remove the hub and plug the USB-C cable power back in.  Within a few seconds, your command line prompt should return (it froze because we unplugged your workstation's USB-C cable).  If the activity LEDs never lit, ensure the hub's Ethernet cable is connected to your router or network switch.

With the command line now active again, type this:
Code: [Select]
cat eth0_info
You will see the IP address of your Cosmo when the USB-C/Ethernet hub is connected; e.g., inet 192.168.1.103

Remove the USB-C cable (again) and insert the hub (again).  Wait for the activity lights to begin flashing, then use ssh to connect to that address:
Code: [Select]
ssh cosmo@192.168.1.103  # Your IP address will be different
Finally, test your Cosmo's Internet connectivity through any means you see fit (ping a remote host, direct a web browser to a remote site, etc.)  The point is to make sure your Internet connection is going through the USB-C/Ethernet hub, not through Wi-Fi.


Step 3: Upgrade Gemian and Android compatibility layer
With the backup out of the way (go back to Step 1 if you didn't do it), and a solid, wired Ethernet connection established, we can start the upgrade.

From now on, perform all commands from your workstation while remotely logged into the Cosmo through the USB-C/Ethernet hub.  This is very important because you will temporarily lose your X server during the upgrade and will not be able to enter commands directly on the Cosmo's console.

Edit the file /etc/apt/sources.list.d/gemian.list (use sudo and a text editor of your choice, such as vi, nano, joe, emacs, xedit, etc.) and make it match the following (change "gemian-buster" to "gemian-planet"):
Code: [Select]
deb https://gemian.thinkglobally.org/buster/ buster main
deb https://gemian-planet.thinkglobally.org/buster/ buster main

Likewise, edit the file /etc/apt/preferences.d/gemian.pref and change from:
Code: [Select]
gemian-buster.thinkglobally.orgto:
Code: [Select]
gemian-planet.thinkglobally.org
This points apt to the new v4 repository.  Now attempt to upgrade all packages:
Code: [Select]
sudo apt update
sudo apt upgrade

You'll see a long list of packages that will be upgraded, possibly followed by another long list of packages that will be downgraded.  This is expected.  Hit Enter to answer "Y" to the "Do you want to continue? [Y/n]" prompt.

Buried in the output, you will see a message about needing to reboot to use the new kernel. Do NOT reboot yet.

Next, do this:
Code: [Select]
sudo apt install media-hub
Then, this:
Code: [Select]
sudo apt install lxc-android
This will attempt to install gemian-system and lxc-android, but will fail after fetching the packages and doing a bit of setup.  This is expected.

In the next command, your Cosmo's graphical interface will freeze (if it hasn't already; sometimes it does earlier) because we will remove the Android device driver compatibility layer.  The X server relies on Hardware Composer (hwcomposer) to render accelerated 2D graphics, but hwcomposer will be upgraded.

With that warning aside, remove the old Android device driver compatibility layer:
Code: [Select]
sudo apt remove droid-hal-cosmopda
After removing droid-hal-cosmopda, it will start setting up the new gemian-system package, which includes downloading a new Android device driver compatibility layer.  You should see output similar to this:
Code: [Select]
Reading package lists... Done
Building dependency tree     
Reading state information... Done
The following packages will be REMOVED:
  droid-hal-cosmopda
0 upgraded, 0 newly installed, 1 to remove and 2 not upgraded.
1 not fully installed or removed.
After this operation, 48.8 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 211312 files and directories currently installed.)
Removing droid-hal-cosmopda (0.1+0~20200719231650.20~1.gbp236b48) ...   
Job for dev-cpuset.mount failed.
See "systemctl status dev-cpuset.mount" and "journalctl -xe" for details.
Job for system.mount failed.
See "systemctl status system.mount" and "journalctl -xe" for details.
Setting up gemian-system (0.1+0~20210323112818.11~1.gbp3d78a2) ...   
Target checksum f8b91bd83795c62f1e9b33be9a4156bdf4a90658
Checking /var/lib/lxc/android/android-rootfs.img
sha1sum: /var/lib/lxc/android/android-rootfs.img: No such file or directory
Checksum
Downloading new Android system image
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed 
... (cut download progress lines) ...
100  125M  100  125M    0     0  2227k      0  0:00:57  0:00:57 --:--:-- 1605k
Moving new system image into place, please reboot to activate

Do as it says and reboot your Cosmo as we must start using the v4 kernel in order to complete the upgrade:
Code: [Select]
sudo reboot
You should see a new Gemian boot logo now, but that's all you'll see until we complete Step 4.


Step 4: Finish installing v4 Linux
In Step 3, we installed lxc-android, but it didn't finish because the new Android system image was not active yet.  Now we'll need to run it again and let it finish, but first we must remove some files that were left over from an automatically attempted systemd upgrade.  If we don't, apt will complain about masked /etc/systemd/system/*.mount unit files.

Log back into the Cosmo via ssh as in Step 2 (you will need to remove and reinsert the USB-C/Ethernet hub first), then do this:
Code: [Select]
for file in `ls -l /etc/systemd/system/ | grep /dev/null | awk '{print $9}'`; do sudo rm -f /etc/systemd/system/$file; done
Now tell apt to finish setting up lxc-android:
Code: [Select]
sudo apt install lxc-android
You should see output similar to this:
Code: [Select]
Reading package lists... Done
Building dependency tree     
Reading state information... Done
lxc-android is already the newest version (0.3+gemian+0~20210422170358.20~1.gbp8861de).
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up lxc-android (0.3+gemian+0~20210422170358.20~1.gbp8861de) ...
Synchronizing state of lxc.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable lxc
Synchronizing state of lxc-net.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable lxc-net
Created symlink /etc/systemd/system/local-fs.target.wants/mnt-vendor-protect_s.mount -> /lib/systemd/system/mnt-vendor-protect_s.mount.
Created symlink /etc/systemd/system/local-fs.target.wants/mnt-vendor-protect_f.mount -> /lib/systemd/system/mnt-vendor-protect_f.mount.
Created symlink /etc/systemd/system/local-fs.target.wants/mnt-vendor-persist.mount -> /lib/systemd/system/mnt-vendor-persist.mount.
Created symlink /etc/systemd/system/local-fs.target.wants/mnt-vendor-nvdata.mount -> /lib/systemd/system/mnt-vendor-nvdata.mount. 
Created symlink /etc/systemd/system/local-fs.target.wants/mnt-vendor-nvcfg.mount -> /lib/systemd/system/mnt-vendor-nvcfg.mount.   
Created symlink /etc/systemd/system/multi-user.target.wants/wlan-module.service -> /lib/systemd/system/wlan-module.service.
Created symlink /etc/systemd/system/multi-user.target.wants/lxc-android.service -> /lib/systemd/system/lxc-android.service.

If you didn't see that but instead received an error about trying to overwrite some file which is also in some package, you will need to remove said package.  For example, I had alsa-utils installed.  I got this error:
Code: [Select]
dpkg: error processing archive /var/cache/apt/archives/lxc-android_0.3+gemian+0~20210422170358.20~1.gbp8861de_all.deb (--unpack):
 trying to overwrite '/lib/udev/rules.d/90-alsa-restore.rules', which is also in package alsa-utils 1.1.8-2

So I simply removed alsa-utils (for now):
Code: [Select]
sudo apt remove alsa-utils
Then I reran the lxc-android installation command above.

If lxc-android successfully installs as shown above, your system is now upgraded to v4!

Reboot and make sure everything boots up cleanly:
Code: [Select]
sudo reboot

Step 5: Confirm that it really is Gemian v4
Let's see if we actually upgraded to v4.  Log back into the Cosmo and check the kernel version:
Code: [Select]
cat /proc/version
As of this writing, the kernel version, compiler version, and build date are:
Code: [Select]
Linux version 4.4.146 (pbuilder@quasar) (gcc version 8.3.0 (Debian 8.3.0-6) ) #1 SMP PREEMPT Mon Apr 19 21:45:57 UTC 2021
Now try installing a v4-only software package.  I tried camera-app.  New for v4 is camera support.  Unfortunately it uses the same low quality MediaTek drivers and libraries as the stock camera app under Android, but it's better than nothing (unless it dissuades you from using a real camera).  If you're interested in a high quality camera app for Android, try my port of Google Camera for the Cosmo here: https://www.oesf.org/forum/index.php?topic=35912.msg297145#msg297145

Some of you might have followed my OpenGL 3D acceleration improvement instructions here: https://www.oesf.org/forum/index.php?topic=36525.msg297543#msg297543
This still works on v4, but not with the camera app.  For the camera app, you should disable gl4es before launching, like this:
Code: [Select]
LD_LIBRARY_PATH= camera-app
My Anbox port, which allows you to run Android apps under Linux on the Cosmo, also works on v4, except this time you shouldn't need to recompile the kernel as the necessary changes were added to the v4 kernel (thanks, Adam Boardman!)
See Anbox for the Cosmo: https://www.oesf.org/forum/index.php?topic=36525.msg297542#msg297542


Step 6: Help!
"I broke it!"
To fully restore your original Linux installation, boot TWRP as in Step 1, plug your USB-C cable from your workstation into the Cosmo's left port, then perform the commands below from your workstation. 

IMPORTANT!  Check your typing carefully before executing!  Flashing the wrong "p" number (in mmcblk0pXX) could erase other parts of your Cosmo, possibly even bricking it.

Note we must supply /dev/block/mmcblk0p41, not /dev/block/by-name/DEBIAN_KDE, and /dev/block/mmcblk0p43, not /dev/block/by-name/linux, when flashing the Linux boot and system partitions.  Otherwise, the boot image won't take, and the system image write will fail with a "No space left on device" error.

First, restore the Linux boot partition.  Type or paste carefully on your workstation:
Code: [Select]
adb push cosmo_linux_boot_backup /dev/block/mmcblk0p41
This push is instant.

Second, restore the Linux system partition.  Type or paste carefully on your workstation:
Code: [Select]
adb push cosmo_linux_partition_backup /dev/block/mmcblk0p43
It took 7,885 seconds (2.2 hours) for my 83 GiB (90 GB) Linux partition to flash, at a rate of 10.9 MB/s.

Your Linux setup should now be exactly as it was before you attempted the upgrade.  Reboot:
Code: [Select]
adb reboot

"I can't use apt!"
If you get a "Temporary failure resolving 'gemian.thinkglobally.org'" error while running apt despite having full Internet access, I can only tell you I had to recompile both the v3 and v4 kernels with CONFIG_ANDROID_PARANOID_NETWORK=n to work around this.  See my post for details:
https://www.oesf.org/forum/index.php?topic=36525.msg297542#msg297542

The v4 kernel compile on Gentoo (but apparently not on Debian) requires a few more symlink workarounds than on the v3 kernel.  Request details if needed.