Skip to content

Instantly share code, notes, and snippets.

@jluebbe
Created October 23, 2018 07:52
Show Gist options
  • Select an option

  • Save jluebbe/d27b2289208791f3805adf69a0dac482 to your computer and use it in GitHub Desktop.

Select an option

Save jluebbe/d27b2289208791f3805adf69a0dac482 to your computer and use it in GitHub Desktop.
ELCE 2018 BoF: Embedded Update Tools

BoF: Embedded Update Tools

Update Tools

Related Software

Possible topics (poll)

How to target different markets and environments? 6

Encrypted updates 10

Alternatives to A/B for constrained systems 20

Detecting a successful update 20

Migration of user data 25

Missing features in existing tools? 3

Automated testing of the update process 15

Base System Update vs. Application Update 8

  • Hybrid setups for mixed requirements (reliable System Partition + Partition for User-Applications)

Signing with HSM/token support 3

Benefits vs. complexity of multiple signatures (TUF) 6

Delta-updates for bandwidth-constrained devices (GSM). 15

  • Binary diffs.

Updating secondary processors/systems 10

Peer to peer update distribution? 2

Atomic bootloader updates? 15

Common format for image with manifest (signature, dependencies) 1

Ease of supporting a new bootloader 1

Migration of user data (25 votes)

add additional properties as json

  • will not confuse the fallback-system if update has failed

two data partitions, copy on update with possible migration script

How to migrate configuration data, i.e. /etc, data or root image data?

templating for the updated file, generate data from existing booted image, inject into template

  • allows update of the template for new config files (e.g. new service versions)

User Story:

regeneration of the config files on shutdown, makes sure user data is saved and available on next boot

store user data in seperate partition, compatible application will use it, incompatible will fallback to default

up to three slots for the config data, allow rollback to a previous configuration state

  • compatiblity check has to be done by the user
  • external configuration tool downloads the update and requires the user to select a compatible configuration state

swupdate based with extra application

User Story:

Move configuration data into the cloud, store a token for the correct configuration state

limited for device purposes, needs definitve internet configuration

configuration migration can be done in the cloud

device has individual token which does authentication

  • also used to track software updates
  • A/B Updates

User Story:

migration script

  • hard to detect correct migration via the script

snapshots of the user data to allow rollback, connect to software version

so far no rollback necessary, preferable for future proofing

User Story:

  • A/B with data
  • overlayfs used for configuration data
  • Application updates config database on startup
  • so far no rollback necessary

Alternatives to A/B for constrained systems (20 votes)

User Story (Rauc):

Small fallback/rescue system with update utility

update requires 2 reboots of the system

rollback not possible, since only one slot is available

  • ostree?

Alternative:

  • many local system with central controller
  • allow network boot for the network system via the controller
  • this allows a recovery via the central controller (which provides a working update)

User Story:

  • OSTree Yocto integration in meta-updater
  • All in one filesystem, hardlinks used to avoid copies
  • requires trust in the filesystem and eMMC
  • small rescue system as alternative to filesystem

Detecting a successful update (20 votes)

systemd Automatic Boot Assessment" ( https://github.com/systemd/systemd/blob/master/docs/AUTOMATIC_BOOT_ASSESSMENT.md )

Ideas:

  • newly installed vs rescue software

how to decide that the software is working correctly

if the kernel breaks, bootloader will boot old kernel

how to detect a break in userspace which requires a rollback

User Story:

  • network of nodes which update
  • Service waiting for systemd
  • talk to all "important" services
  • check the hardware and query hardware information
  • how to decide if the device is wrong or the network is wrong

User Story:

project specific area

maybe create an API all updaters can use to query the system

API is really narrow

  • could also handle rollback signaling

standard way to signal the updater

Networking, multi components:

  • higher level orchestration
  • individual system tags boot vs detection of the network as a whole
  • detect failure of a whole portion of the network failing
  • tools are the same, e.g. json, mqtt
  • but very application specific

Use boot integrity to verify the image?

  • all updaters already check during installation and try to detect corruption
  • application bugs are not detectable, often filesystem integrity not a problem

User story:

  • Link software which monitor the application to watchdog
  • if the software faulty trigger a reboot of the system
  • application does the sanity check (it might already be done as sanity check during normal operation)
  • wait for a longer time before the system is marked good
  • works well if the application has watchdog access

Two kinds of successful:

  • device is running
  • device is updatable

Delta-updates for bandwidth-constrained devices (GSM). (15 votes)

Binary diffs.

RAUC supports casync

  • does delta updates of the system
  • currently no support for binary deltas for offline updates

OSTree supports statically generated deltas

User Story:

  • using GSM cellular modems
  • per MB pricing
  • delta updates absolute requirement
  • open-vcdiff
  • crochet
  • implements all three binary updates, choose smallest one
  • full-update also possible for development case
  • Space saving: 20MB rootfs squashfs, few kilobytes to Megabytes range

User Story:

  • rsync A/B system
  • GSM
  • rsync can write delta to batch file, allows offline update
  • always online system, no offline update necessary

User Story:

  • slow device
  • update at the file level
  • Rescue/Production scheme, worst case rollback to production

New class of problems:

  • make sure that the system gets the correct updates
  • what if multiple versions need to apply in sequence
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment