Photo: Thoughts on an infrastructure for distributing Linux application bundles tl;dr: A system to distribute Linux application bundles and exchange metadata about them is needed, and this writeup ponders on how to create the infrastructure needed for that.

+Sebastian Kügler  introduced at #QtCon the #KDE Software Store https://dot.kde.org/2016/09/03/kde-software-store, a central location to get themes, wallpapers and other non-compiled assets for applications or the desktop. What is even more exciting, he also announced that it will support "containerized apps", more adequately known as "application bundles" in formats such as Flatpak, Snappy and/or #AppImage in the future.

In this writeup, I will outline my ideas for the KDE Software Store and for distributing #Linux application bundles in general. This writeup is inspired by various chats I had on this year's QtCon. I can't name you all, but thanks especially to starbuck, +Matthias Klumpp, +Aleix Pol, +Sebastian Kügler, +Nicolas Froment, and +Thomas Bonte.

Linux distributions have been doing an excellent job at providing operating systems composed of many individual components. They have failed, however, to provide an easy way to sideload and run applications that are not core to the operating system. Getting the latest #GIMP for #CentOS is an experience like upgrading from Windows 7 to Windows 10 "Anniversary Update" just to get the latest version of Photoshop - no wonder the Linux market share on the desktop is still so low.
Up to very recently, even the #Qt -based application started by Linus Torvalds, Subsurface, was available on the download page for Windows and for macOS but not for Linux, and the packages in the Linux distributions were either missing or utterly outdated. This is where Linux application bundles and app stores (called so even if what they offer is free) such as https://www.linux-apps.com and app centers such as GNOME Software https://wiki.gnome.org/Apps/Software and KDE Discover https://github.com/KDE/discover come into the picture.

Let's take a concrete example. As of today, +QOwnNotes 16.09.2 is one of the latest apps on the site. If I click on the "Files" tab, it takes me to http://www.qownnotes.org/installation where I can find installers for +Ubuntu, +Linux Mint, +openSUSE , +Fedora Project, +Arch Linux, KaOS Linux, +Debian, +Gentoo, +Slackware, and +CentOS. Wow, that must have been a lot of work for the QOwnNotes developer +Patrizio Bekerle! Not every application project will have the time or resources to do that.

As do most open soure software projects these days, QOwnNotes uses +GitHub for soure code and release management at https://github.com/pbek/QOwnNotes. It also uses Travis CI to do continuous builds. Whenever someone pushes source code to the repository, Travis CI builds and tests the code at https://travis-ci.org/pbek/QOwnNotes. After a release build, it is deployed automatically to GitHub Releases at https://github.com/pbek/QOwnNotes/releases. After each build, notifications are sent out to the qownnotes IRC channel and to a mail address of the developer.

This is a quite typical combination for independent open source software projects, and here is where I would like Linux app stores to hook in.

Imagine that the Travis CI build would also generate a generic application bundle for Linux, like it already does today for macOS using macdeployqt to produce the actual bundle. Now imagine that linuxdeployqt https://github.com/probonopd/linuxdeployqt will do the exact same thing but for Linux (although the developer also could do it by hand like some https://github.com/search?l=yaml&q=appimage+travis&type=Code&utf8=%E2%9C%93 do).

Now, wouldn't it be great if the developer could add another snippet to set up a webhook notification https://docs.travis-ci.com/user/notifications/#Webhook-notifications that would inform the app store that the new version is available? For that to happen, each app store would need to have a URL at which it could be informed about new releases.

Automation is sorely needed so that information doesn't get outdated - on the +Krita page https://www.linux-apps.com/p/1126921/ you'll find versions from 10 years ago, complete with discussions on klik (AppImage predecessor), despite that Krita provides each release as an #AppImage these days.

But wait, we don't just want to tell one app store "here is a new release of QOwnNotes". Ideally, we want to inform all app stores that "here is a new release of QOwnNotes, here is its description, here are it screenshots, here is the link to its homepage", etc.

Which means a format for exchanging metadata about applications is needed. This is exactly what AppStream https://www.freedesktop.org/software/appstream/docs/sect-Metadata-Application.html is (already widely in use). The application author can provide a /usr/share/metainfo/qownnotes.appdata.xml file that would specify all of these things that describe the application, but not the binaries (e.g., download links). For those, the AppStream system mandates that a so-called AppStream Hub (also known as a "compose server") collects the information about the available binaries and puts them into what is called "collection data". AppImage support was recently added to the AppStream specification https://github.com/ximion/appstream/commit/1df590df8557625b14c482d5af50cb30a35d47ca.

Maybe the application author wants to provide different channels with different versions for different user groups. One type of user might want the latest released versions, while aother type of user might want every bleeding-edge continuous build. The author should be able to define channels like "release" and "continuous" which users could then select, with the default being "release".

So, the flow should go like this: The QOwnNotes author uses Travis CI and linuxdeployqt (or an own script) to produce a cross-distro Linux bundle, e.g., an AppImage. He adds a webhook notification to an AppStream Hub, informing it about a) the location of the qownnotes.appdata.xml file and b) the location of the AppImage. The AppStream Hub would use that to update its collection data (the information about all the available software) and send that to app stores like https://www.linux-apps.com and, in the future https://store.kde.org/. If someone writes a plugin for Gnome Software and KDE Discover, it would also show up in the distribution's native package mangager GUIs, along with the native distribution packages. The AppStream Hub could download and inspect the AppImage, e.g., to make sure it really runs on at least a defined set of distributions, and to extract further information (e.g., the download size) from the AppImage. It could also scan for outdated versions of libraries contained in the AppImgae and perform various lint tests, https://github.com/probonopd/AppImages/blob/master/appdir-lint.sh is just a starting point and could do much more.

Now, what about updating the bundled applications on the users' systems, you might ask. AppImageUpdate can do binary delta updates of installed applications, which means that to update from one continuous build to the next only a few MB need to be downloaded, rather than the entire bundle https://github.com/probonopd/AppImageKit/blob/master/AppImageUpdate.AppDir/README.md.

What about security? The system should check for gpg signatures to make sure that the bundled applications really come from where they claim to come from.

Unlike other solutions in the making, this would not be slanted to any particular Linux distribution, store, or set of tools (yes I mean you, +Mark Shuttleworth).

This should make everyone happy: The application developer can continue to use the well-known workflow consisting of GitHub, Travis CI, GitHub Releases, and Notifications. He doesn't have to produce a gazillion of different native distribution packages anymore (but of course can choose to continue doing so). The app store gets the most recent information about new versions together with the most recent download links for cross-distro application bundles. The user can go to the app store and download software without having to deal with repositories and without having to upgrade anything in his stable base system. And all of this works entirely independent from the distribution's package mangager.

So, here are my questions:
* How would you like a solution like this?
* Would it make your life easier?
* Do you think it would make your users happier?
* Would you be willing to use such a system?
* Would you be willing to invest some time in setting up the infrastructure for this?
* Are you aware of better solutions?
* What else comes to your mind?
* Who would be in the best position to operate the AppStream Hub?
Loading...
probono
Public
Thoughts on an infrastructure for distributing Linux application bundles tl;dr: A system to distribute Linux application bundles and exchange metadata about them is needed, and this writeup ponders on how to create the infrastructure needed for that.

+Sebastian Kügler  introduced at #QtCon the #KDE Software Store https://dot.kde.org/2016/09/03/kde-software-store, a central location to get themes, wallpapers and other non-compiled assets for applications or the desktop. What is even more exciting, he also announced that it will support "containerized apps", more adequately known as "application bundles" in formats such as Flatpak, Snappy and/or #AppImage in the future.

In this writeup, I will outline my ideas for the KDE Software Store and for distributing #Linux application bundles in general. This writeup is inspired by various chats I had on this year's QtCon. I can't name you all, but thanks especially to starbuck, +Matthias Klumpp, +Aleix Pol, +Sebastian Kügler, +Nicolas Froment, and +Thomas Bonte.

Linux distributions have been doing an excellent job at providing operating systems composed of many individual components. They have failed, however, to provide an easy way to sideload and run applications that are not core to the operating system. Getting the latest #GIMP for #CentOS is an experience like upgrading from Windows 7 to Windows 10 "Anniversary Update" just to get the latest version of Photoshop - no wonder the Linux market share on the desktop is still so low.
Up to very recently, even the #Qt -based application started by Linus Torvalds, Subsurface, was available on the download page for Windows and for macOS but not for Linux, and the packages in the Linux distributions were either missing or utterly outdated. This is where Linux application bundles and app stores (called so even if what they offer is free) such as https://www.linux-apps.com and app centers such as GNOME Software https://wiki.gnome.org/Apps/Software and KDE Discover https://github.com/KDE/discover come into the picture.

Let's take a concrete example. As of today, +QOwnNotes 16.09.2 is one of the latest apps on the site. If I click on the "Files" tab, it takes me to http://www.qownnotes.org/installation where I can find installers for +Ubuntu, +Linux Mint, +openSUSE , +Fedora Project, +Arch Linux, KaOS Linux, +Debian, +Gentoo, +Slackware, and +CentOS. Wow, that must have been a lot of work for the QOwnNotes developer +Patrizio Bekerle! Not every application project will have the time or resources to do that.

As do most open soure software projects these days, QOwnNotes uses +GitHub for soure code and release management at https://github.com/pbek/QOwnNotes. It also uses Travis CI to do continuous builds. Whenever someone pushes source code to the repository, Travis CI builds and tests the code at https://travis-ci.org/pbek/QOwnNotes. After a release build, it is deployed automatically to GitHub Releases at https://github.com/pbek/QOwnNotes/releases. After each build, notifications are sent out to the qownnotes IRC channel and to a mail address of the developer.

This is a quite typical combination for independent open source software projects, and here is where I would like Linux app stores to hook in.

Imagine that the Travis CI build would also generate a generic application bundle for Linux, like it already does today for macOS using macdeployqt to produce the actual bundle. Now imagine that linuxdeployqt https://github.com/probonopd/linuxdeployqt will do the exact same thing but for Linux (although the developer also could do it by hand like some https://github.com/search?l=yaml&q=appimage+travis&type=Code&utf8=%E2%9C%93 do).

Now, wouldn't it be great if the developer could add another snippet to set up a webhook notification https://docs.travis-ci.com/user/notifications/#Webhook-notifications that would inform the app store that the new version is available? For that to happen, each app store would need to have a URL at which it could be informed about new releases.

Automation is sorely needed so that information doesn't get outdated - on the +Krita page https://www.linux-apps.com/p/1126921/ you'll find versions from 10 years ago, complete with discussions on klik (AppImage predecessor), despite that Krita provides each release as an #AppImage these days.

But wait, we don't just want to tell one app store "here is a new release of QOwnNotes". Ideally, we want to inform all app stores that "here is a new release of QOwnNotes, here is its description, here are it screenshots, here is the link to its homepage", etc.

Which means a format for exchanging metadata about applications is needed. This is exactly what AppStream https://www.freedesktop.org/software/appstream/docs/sect-Metadata-Application.html is (already widely in use). The application author can provide a /usr/share/metainfo/qownnotes.appdata.xml file that would specify all of these things that describe the application, but not the binaries (e.g., download links). For those, the AppStream system mandates that a so-called AppStream Hub (also known as a "compose server") collects the information about the available binaries and puts them into what is called "collection data". AppImage support was recently added to the AppStream specification https://github.com/ximion/appstream/commit/1df590df8557625b14c482d5af50cb30a35d47ca.

Maybe the application author wants to provide different channels with different versions for different user groups. One type of user might want the latest released versions, while aother type of user might want every bleeding-edge continuous build. The author should be able to define channels like "release" and "continuous" which users could then select, with the default being "release".

So, the flow should go like this: The QOwnNotes author uses Travis CI and linuxdeployqt (or an own script) to produce a cross-distro Linux bundle, e.g., an AppImage. He adds a webhook notification to an AppStream Hub, informing it about a) the location of the qownnotes.appdata.xml file and b) the location of the AppImage. The AppStream Hub would use that to update its collection data (the information about all the available software) and send that to app stores like https://www.linux-apps.com and, in the future https://store.kde.org/. If someone writes a plugin for Gnome Software and KDE Discover, it would also show up in the distribution's native package mangager GUIs, along with the native distribution packages. The AppStream Hub could download and inspect the AppImage, e.g., to make sure it really runs on at least a defined set of distributions, and to extract further information (e.g., the download size) from the AppImage. It could also scan for outdated versions of libraries contained in the AppImgae and perform various lint tests, https://github.com/probonopd/AppImages/blob/master/appdir-lint.sh is just a starting point and could do much more.

Now, what about updating the bundled applications on the users' systems, you might ask. AppImageUpdate can do binary delta updates of installed applications, which means that to update from one continuous build to the next only a few MB need to be downloaded, rather than the entire bundle https://github.com/probonopd/AppImageKit/blob/master/AppImageUpdate.AppDir/README.md.

What about security? The system should check for gpg signatures to make sure that the bundled applications really come from where they claim to come from.

Unlike other solutions in the making, this would not be slanted to any particular Linux distribution, store, or set of tools (yes I mean you, +Mark Shuttleworth).

This should make everyone happy: The application developer can continue to use the well-known workflow consisting of GitHub, Travis CI, GitHub Releases, and Notifications. He doesn't have to produce a gazillion of different native distribution packages anymore (but of course can choose to continue doing so). The app store gets the most recent information about new versions together with the most recent download links for cross-distro application bundles. The user can go to the app store and download software without having to deal with repositories and without having to upgrade anything in his stable base system. And all of this works entirely independent from the distribution's package mangager.

So, here are my questions:
* How would you like a solution like this?
* Would it make your life easier?
* Do you think it would make your users happier?
* Would you be willing to use such a system?
* Would you be willing to invest some time in setting up the infrastructure for this?
* Are you aware of better solutions?
* What else comes to your mind?
* Who would be in the best position to operate the AppStream Hub?

6 plus ones
2 comments