Libervia progress note 2021-W18

goffi 06/05/2021, 11:39 jabber-xmpp-en SàT Libervia project libre Libervia progress SàT progress

Hi,

again, lot of things have happened since last progress note, so I'll only talk about major changes here.

"Salut à Toi" is now "Libervia"

The project has been renamed to "Libervia". Even if I personally loved the former name (which was a reference to a French punk band song, an which could be translated to "hi to you", a nice fit for a communication tool), it proved to be hard to pronounce and remember for non French speakers, and the many names of frontends and project components were confusing. The name change has been discussed for long in the association, but the new ActivityPub/Pubsub end-to-end encryption project accelerated things: after a talk with NLnet, we decided to move forward on this so project name would not change in the middle of its development.

After doing a quick poll, we confirmed that "Libervia" (which was formerly the name of the web frontend only) would be the new name.

All parts are now named in straightforward way: "Libervia Backend", "Libervia Web", "Libervia Desktop/Mobile" (currently the same Kivy frontend for both), "Libervia TUI" and "Libervia CLI", with matching executable names (libervia-backend, libervia-web, libervia-desktop, libervia-tui, libervia-cli also aliased as li). The former names are kept internally and as aliases.

The non-profit (French "loi 1901") association behind it stays with the name "Salut à Toi".

This renaming has involved a lot for work, it took weeks to update code, web sites, doc, etc. and according to our statuses, we had to make a general assembly to discuss this decision. It's still not fully finished (notably the official web site URL is still https://salut-a-toi.org, while https://www.libervia.org is currently used for the demo server), and source code repositories have not been modified for the moment, but most of the renaming is done, and you can now reference the whole project as "Libervia"

Official Website Update

Following the changes in Libervia Web themes, the official website one has also been updated and is now based on Libervia's Web Bulma theme. The news now links to my personal blog as it is where you'll have most up-to-date informations about Libervia development (and the former page was broken). Tickets/Bug tracker is now directly accessible from the official site, as it makes more sense to have it there. It's still accessible from goffi.org, and thanks to its decentralised nature, it's usable transparently on both locations.

I've also temporarily disabled account registration on the bug-tracker due a wave of spammy accounts. I will have to put in place a protection for that, but I'm reluctant to use popular non-libre options.

Flatpak and Docker

While working on the renaming and website, I've updated the Flatpaks (they were really outdated), and Docker images. Flatpaks is for now using a specific dev repos, but I hope to see Libervia on Flathub after the release.

I've created Docker images and Docker Compose file to run quickly a local demo of Libervia Web, you can see the instruction on the Official Website.

Ideally, I would like to also create Snaps, Appimages, Nix packages, etc. But I'm lacking time (Flatpak and Docker are already too much time consuming) and prefer to focus on the code rather than on the packaging, help is more than welcome though.

User Friendly URLs

As you may have noticed on the last blog posts, URLs are now more user friendly:

A blog post is referenced using its item ID, and previously a unique ID was used for that, which is relatively long and doesn't give any information about the content, but is necessary to avoid conflict (writing a blog post with an existing ID will overwrite the previous one).

To make it more pleasant, a URL friendly extension was then added, and not used to retrieve the item, so in the example above, www.goffi.org/b/LFMqr7xC2aNf4MDgkbamBY links to the same blog post as www.goffi.org/b/LFMqr7xC2aNf4MDgkbamBY/sat-progress-note-2020-w53. The resulting URL is long and not easy to read, but the item is unique.

The new behaviour directly use URL friendly item IDs, and to avoid conflict, a short random suffix is appended (on the example above, QGqK is the suffix). After some tests, the collision risk for a short suffix like that is not that high (I've tested millions of IDs without collision), and it may anyway happen only if 2 blog posts have the exact same title, so the risk is very low. The resulting URL is more pleasant.

This URL friendly ID is used by default when a blog post is created, but it can be deactivated if user_friendly_id is set to false in blog post metadata, or by specifying manually an item id.

To accompany this change, a new Libervia CLI rename subcommand has been added to li blog and li pubsub, which will change the ID of an item. As there is no standard rename operation in XMPP Pubsub, this is done by copying the item to the new ID, then delete the former one in case of success.

Navigation Helpers in Libervia Web

It was not really easy so far to know where we were in Libervia Web. To help with this, the selected menu is now shown activated, and a breadcrumb has been added.

The breadcrumb is only shown when there are at least 2 elements to show (i.e. not on root pages). It is generated automatically by default, but can be customised with specific label, sub-elements, or even icons, like in the file sharing screenshot below:

Libervia Web 0.8 Breadcrumbs Screenshot

Blog Editor

As it was not possible anymore to write a new blog item with Libervia Web, I've made a blog item editor, which is relatively basic for now, but working. If JavaScript is activated, you'll get a tags editor, preview, and autosaving:

Libervia Web 0.8 Blog Editor Screenshot

File Sharing Quotas

One last missing piece I was needing before release was to put in place quotas on the file sharing component, this is now done.

Indeed, this component doesn't work with a per-file limit like most others do, but with a per-user quota, and you can upload any file size you want at long as you're not over quota.

Release to come

It's more than time to think about the release. I wanted to improve the chat notably in Libervia Web where it's still really basic since we moved out from the former frontend, but finally I've decided to report this to next release, as I plan to refactor messages handling, and for now I need to concentrate on the ActivityPub gateway.

So I'll soon prepare a beta version, and plan to do the release in a couple of weeks. I'll do bugfix on the 0.8 version during this time, but will avoid any important new development.

ActivityPub gateway project

With all the work done above (and other things, I've not mentioned everything), I've been late to start working on the ActivityPub project, but now I can focus on it. The first task is about developing a Pubsub cache as Libervia is currently getting its data for Pubsub related feature directly from the services.

Beside the obvious speed improvement, having a local cache will give the possibility to do data search/manipulation (such as doing Full-Text Search when the Pubsub service doesn't implement it, or doing feature-specific data analysis), handle message received unordered, allow to keep decrypted data when received from e2ee items, etc.

So far, SQLite was used for data storage in Libervia, by using Twisted's adbapi and custom semi-automatic schema update/data migration. It has been working relatively well so far, but it's no pleasant to maintain.

Fortunately, SQLAlchemy has recently added support for AsyncIO, thus it can now be used in Libervia. This is great, as SQLAlchemy is popular and rock solid, so I've decided to go with it. This will open the possibility to use other backends (like PostgreSQL), and refactor Libervia to use SQLAlchemy's ORM. Logically, Alembic will be used for data migration, which should make database modifications easier.

Such a cache will make possible to implements things like items discovery based on categories (or search by "hashtags" as it named in other software).

That's all for this note, see you soon.

06/05/2021, 14:27

M

06/05/2021, 13:40

E

06/05/2021, 11:59

E

06/05/2021, 11:40

Passerelle ActivityPub et chiffrement de bout en bout Pubsub

goffi 23/04/2021, 11:48 planet-libre seenthis jabber-xmpp SàT Libervia project libre ActivityPub e2ee nlnet

Bonjour,

c'est avec grand plaisir que je vous annonce qu'un projet de passerelle ActivityPub <=> XMPP doublé d'une adaptation du chiffrement de bout en bout à PubSub a été sélectionné pour une subvention par NLnet/NGI0 Discovery Fund (avec un soutien financier de la commission européenne "Next Generation Internet") : https://nlnet.nl/project/Libervia/

Ce gros projet est divisé en 27 étapes, et va occuper la plupart de mon temps dédié au projet Libervia (anciennement "Salut à Toi", le projet a été renommé, je vais expliquer cela, en anglais, dans une "progress note" à venir).

La passerelle XMPP <=> ActivityPub va joindre 2 protocoles ouverts et décentralisés majeurs. En pratique, ce sera un composant pour serveur XMPP (utilisable avec n'importe quel serveur), et il va implémenter le protocole "serveur à serveur" de ActivityPub (aussi connu comme « protocole de fédération » ou « Federation Protocol »). Côté XMPP, ce sera principalement un service PubSub (avec quelques extras, comme les messages privés convertis en messages XMPP).

Le blogage XMPP (XEP-0277: Microblogging over XMPP) va être utilisé, et donc n'importe quel client qui l'implémente aura accès aux publications ActivityPub (Libervia et Movim par exemple).

Pour les fonctionnalités présentes dans ActivityPub et pas encore dans XMPP, il est prévu de proposer des « protoXEPs » (c.-à-d. des propositions d'extensions de XMPP) pour les implémenter. Les évènements font aussi parti du projet, et une compatibilité entre Mobilizon et Libervia est prévue, ainsi qu'une protoXEP pour standardiser cela côté XMPP.

Tout ceci est très excitant, parce que ça va étendre les deux réseaux, et donner un coup d'accélérateur aux projets intégrant le blogage et le chat XMPP.

La seconde partie du projet concerne le chiffrement de bout en bout. XMPP a connu des améliorations majeures sur le chiffrement de bout en bout suite au travail fait sur OMEMO, notamment initié avec Conversations, et sur OX, une intégration moderne de OpenPGP. C'est excellent, mais cela se concentrait jusqu'ici sur la messagerie instantanée. Le but va être d'ajouter le chiffrement de bout en bout au PubSub XMPP, avec les protoXEPs et l'implémentation dans Libervia. En d'autres termes, à la fin de ce projet, il sera possible d'utiliser le chiffrement de bout en bout avec toutes les fonctionnalités basées sur Pubsub (comme les blogs, forums, listes ou autre évènements que vous pouvez utiliser avec Libervia), c'est énorme ! Les signatures feront partie du projet aussi, signifiant qu'il sera possible d'authentifier un élément comme un billet de blog d'une façon standardisée.

En plus des standards qui vont bénéficier à la communauté XMPP entière, tout ceci sera implémenté dans Libervia, ce qui inclus mettre à jour l'implémentation actuelle pour arriver à l'état de l'art (c.-à-d. mettre à jour l'implémentation actuelle d'OMEMO et implémenter OX).

La dernière partie du projet sera l'implémentation du chiffrement de bout en bout dans le frontal web. En effet, l'architecture spécifique de Libervia fait qu'OMEMO n'est actuellement pas utilisable depuis le navigateur (l'implémentation est faite dans le « backend »). Pour rendre ceci possible, la bibliothèque Python OMEMO qui est actuellement utilisée va être portée sur WebAssembly et Brython, ce qui va permettre de faire du chiffrement et du déchiffrement directement depuis le navigateur.

Comme vous pouvez vous en rendre compte, c'est beaucoup de travail. Je vais faire la majeure partie de tout ceci, mais je ne serai pas seul (en particulier l'auteur de Python OMEMO fera le port sur wasm/Brython à travers ce projet). Le développement devrait prendre environ un an, la passerelle ActivityPub <=> XMPP étant la première partie.

Je profite de ce billet pour remercier encore NLnet et le programme NGI de l'UE, ainsi que mon employeur (Sourcefabric, qui développe en particulier Superdesk) pour m'avoir permis d'adapter mon rythme de travail.

J'ai adapté les tâches au traqueur de bogues de Libervia (qui fonctionne avec XMPP), aussi vous pouvez voir leur détails (en anglais) et suivre leur progression à cette adresse : https://salut-a-toi.org/bugs?search=nlnet

Enfin, je vais continuer à publier des notes de progression (en anglais), qui devraient aussi être disponible sur ActivityPub plus tard dans l'année ;). Le site web a été mis à jour, avec de nouvelles installations via Flatpak et Docker, RDV sur https://www.salut-a-toi.org.

Si vous avez des questions ou commentaires, n'hésitez pas à vous rentre sur le salon officiel XMPP de Libervia à sat@chat.jabberfr.org, ou à me joindre par exemple via ActivityPub (@Goffi@mastodon.social).

À bientôt.

E

06/05/2021, 11:13

23/04/2021, 18:56

ActivityPub Gateway and Pubsub e2ee

goffi 23/04/2021, 10:21 jabber-xmpp-en SàT Libervia project libre ActivityPub e2ee

Hello,

it's my pleasure to announce that an ActivityPub <=> XMPP gateway doubled with Pubsub end-to-end encryption project has been selected for a grant by NLNet/NGI0 Discovery Fund (with financial support from European Commission's Next Generation Internet programme): https://nlnet.nl/project/Libervia/

This big project is split in 27 steps, and will take most of my time dedicated to the Libervia project (formerly "Salut à Toi", the project has been renamed, I'll explain that in a upcoming progress note).

The XMPP <=> ActivityPub gateway will join two major open and decentralised protocols. In practice it will be a XMPP server component (usable with any server), and implement the ActivityPub server to server protocol (or "Federation Protocol"). On XMPP side, it will be mostly a Pubsub service (with some extra, like private messages converted to XMPP message stanza).

XMPP blogging (XEP-0277: Microblogging over XMPP) will be used, and thus any client supporting it will have access to ActivityPub publications (Libervia and Movim for instance).

For features present in ActivityPub and not yet in XMPP, it is planned to propose protoXEPs (i.e. proposition of XMPP extensions), to implement them. Events will also be part of the project, with a compatibility between Mobilizon and Libervia expected, and a protoXEP to have this standardised on XMPP side.

This is quite exiting, as it will extend both networks, and boost projects integrating blogging and XMPP chat.

The second part of the project is about end-to-end encryption. XMPP has enjoyed major improvements on end-to-end encryption following the work done on OMEMO, notably initiated with Conversations, and on OX, modern OpenPGP integration. This is great, but has been so far mainly focusing on instant messaging. The goal will be here to add end-to-end encryption to XMPP Pubsub, which includes protoXEPs and implementation in Libervia. In other terms, at the end of this project, it will be possible to use e2ee with all Pubsub based features (like blogs, forums, lists, or events that you can do on Libervia), this is huge! Signing will be part of the project too, meaning that it will be possible to authenticate something like a blog post in a standardised way.

Beside the standards which will benefit to the whole XMPP community, all of this will be implemented in Libervia, this include updating current implementation to the state of the art (i.e. updating current OMEMO implementation and implementing OX).

Last part of the project will be the implementation of e2ee in the web frontend. Due to Libervia specific architecture, OMEMO is not currently usable from the browser (the implementation is done on the backend). To make this possible, the Python OMEMO library which is currently used will be ported to WebAssembly and Brython, allowing to do encryption and decryption directly within the browser.

As you can see this is massive. I'll do most of this but I won't be alone (notably the author of Python OMEMO will do the wasm/Brython port as part of the project). The project should last circa one year, with the ActivityPub <=> XMPP gateway being the first part worked on.

I would like to thanks again NLNet and EU's NGI for allowing this, and my employer (Sourcefabric, which notably develops Superdesk) for letting me adapt my working schedule.

I've adapted the tasks to Libervia's (XMPP powered) bug tracker, so you can see step details and follow progress there: https://salut-a-toi.org/bugs?search=nlnet

Also I'll continue to publish progress notes so stay tuned to this blog (that should be available also on ActivityPub later this year ;) ). The website has been updated, with new Flatpak and Docker installations, check https://www.salut-a-toi.org.

If you have questions or comments, feel free to join Libervia's XMPP room at sat@chat.jabberfr.org or to contact me for instance via ActivityPub (@Goffi@mastodon.social).

See you soon.

E

06/05/2021, 11:13

23/04/2021, 18:57