SàT progress note 2019-W35

goffi 5 years ago jabber-xmpp-en SàT project libre SàT progress

Hi everybody,

I've skipped last week note because I could not find the time, and anyway I did the same thing as this week, so it was not really needed.

I'm currently working on porting jp (the CLI frontend) to Python's asyncio, and at the same time I'm documenting every command (I need to review them one by one, so it's a good occasion for that). It's long and really boring, but it's a good thing to do.

Let me explain why I'm doing that. SàT and its frontends are historically using D-Bus to communicate (it's an IPC, and we call it internally the bridge). SàT is 10 years old, and the Python implementation that we have used was the reference at the time : Python D-Bus. This implementation has flaws, that they recognise themselves (like trying to guess signature, see " Problems and alternatives " in the page linked). Furthermore, it's the main reason why we use GLib and its loop in SàT, which can cause trouble to install (mainly due to compilation).

Frontends and bridges are using a callbacks mechanism, which can be hard to code and maintain in some cases. Python-dbus can work with callbacks or in a blocking way, and historically we have used this feature in jp. For instance, to get backend version we have a getVersion() method, and to avoid too much callbacks hell, it was used with the bridge in a blocking way (blocking jp is not such a big deal as there is no UI to maintain or things which must run in background). This was wrong and experience shows why, you are either blocking or asynchronous, not a mix. Using D-Bus bridge in a blocking way, even if it's only in a few places, makes the code unusable with other bridges, so jp could only work with D-Bus, and today we have other ones (like Twisted Perspective Broker which is the bridge used on Android).

So in one hand we had the callback issue, and on the other hand some bridge calls using blocking mode to limit callbacks hell, and causing design issues.

The proper solution to that is to use coroutines, which is a way to write asynchronous code nearly like we write blocking code. Twisted has been offering this possibility for years thanks to the inlineCallbacks which is using Python generators in a smart way. It's really usable and indeed used in many parts of the backend: you basically have to use a decorator and yield syntax.
This could have been used in jp but this would have forced the use of Twisted (and we want jp to work with other frameworks if other bridges are used). Furthermore, inlineCallbacks are not easy to debug: in a debugger if you don't add a breakpoint just after the yield, a step over brings you in the execution workflow of the main loop.

And here come asyncio and the great async/await syntax. This syntax is doing basically the same thing as Twisted inlineCallbacks, but in a cleaner and more readable way, in locations where yield is not possible (in a loop for instance), and it is nowadays well handled by debuggers (at least pudb that I'm using). Thanks to that, it is possible to convert easily blocking parts of the code, and to simplify many things.

Doing something like repeating a method on several pages of result (when RSM is used for instance) becomes easy and pleasant. One of the main goal of asyncio was to have a common loop/framework for all asynchronous code, and Twisted made the compatibility layer so we can use Twisted and asyncio at the same time with async/await syntax.

I'm currently finishing that, for the moment I'm keeping 2 versions of the bridge (one callbacks based and one using async/await) to progressively do the conversion, but the goal is to have all frontends using the new syntax soon. It is one of the many things that has been unlocked by the port to Python 3. It's taking lot of time because of the documentation made at the same time, but it is definitely worth it.

That's all for this note, see you next week.

E

errormovim 4 years ago