emacs matrix client, with ement.el & pantalaimon (for encryption support)

Hello!

I'm trying to get a matrix client on emacs, with ement.el, and to work with encryption, using pantalaimon.

I hoping not to have put secret info/data/logs in the command output extract; plz don't hack me (there is already my user_id and too much info, damn i'm public).

Background

I've already got emacs installed on my machine (also using EXWM; but this is out of topic, just to mention the setup); my computer is running linux with distribution arch (pacman as the package manager).

(use-package ement
  :quelpa (ement :fetcher github :repo "alphapapa/ement.el" :upgrade t)
  :preface (use-package plz
             :quelpa (plz :fetcher github :repo "alphapapa/plz.el" :upgrade t))
  :defer (ement-connect :uri-prefix "http://localhost:8009"))

This is how ement.el is installed (for me), using use-package and quelpa-use-package.

Also, my machine already has dbus installed (seems pantalaimon requires it):

$ dbus-daemon --version
D-Bus Message Bus Daemon 1.14.10
Copyright (C) 2002, 2003 Red Hat, Inc., CodeFactory AB, and others
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Installation

Following the installation steps, and running pip install pantalaimon gives me an error, because of arch linux.

$ pip install pantalaimon
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using 'python -m venv path/to/venv'.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.
    
    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. Make sure you have python-pipx
    installed via pacman.

Running pacman -S python-pantalaimon gives an error, the (python) package does not exist (as a arch linux package).

Fortunately yay (to access the arch linux AUR repository), seems to exist.

yay -Ss pantalaimonaur
aur/pantalaimon-git e62cfe0-1 (+1 0.02) 
    Pantalaimon is an end-to-end encryption aware Matrix reverse proxy daemon.
aur/pantalaimon 0.10.5-5 (+5 0.00) (Orphaned) (Out-of-date: 2023-01-26) 
    Pantalaimon is an end-to-end encryption aware Matrix reverse proxy daemon.

Let's install aur/pantalaimon-git with the command yay -S pantalaimon-git, which seems to install correctly (the command is now available):

$ pantalaimon --help
Usage: pantalaimon [OPTIONS]

  pantalaimon is a reverse proxy for matrix homeservers that transparently
  encrypts and decrypts messages for clients that connect to pantalaimon.

Options:
  --version                       Show the version and exit.
  --log-level [error|warning|info|debug]
  --debug-encryption
  -c, --config PATH
  --data-path PATH
  --help                          Show this message and exit.

Note: there seems to be no command to clean stop pantalaimon, so I'm going with killall pantalaimon, which seems to work fine.

The olm library is also required, not sure if I installed it before, or if it was installed as a dependency of pantalaimon.

$ pacman -Ss olm
extra/libolm 3.2.15-1 [installed]
    Implementation of the Olm and Megolm cryptographic ratchets
extra/python-olm 3.2.15-1 [installed]
    Python bindings for Olm

Configuration

Now that the pantalaimon is avaialble, let's follow the "usage" on the git readme.

Created the file ~/.config/pantalaimon/pantalaimon.conf, and saved it after adding the following (default) content:

[local-matrix]
Homeserver = https://localhost:8448
ListenAddress = localhost
ListenPort = 8009

Note: my user id is @ugp:matrix.org, so should the Homeserver value be https://matrix.org instead? Not sure, but it seems to be what works, so here is the actual config that seems to work for me.

[local-matrix]
Homeserver = https://matrix.org
ListenAddress = localhost
ListenPort = 8009

Now, I run the command pantalaimon --log-level debug which should use the ement.el configuration (seen on the top of this article), from my emacs config:

(ement-connect :uri-prefix "http://localhost:8009")

Trying to make it work

Before connecting to ement with my user, let's run the encryption service, with the command pantalaimon --log-level debug, it gives us this output, with a "running daemon":

$ pantalaimon --log-level debug
[2023-11-07 10:10:54.822735] ERROR: pantalaimon: Notifications are enabled but no notification server could be found, disabling notifications.
======== Starting daemon for homeserver local-matrix on http://127.0.0.1:8009 ========
(Press CTRL+C to quit)

When now trying, in emacs, the command M-x ement-connect, it seems my user connects, but cannot really use the software yet.

Let's try to ement-disconnect, and start again.

error in process sentinel: Ement API error: "Ement: Unrecognized network error", #s(plz-error nil nil "curl process killed") [2 times]
Ement: Disconnected (@ugp:matrix.org)

And let's connect again M-x ement-connect → ask for user id @ugp:matrix.org → and then ask for password my_super_secret_pw_l0lz, and gives this output:

Ement: Checking server’s login flows...
Ement: Logging in with password...
Ement: Sync request sent, waiting for response...

The running pantalaimon daemon process, seems to give no log our output during this process.

I try to ement-disconnect (Ement: Disconnected (@ugp:matrix.org)) and eval within emacs the (ement-connect :uri-prefix "http://localhost:8009") emacs-lisp code, with output:

"Ement: Checking server’s login flows..."
Ement: Logging in with password...
Ement: Sync request sent, waiting for response...

Trying ement-list-rooms outputs the error ement-list-rooms: Wrong type argument: number-or-marker-p, nil.

Somehow getting some output from pantalaimon now:

[2023-11-07 10:20:03.035463] INFO: pantalaimon: New user logging in: @ugp:matrix.org
[2023-11-07 10:20:03.926763] INFO: pantalaimon: User: @ugp:matrix.org successfully logged in, starting a background sync client.
[2023-11-07 10:20:03.943219] INFO: pantalaimon: Indexing disabled.
[2023-11-07 10:20:05.200447] INFO: pantalaimon: Successfully started new background sync client for @ugp:matrix.org
[2023-11-07 10:20:05.210826] DEBUG: pantalaimon: UI loop received message UpdateUsersMessage(server='local-matrix', user_id='@ugp:matrix.org', device_id='KMOZYEPFFY')
[2023-11-07 10:20:05.320476] ERROR: pantalaimon: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See https://pypi.org/project/keyring for details.
[2023-11-07 10:20:05.320571] INFO: pantalaimon: Starting sync loop for @ugp:matrix.org
[2023-11-07 10:20:56.323802] INFO: pantalaimon: Trying to decrypt sync
[2023-11-07 10:20:56.323926] INFO: pantalaimon: Decrypting sync
[2023-11-07 10:20:56.324721] WARNING: pantalaimon: Error decrypting megolm event, no session found with session id 5jWYZKpO0EyZaE9ETc7EnhrbxnD/LqUTdfkunSYE6Zk for room !BmLrrUnvrTBEKaOdHM:matrix.org
[2023-11-07 10:20:56.324868] INFO: pantalaimon: Error decrypting sync, waiting for next pan sync
[2023-11-07 10:20:59.411624] INFO: pantalaimon: Trying to decrypt sync
[2023-11-07 10:20:59.411697] INFO: pantalaimon: Decrypting sync
[2023-11-07 10:20:59.411966] WARNING: pantalaimon: Error decrypting megolm event, no session found with session id 5jWYZKpO0EyZaE9ETc7EnhrbxnD/LqUTdfkunSYE6Zk for room !BmLrrUnvrTBEKaOdHM:matrix.org
[2023-11-07 10:20:59.412009] INFO: pantalaimon: Error decrypting sync, waiting for next pan sync
[2023-11-07 10:21:06.331570] INFO: pantalaimon: Decryption attempt timed out, decrypting with failures
[2023-11-07 10:21:06.331843] INFO: pantalaimon: Decrypting sync
[2023-11-07 10:21:06.332925] WARNING: pantalaimon: Error decrypting megolm event, no session found with session id 5jWYZKpO0EyZaE9ETc7EnhrbxnD/LqUTdfkunSYE6Zk for room !BmLrrUnvrTBEKaOdHM:matrix.org
[2023-11-07 10:21:06.333881] WARNING: pantalaimon: Error decrypting megolm event, no session found with session id 5jWYZKpO0EyZaE9ETc7EnhrbxnD/LqUTdfkunSYE6Zk for room !BmLrrUnvrTBEKaOdHM:matrix.org
[2023-11-07 10:21:06.334683] WARNING: pantalaimon: Error decrypting megolm event, no session found with session id 5jWYZKpO0EyZaE9ETc7EnhrbxnD/LqUTdfkunSYE6Zk for room !BmLrrUnvrTBEKaOdHM:matrix.org
[2023-11-07 10:21:06.335519] WARNING: pantalaimon: Error decrypting megolm event, no session found with session id 5jWYZKpO0EyZaE9ETc7EnhrbxnD/LqUTdfkunSYE6Zk for room !BmLrrUnvrTBEKaOdHM:matrix.org

Also now stopping the pantalaimon daemon, and running it again:

$ pantalaimon --log-level debug
[2023-11-07 10:26:03.638520] ERROR: pantalaimon: Notifications are enabled but no notification server could be found, disabling notifications.
[2023-11-07 10:26:03.707123] ERROR: pantalaimon: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See https://pypi.org/project/keyring for details.
[2023-11-07 10:26:03.707204] WARNING: pantalaimon: Not restoring client for @ugp:matrix.org KMOZYEPFFY, missing access token.
======== Starting daemon for homeserver local-matrix on http://127.0.0.1:8009 ========
(Press CTRL+C to quit)

Connecting with ement-connect gives output:

Ement: Logging in with password...
Ement: Sync request sent, waiting for response...

Trying ement-directory gives output:

Ement: Listing 100 rooms on matrix.org...
[1 minute passes]
101068 rooms remaining (use + to fetch more)
Ement: Sync timed out (@ugp:matrix.org).  Syncing again...

This command opens a *Ement Directory: matrix.org* buffer with listing matrix.org public rooms.

At this time there is still no output in the newly re-launched pantalaimon, so doing the following steps:

ement-disconnect
stop pantalaimon daemon && clear bash shell (to get a cleaner output with our fresh re-start)
re-start pantalaimon daemon pantalaimon --log-level debug

pantalaimon --log-level debug
[2023-11-07 10:35:07.277111] ERROR: pantalaimon: Notifications are enabled but no notification server could be found, disabling notifications.
[2023-11-07 10:35:07.353486] ERROR: pantalaimon: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See https://pypi.org/project/keyring for details.
[2023-11-07 10:35:07.353574] WARNING: pantalaimon: Not restoring client for @ugp:matrix.org KMOZYEPFFY, missing access token.
======== Starting daemon for homeserver local-matrix on http://127.0.0.1:8009 ========
(Press CTRL+C to quit)

eval in emacs (ement-connect :uri-prefix "http://localhost:8009") → enter user_id + password At this stage, pantalaimon daemon gives the following output:

[2023-11-07 10:35:44.942624] INFO: pantalaimon: New user logging in: @ugp:matrix.org
[2023-11-07 10:35:45.478457] INFO: pantalaimon: User: @ugp:matrix.org successfully logged in, starting a background sync client.
[2023-11-07 10:35:45.482359] INFO: pantalaimon: Indexing disabled.
[2023-11-07 10:35:47.051543] INFO: pantalaimon: Successfully started new background sync client for @ugp:matrix.org
[2023-11-07 10:35:47.051736] ERROR: pantalaimon: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See https://pypi.org/project/keyring for details.
[2023-11-07 10:35:47.051807] INFO: pantalaimon: Starting sync loop for @ugp:matrix.org
[2023-11-07 10:35:47.125745] DEBUG: pantalaimon: UI loop received message UpdateUsersMessage(server='local-matrix', user_id='@ugp:matrix.org', device_id='JOQYMXUQWL')

Then a few seconds/minutes later, the output is much longer, some errors, some warnings, some event logs, such as a selection (from the thousands of logs; my user has joined many rooms, encrypted or not):

[...]
[2023-11-07 10:23:00.290774] WARNING: pantalaimon: Error decrypting megolm event, no session found with session id d/OAg8MtWjp/waDiaNT4NzmSevLc6+8QZJn6sMPpJMM for room !yzcsjHevlHTgOzsdIk:matrix.org
[2023-11-07 10:23:41.152781] INFO: pantalaimon: Unable to decrypt event from @ugp:matrix.org via DIAWCEKWIG.
[2023-11-07 10:23:41.152874] INFO: pantalaimon: Requesting room key for undecrypted event.
[...]

At this stage, pantalaimon seems to work, as in "it gives output related to my user".

Also, in element web client there seems to be a new device (good I had it open!); the element popup says:

New login. Was this you?
(1/2)
pantalaimon
Unverified · Last activity 11:20 · 32.161.181.106 · KMOZYEPFFY
No — Yes, it was me

So I clicked YES, because it was me (actually I had to accept 2 times, since I've tried to run the commands multiple times).

Usage

Now that ement.el is installed in my emacs, and setup to communicate with pantalaimon, which seems to receive data from me using ement, let's see what can be done, and what the panctl command can help us with.

Trying M-x ement-join-room and then giving it the room #ement.el:matrix.org (which I had already join in element web app) seems to output Joined room: !NicAJNwJawmHrEhqZs:matrix.org.

systemd

For convenience a systemd service file is provided.

Can be found in the github

Run (with sudo) the command systemctl edit --force --full pantalaimon.service, to create a new systemd service file, and add the content:

[Unit]
Description=Pantalaimon E2E Matrix reverse proxy

[Service]
ExecStart=/usr/bin/pantalaimon

[Install]
WantedBy=default.target

This file will be saved in /etc/systemd/system/pantalaimon.service.

Maybe add --log-level debug as argument after ExecStart=/usr/bin/pantalaimon, to debug?

Note: it will be opened with vim, so type :wq to save (write) and quit after having made the changes (i to insert).

To get the status, start and enable the systemd service:

systemctl status pantalaimon # gives status report
systemctl start pantalaimon # starts the service (once)
systemctl stop pantalaimon # stops the service if started (not "disabled" if "enabled")
systemctl enable pantalaimon # enables the service, so it starts with the machine
systemctl disable pantalaimon # to stop the service from starting with booting the machine

Note: when enabled it will output: Created symlink /etc/systemd/system/default.target.wants/pantalaimon.service → /etc/systemd/system/pantalaimon.service. and the service should still be "started" (or reboot the machine)

To edit the service sudo systemctl edit pantalaimon.

To apply changes made to the unit, execute sudo systemctl daemon-reload

Restart the service to loads the new service configuration (Unit file must be restated if you modify the running unit file) sudo systemctl restart pantalaimon.

Notes

Did not fully yet grasp how it should work, and why it does not seem towork fully; but this installation project is a WIP for me (both ement and pantalaimon seems like robust well coded software).

My questions are

how to simplify installation, cofiugration, and documentation, so noobs like me can make it work EZ without hassling.
how to run pantalaimon so it is a background process (and not in a shell i gotta keep open) → pantalaimon & seems to work; maybe a systemd thing too?
how to run pantalaimon so it starts when my machine starts
how to best autoconnect my matrix user when my machine/emacs starts, and where/how to save my SECRET_PASSWORT
how to live the good life and stop paying rent to them hotdamn capitalists
why Homeserver is not configured. for systemd service? pantalaimon works when run manually

Some debug technics:

M-x and toggle-debug-on-error to have a *backtrace* buffer open when emacs errors (usefull to debug an ement command)

About `olm` encryption lib

Seems pantalaimon is using the olm library in C (code here https://gitlab.matrix.org/matrix-org/olm). Also, in current days, as far as I understand, it would seem that the matrix team is promoting the rust implementation (code here https://github.com/matrix-org/vodozemac), which also can be used in the web via wasm. Maybe there is some plan upcoming for pantalaimon, to use the rust olm, or be re-written in rust? No idea, and cannot write these languages, but maybe a new direction…

Errors

If `pantalaimon` is stopped after `ement` was run:

Debugger entered--Lisp error: (ement-api-error "Ement: Network error: 7: Failed to connect to host..." #s(plz-error :curl-error (7 . "Failed to connect to host.") :response nil :message nil))
  signal(ement-api-error ("Ement: Network error: 7: Failed to connect to host..." #s(plz-error :curl-error (7 . "Failed to connect to host.") :response nil :message nil)))
  #f(compiled-function (plz-error) #<bytecode 0x1850da3abd4753f2>)(#s(plz-error :curl-error (7 . "Failed to connect to host.") :response nil :message nil))
  #f(compiled-function (code) #<bytecode 0xf22048be28f7fbc>)("7")
  plz--sentinel(#<process plz-request-curl> "exited abnormally with code 7\n")

Starting the systemd service

× pantalaimon.service - Pantalaimon E2E Matrix reverse proxy
     Loaded: loaded (�]8;;file://c2/etc/systemd/system/pantalaimon.service/etc/systemd/system/pantalaimon.service�]8;;; enabled; preset: disabled)�]8;;�\
     Active: failed (Result: exit-code) since Tue 2023-11-07 18:24:01 CET; 1min 42s ago
   Duration: 733ms
    Process: 1080463 ExecStart=/usr/bin/pantalaimon (code=exited, status=2)
   Main PID: 1080463 (code=exited, status=2)
        CPU: 612ms

Nov 07 18:24:00 c2 systemd[1]: Started Pantalaimon E2E Matrix reverse proxy.
Nov 07 18:24:01 c2 pantalaimon[1080463]: Usage: pantalaimon [OPTIONS]
Nov 07 18:24:01 c2 pantalaimon[1080463]: Try 'pantalaimon --help' for help.
Nov 07 18:24:01 c2 pantalaimon[1080463]: Error: Homeserver is not configured.
Nov 07 18:24:01 c2 systemd[1]: pantalaimon.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 07 18:24:01 c2 systemd[1]: pantalaimon.service: Failed with result 'exit-code'.

4www/ement-el-pantalaimon.md

Select an option

No results found

Select an option

No results found

emacs matrix client, with ement.el & pantalaimon (for encryption support)

Background

Installation

Configuration

Trying to make it work

Usage

systemd

Notes

My questions are

Some debug technics:

About `olm` encryption lib

Errors

If `pantalaimon` is stopped after `ement` was run:

Starting the systemd service

4www/ement-el-pantalaimon.md

emacs matrix client, with ement.el & pantalaimon (for encryption support)

Background

Installation

Configuration

Trying to make it work

Usage

systemd

Notes

My questions are

Some debug technics:

About olm encryption lib

Errors

If pantalaimon is stopped after ement was run:

Starting the systemd service

About `olm` encryption lib

If `pantalaimon` is stopped after `ement` was run: