Other articles in this series:

• Monal Internals - Handlers framework
• Monal Internals - XML Query Language

Serializable Promise Framework

XMPP as a protocol is, as most protocols are, inherently asynchronous. In Monal we therefore use the popular PromiseKit library to let the UI know when a XMPP action finished or failed.

But this has a huge drawback: PromiseKit-based promises aren’t serializable, so we can’t respond to events that got handled by the Notification Service App Extension while the app was suspended. The serializable promise framework is a new framework in Monal, which exactly fills that gap.

Overall Process

For instance, imagine you want to remove the avatar of a group chat in Monal, while on a slow, unreliable network.

1. You submit the request, and the loading screen appears over the view (0:03)
2. Because the network is slow, the loading screen persists for a long time
3. You get frustrated and switch apps, sending Monal to the background (0:06)
4. After 30s in the background, Monal gets suspended (0:36)
5. The server fails to remove the avatar and sends an error to Monal
6. The response is handled by Monal’s app extension in the background (0:51)
7. You switch apps back to Monal (0:57)

Without the promise framework, the app would have continued to show the loading screen indefinitely, requiring you to fully close and open the app.

However, with the promise framework, when you switch back to the app, the loading screen disappears and correctly shows the error returned by the server:

The promise framework allows any such interaction, where the UI has to update in response from the server, to be handled in a general way.

Background

We will briefly describe the important parts below. But these links provide more detailed context and are recommended reading:

• The app decomposition slides explain the split between the main app and app extension.
• The handler framework blog post explores some of the consequences of this split.

Main app and app extension

In Monal there are two separate processes: the “main app” (MonalAppDelegate) and “app extension” (NotificationService), which is sometimes abbreviated to “appex”.

Since they do not share memory, they hand over to each other via the handler framework.

An xmpp stanza can be processed by either, depending on whether the app is running or not:

App states

Below is a slightly simplified diagram of the app’s lifecycle. A more detailed lifecycle from Apple’s perspective is available here.

• The app moves from “active” to “in background” when it is swiped away without being closed
  • It continues to run and process stanzas for ~30s in the background
• Once the 30s are complete, it moves to “suspended”
  • Apple saves the UI’s state
  • When the app is reopened, it returns to the same screen as before
  • Any stanzas that were processed in the meantime were processed by the appex

Appex states

Meanwhile, the appex has a more simple lifecycle - it is either running, or not running:

• Once the appex starts, it will connect to the server and run for 30s
• If the app is opened while the appex is running, the appex will return to “not running” state in deference to the app.

Serializable promise framework

Ok, now the background is out of the way, time to explain the promise framework!

While promises are very useful in SwiftUI, there is one problem: they only exist in that particular process, and cannot easily be passed between the app and appex.

This means that, if the promise was resolved in the appex, once the app is reopened (moves from suspended to active), it will not be consumed. This is because the appex has no way of tying the result back to the promise that updates the UI.

Since the handler framework is what allows the seamless handoff between the app and appex, we need a way to trigger consumption of a promise as a result of processing in the handler. This is precisely what the promise framework does!

High-level overview

We create a “serializable promise” class called MLPromise. This class stores a “UI promise” which is an AnyPromise from PromiseKit. Consumption of this AnyPromise is what ultimately causes the UI to update.

Whenever we want to bind a UI action to the result of a handler, we create an MLPromise and pass it as an argument to the handler. We then return the MLPromise’s AnyPromise to the UI.

Unlike the AnyPromise it contains, the MLPromise is (mostly) serializable. When the MLPromise is created, and whenever it is resolved, it persists itself to a new promises table in the database.

Meanwhile, whenever the app or appex is unfrozen, and the promise table is read into memory, the resolved arguments of any already existing promises are overwritten.

Note that this persisted version of the MLPromise is not complete - while the resolved argument can be persisted, the UI promise is not persisted at any stage.

This means that, while the resolved argument can be passed betwen the app and appex, the UI promise itself cannot. The consequence of this is that a promise cannot be consumed in the appex - it can only be resolved there, leaving consumption for when the app becomes active again:

This makes some sense, as the whole purpose of the AnyPromise is to update the UI as a result of some backend action. It only makes sense to update the UI from the process which manages the UI - the app itself.

As a result, whenever the app becomes active again, all outstanding MLPromises check the version retrieved from the DB if they have been resolved in the meantime, and if so, consume themselves. This is how the loading overlay gets removed in the above example.

Step-by-step overview

Let’s return to the initial scenario of removing the avatar of a group chat. The user chooses the new avatar and presses submit. Then, the following happens:

(1) UI code calls backend

The UI code responding to requests to remove the avatar calls the backend method to do this.

Once the backend function completes, it will return an AnyPromise, allowing the UI code to continue immediately:

showPromisingLoadingOverlay(overlay, headlineView:Text("Removing avatar..."), descriptionView:Text("")) {
    // this returns an AnyPromise used by the loading overly to hide itself once it gets resolved
    self.account.mucProcessor.publishAvatar(nil, forMuc: contact.contactJid)
}

(2) Core code creates promise

The core code creates an MLPromise:

-(AnyPromise*) publishAvatar:(UIImage* _Nullable) image forMuc:(NSString*) room
{
    MLPromise* promise = [MLPromise new];
    // ...

(3) Core code creates handler and (4) sends IQ

The core code creates a new handler, and passes the MLPromise to it as an argument. Recall that handlers serialize their arguments - this means that, even if the app is suspended and the handler is called in the appex, the MLPromise will be available.

[_account sendIq:vcard withHandler:$newHandlerWithInvalidation(self, handleAvatarPublishResult, handleAvatarPublishResultInvalidation, $ID(room), $ID(promise))];

(5) Promise returned to UI

Internally, the MLPromise has created an AnyPromise. This is a type provided by PromiseKit that can be returned directly to the UI, and that the SwiftUI code understands. It gets returned to the UI code.

return [promise toAnyPromise];

(6) XMPP server provides response

This happens on the server side and is not relevant to Monal.

However, while waiting for the response, the user puts the app into the background, and the app gets suspended.

(7) Handler resolves promise

Now, the response from the server activates the appex, since the app was suspended.

The handler is called with the response from the server. It resolves the promise (via either reject or fulfill) - passing that response to the promise.

$$instance_handler(handleAvatarPublishResult, account.mucProcessor, $$ID(xmpp*, account), $$ID(XMPPIQ*, iqNode), $$ID(MLPromise*, promise))
    if([iqNode check:@"/<type=error>"])
    {
        // ...
        [promise reject:error];
        // ...
    }
    // ...
    [promise fulfill:nil];
$$

(8) Promise waits until app is open

If the promise had been resolved from the app, it could have been consumed immediately. However, since it was resolved from the appex, the AnyPromise was not available to call at that time.

Therefore, the promise does not consume itself yet - it instead waits for the app to return to active state.

This waiting is performed by the following observer - whenever either the app or appex is unfrozen, the deserialize method is called:

-(instancetype) init
{
    // ...
    [[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(deserialize) name:kMonalUnfrozen object:nil];
    // ...
}

deserialize calls attemptConsume on each run. attemptConsume checks if we are inside the appex (in which case we do not consume the promise), and only if we are inside the app does it consume the promise:

-(void) attemptConsume
{
    DDLogDebug(@"Intend to consume promise %@ with uuid %@ and argument %@", self, self.uuid, self.resolvedArgument);

    if([HelperTools isAppExtension])
    {
        DDLogDebug(@"Not consuming promise %@ with uuid %@ as we are in the app extension", self, self.uuid);
        return;
    }

    // ...
}

Note that both reject and fulfill also call attemptConsume after resolving the promise. This would let the promise be consumed immediately if we are inside the app instead of the appex.

(9) Promise consumes UI promise

Once we progress past the checks in attemptConsume, we finally consume the promise by calling the resolve callback tied to the PromiseKit AnyPromise returned earlier. This in turn prompts the UI to hide the loading overlay.

-(void) attemptConsume
{
    // ...
    PMKResolver resolve = _resolvers[self.uuid];
    // ...
    resolve(self.resolvedArgument);
    // ...
}

Other articles in this series:

• Monal Internals - Handlers framework
• Monal Internals - Serializable Promise framework

The MLXMLNode methods

All incoming and outgoing XMPP stanzas are parsed to/from an instance of nested MLXMLNode elements. This class therefore provides some methods for creating such elements as well as querying them. In this chapter I want to briefly introduce some parts of the MLXMLNode interface before diving into our XML Query Language in the next chapter.

Creating an MLXMLNode

There are several initializers for MLXMLNode:

-(id) initWithElement:(NSString*) element;
-(id) initWithElement:(NSString*) element andNamespace:(NSString*) xmlns;
-(id) initWithElement:(NSString*) element andNamespace:(NSString*) xmlns withAttributes:(NSDictionary*) attributes andChildren:(NSArray*) children andData:(NSString* _Nullable) data;
-(id) initWithElement:(NSString*) element withAttributes:(NSDictionary*) attributes andChildren:(NSArray*) children andData:(NSString* _Nullable) data;
-(id) initWithElement:(NSString*) element andData:(NSString* _Nullable) data;
-(id) initWithElement:(NSString*) element andNamespace:(NSString*) xmlns andData:(NSString* _Nullable) data;

The initializers not taking a namespace argument will create XML nodes that automatically inherit the namespace of their containing node, once added to a tree of XML nodes.

When nesting MLXMLNodes , it looks like this:

MLXMLNode* exampleNode = [[MLXMLNode alloc] initWithElement:@"credentials" andNamespace:@"urn:xmpp:extdisco:2" withAttributes:@{} andChildren:@[
    [[MLXMLNode alloc] initWithElement:@"service"  withAttributes:@{
        @"type": service[@"type"],
        @"host": service[@"host"],
        @"port": service[@"port"],
    } andChildren:@[] andData:nil]
] andData:nil]

Querying a (possibly nested) MLXMLNode

All XML queries are implemented as an interface of MLXMLNode as well. For XML queries this class has three different methods:

-(NSArray*) find:(NSString* _Nonnull) queryString, ... NS_FORMAT_FUNCTION(1, 2);
-(id) findFirst:(NSString* _Nonnull) queryString, ... NS_FORMAT_FUNCTION(1, 2);
-(BOOL) check:(NSString* _Nonnull) queryString, ... NS_FORMAT_FUNCTION(1, 2);

find: will return an NSArray listing all results matching your query, findFirst: will only return the first result of your query (or nil if the resulting NSArray was empty). This should be used, if you are certain that there should only be one element matching (or none at all). check: can be used to determine if find: would return an empty NSArray.

All three methods take a string argument possibly containing printf-style format specifiers including the %@ specifier as supported by NSString stringWithFormat: and a variable argument list for providing the values for these format specifiers.

The Query Language

To query single values out of a complex XML stanza, we use a XML query language inspired by XPath, but not compatible with it. Instead, our language, as implemented in Monal, is a strict superset of Prosody’s query language as documented in Prosody’s documentation of util.stanza. This makes it possible to copy over queries from Prosody and directly use them in Monal without any modification.

The query language consists of a path followed by an optional extraction command and conversion command and is parsed by complex regular expressions in MLXMLNode.m. These regular expressions and the usage of the xml language throughout Monal were security audited in 2024.

Note: If the following description talks about the find: method, the findFirst: and check: methods are automatically included.

Path Segments

The path is built of /-separated segments each representing an XML node, selected by either an XML namespace or an element name or both. The XML namespace is wrapped in { } and prefixes the element name. Each path segment is used to select all XML nodes matching the criteria listed in this path segment. The special wildcard value * for element name or namespace mean “any namespace” or “any element”.

If the namespace is omitted, the namespace of the parent node in the XML tree the query is acted upon is used (or * , if there is no parent node), see example 0. The namespace of the parent node is used even if the find: method is executed on a child XML node, see example 1. The element name can not be omitted and should be set to * if unknown.

A path beginning with a / is called a rooted query. That means the following first path segment is to be used to select the node the find: method is called on, if the leading / is omitted, the first path segment is used to select the child nodes of the node the find: method is called on.

Note: If using such a rooted query to access attributes, element names etc. of the XML node the whole query is acting upon, both the element name and namespace can be fully omitted and are automatically replaced by {*}*. This allows us to write queries like /@h|int" or /@autojoin|bool.

The special path segment with element name .. not naming any namespace or other selection criteria (e.g. /../) will ascend one node in the XML node tree to the parent of the XML node that the query reached and apply the remaining query to this XML node. Thus using /{jabber:client}iq/{http://jabber.org/protocol/pubsub}pubsub/items/../../../@type will return the value of the type attribute on the root element (the {jabber:iq}iq).

Note: Not using an extraction command (see the next chapter below) will return the matching MLXMLNodes as reference. Changing the attributes etc. of that reference will change the original MLXMLNode in the XML tree it is part of. If you don’t want that, you’ll have to call copy on the returned MLXMLNodes to decouple them from their original.

Example 0:

<message from='test@example.org' id='some_id' xmlns='jabber:client'>
    <body>Message text</body>
    <body xmlns='urn:some:different:namespace'>This will NOT be used</body>
</message>

MLXMLNode* message = <the stanza above as MLXMLNode tree>;
NSArray<NSString*>* bodyStrings = [message find:@"body#"];

MLAssert(bodyStrings.count == 1, @"Only one body text should be returned!");
MLAssert([bodyStrings[0] isEqualToString:@"Message text", @"The body with inherited namespace 'jabber:client' should be used!");

Example 1:

<message from='test@example.org' id='some_id' xmlns='jabber:client'>
    <body>Message text</body>
</message>

MLXMLNode* message = <the stanza above as MLXMLNode tree>;
NSString* messageId = [message findFirst:@"/@id"];

MLAssert([messageId isEqualToString:@"some_id", @"The extracted message id should be 'some_id'!");

More selection criteria

• Not element name:
  If you want to select all XML nodes not having a specified name, you’ll have to prefix the element name with !. This will negate the selection, e.g. !text will select all XML nodes not named text, see example 2.
• Element attribute equals value:
  If you want to select XML nodes on the basis of their XML attributes, you can list those attributes as attributeName=value pairs each inside < >, see example 3. You can use format string specifiers in the value part of those pairs to replace those with the variadic arguments of find:. The order of variadic arguments has to resemble all format specifiers of the complete query string given to find: Note: the value part of those pairs can not be omitted, use regular expression matching to select for mere XML attribute presence (e.g. <attributeName~^.*$>).
• Element attribute matches regular expression:
  To select XML nodes on the basis of their XML attributes, but using a regular expression, you’ll have to use attributeName~regex pairs inside < >. No format string specifiers will be replaced inside your regular expression following the ~. You’ll have to use ^ and $ to match begin and end of the attribute value yourself, e.g. <attributeName~.> will match all attribute values having at least one character, while <attributeName~^.$> will match all attribute values having exactly one character.

Example 2:

<stream:error>
    <not-well-formed xmlns='urn:ietf:params:xml:ns:xmpp-streams'/>
    <text xmlns='urn:ietf:params:xml:ns:xmpp-streams'>Some descriptive Text...</text>
</stream:error>

MLXMLNode* streamError = <the stanza above as MLXMLNode tree>;
NSString* errorReason = [streamError findFirst:@"{urn:ietf:params:xml:ns:xmpp-streams}!text$"];

MLAssert([errorReason isEqualToString:@"not-well-formed"], @"The extracted error should be 'not-well-formed'!");

Example 3 (also using an extraction command, see below):

<iq id='605818D4-4D16-4ACC-B003-BFA3E11849E1' to='user@example.com/Monal-iOS.15e153a8' xmlns='jabber:client' type='result' from='asdkjfhskdf@messaging.one'>
    <pubsub xmlns='http://jabber.org/protocol/pubsub'>
        <subscription node='eu.siacs.conversations.axolotl.devicelist' subid='6795F13596465' subscription='subscribed' jid='user@example.com'/>
    </pubsub>
</iq>

MLXMLNode* iq = <the stanza above as MLXMLNode tree>;
NSString* subscriptionStatus = [iq findFirst:@"/<type=result>/{http://jabber.org/protocol/pubsub}pubsub/subscription<node=%@><jid=%@>@subscription", @"eu.siacs.conversations.axolotl.devicelist", @"user@example.com"];

MLAssert([subscriptionStatus isEqualToString:@"subscribed"], @"The extracted value of the subscription attribute should be 'subscribed'!");

Extraction Commands

An extraction command can be appended to the last path segment. Without those extraction commands, find: will return the full MLXMLNode matching the selection criteria of the XML query. If you rather want to read a special attribute, element value etc. of the full XML node, you’ll have to use one of the extractions commands below

• @attributeName:
  This will return the value of the attribute named after the @ as NSString, use a conversion command to convert the value to other data types.
• @@: This will return all attributes of the selected XML node as key-value-pairs in an NSDictionary. No conversion commands can be used together with this extraction command.
• #: This will return the text contents of the selected XML node as NSString, use a conversion command to convert the value to other data types.
• $: This will return the element name of the selected XML node as NSString. This is only really useful if the last path segment contained a wildcard element name or its element name was negated. A Conversion command can be used to convert the returned element name to other data types as well.

For data-form (XEP-0004) subqueries, see the corresponding section below.

Conversion Commands

Conversion commands can be used to convert the returned NSString of an extraction command to some other data type. Conversion commands can not be used without an extraction command and must be separated from the preceeding extraction command by a pipe symbol (|). The following conversions are currently defined:

• bool:
  This will convert the extracted NSString to an NSNumber representing a BOOL. true/1 becomes @YES and false/0 becomes @NO. This is in accordance to the representation of truth values in XMPP.
• int:
  This will convert the extracted NSString to an NSNumber representing a NSInteger (integerValue property).
• uint:
  This will convert the extracted NSString to an NSNumber representing a NSUInteger (unsignedIntegerValue property).
• double:
  This will convert the extracted NSString to an NSNumber representing a double (doubleValue property).
• datetime:
  This will use the HelperTools method parseDateTimeString: to parse the given NSString into an NSDate object.
• base64:
  This will use the HelperTools method dataWithBase64EncodedString: to parse the given NSString into an NSData object.
• uuid:
  This will try to parse the given NSString into an NSUUID object using the initWithUUIDString initializer of NSUUID. This will return nil for an invalid string, which will omit this result from the NSArray returned by find: (findFirst: will return nil, and check: will return NO).
• uuidcast:
  This will do the same as the uuid conversion command for valid uuid strings, but use the HelperToolsmethod stringToUUID to cast any other given string to a UUIDv4 by hashing it using SHA256 and arranging the result to resemble a valid UUIDv4.

Example 4 (attribute extraction command together with a bool conversion command):

<iq type='result' id='juliet1'>
  <fin xmlns='urn:xmpp:mam:2' complete='true'>
    <set xmlns='http://jabber.org/protocol/rsm'>
      <first index='0'>28482-98726-73623</first>
      <last>09af3-cc343-b409f</last>
    </set>
  </fin>
</iq>

MLXMLNode* iqNode = <the stanza above as MLXMLNode tree>;
if([[iqNode findFirst:@"{urn:xmpp:mam:2}fin@complete|bool"] boolValue])
    DDLogInfo(@"Mam query finished")

Example 5 (attribute extraction command together with a datetime conversion command):

<message from='romeo@montague.net/orchard' to='juliet@capulet.com' type='chat'>
    <body>O blessed, blessed night! I am afeard.</body>
    <delay xmlns='urn:xmpp:delay' from='capulet.com' stamp='2002-09-10T23:08:25Z'/>
</message>

MLXMLNode* messageNode = <the stanza above as MLXMLNode tree>;
NSDate* delayStamp = [messageNode findFirst:@"{urn:xmpp:delay}delay@stamp|datetime"];

MLAssert(delayStamp.timeIntervalSince1970 == 1031699305, @The delay stamp should be 1031699305 seconds after the epoch!");

Some more queries as found in our codebase:

• {urn:xmpp:jingle:1}jingle<action~^session-(initiate|accept)$>
• error/{urn:ietf:params:xml:ns:xmpp-stanzas}item-not-found
• {urn:xmpp:avatar:metadata}metadata/info
• {urn:xmpp:avatar:data}data#|base64

The data-forms (XEP-0004) query language extension

To query fields etc. of a XEP-0004 data-form, the last path segment of an XML query can contain a data-forms subquery. Thes parser for these subqueries is an MLXMLNode extension implemented in XMPPDataForm.m and glued into MLXMLNode.m as the extraction command \ (backslash). This extraction command is also special as it has to be terminated by a \ (optionally followed by a conversion command, see below).

Note: since our query is a string, double backslashes (\\) have to be used because of string escaping rules.

Like other extraction commands, these subqueries must be in the last path segment. Naming the element name and namespace of the node this extraction command is applied to, is optional and automatically defaults to name x and namespace jabber:x:data as defined by XEP-0004.

This query language extension is its own small query language tailored to data-forms implemented in -(id _Nullable) processDataFormQuery:(NSString*) query;. To ease its use, this language reuses some constructs of the main query language, but gives them a new meaning:

• “Namespace” and “element name”:
  The subquery can begin with something looking like a namespace and element name (both optional) like so: {http://jabber.org/protocol/muc#roominfo}result. The “element name” is used to select data forms with this form-type (result in this case). The “namespace” is used to select data-forms with a form field (usually of type hidden) with name FORM_TYPE having this value, see example 6. The special form-type * and FORM_TYPE value * can be used to denote “any form-type” and “any FORM_TYPE field value”.
• Item index:
  This is something not present in the main query language. Between the form-type (the “element name”, see above) and the “extraction command” (see below) an index in square brackets is allowed ([0]). An example query using an index as seen in our codebase would be \\result[0]@expire\\ or \\[0]@expire\\. An index is only allowed for data-forms having multiple item elements encapsulating the form fields, see example 8 of XEP-0004. If the index is out of bounds (e.g. greater than or equal to the count of <item/> XML nodes in the form), the data-form query will return nil, which will be omitted from the resulting NSArray by the MLXMLNode implementation of find: (findFirst: will return nil, and check: will return NO).
• Extraction command:
  Data-Form subqueries have only two extraction commands: @fieldName and &fieldName. @fieldName is used to extract the value of that field, while &fieldName returns an NSDictionary describing that field, like returned with the -(NSDictionary* _Nullable) getField:(NSString* _Nonnull) name; method of XMPPDataForm.

Note: The implementation in XMPPDataForm.m has many useful methods for creating and working with XEP-0004 data-forms. Make sure to check out XMPPDataForm.h or the implementation in XMPPDataForm.m.

Note: An @fieldName extraction command can be used together with a conversion command, see example 6. Conversion commands are not allowed for &fieldName extraction commands or data-form queries not using an extraction command at all (e.g. returning the whole data-form).

Example 6:

<iq from='upload.montague.tld' id='step_02' to='romeo@montague.tld/garden' type='result'>
  <query xmlns='http://jabber.org/protocol/disco#info'>
    <identity category='store' type='file' name='HTTP File Upload' />
    <feature var='urn:xmpp:http:upload:0' />
    <x type='result' xmlns='jabber:x:data'>
      <field var='FORM_TYPE' type='hidden'>
        <value>urn:xmpp:http:upload:0</value>
      </field>
      <field var='max-file-size'>
        <value>5242880</value>
      </field>
    </x>
  </query>
</iq>

MLXMLNode* iqNode = <the stanza above as MLXMLNode tree>;
NSInteger uploadSize = [[iqNode findFirst:@"{http://jabber.org/protocol/disco#info}query/\\{urn:xmpp:http:upload:0}result@max-file-size\\|int"] integerValue];

MLAssert(uploadSize == 5242880, @"Extracted upload size should be 5242880 bytes!");

Some more data-form queries as found in our codebase:

• {http://jabber.org/protocol/disco#info}query/\\{http://jabber.org/protocol/muc#roominfo}result@muc#roomconfig_roomname\\
• {http://jabber.org/protocol/commands}command<node=urn:xmpp:invite#invite>/\\[0]@expire\\|datetime (the form-type and FORM_TYPE field value was omitted, the query matches every data-form)
• {http://jabber.org/protocol/commands}command<node=urn:xmpp:invite#invite>/\\@expire\\|datetime (the form-type and FORM_TYPE field value was omitted, the query matches every data-form)

In short this consists of the following tasks (in no special order).

• Implement Dialpad: Add Dialpad to our Call-UI and backend code to be able to send DTMF tones in A/V calls. This will make Monal fully compatible with jmp.chat, like Snikket is already.
• Rewrite Chat UI: Our current chat UI is still UIKit-based and it’s hard to improve it or fix some UI glitches. We want to rewrite and modernize the whole chat UI using SwiftUI. This will not only simplify maintenance a lot and allow us to fix these UI glitches, but also enable us to implement modern XMPP features like message reactions, message styling, message replies or mentions.
• UI work: Implement Message Reactions, Rich Replies and Stickers: Implement UI and backend for message reactions (XEP-0444), rich replies (XEP-0461) and Stickers, once the chat UI is ported to SwiftUI
• XSF work: After having successfully worked on the SASL2 XEP-suite, I want to modernize XEP-0389: Extensible In-Band Registration to also send only password hashes instead of cleartext passwords (similar to password upgrades specced in XEP-0480: SASL Upgrade Tasks)
• Write documentation of Monal internals: After having started to publish a blog series and wiki articles about Monal’s internals (beginning with the Handlers Framework), I want to publish blog and wiki articles on our XML query-language (intentionally based on the XPath-like syntax of Prosody’s query language), the PubSub/PEP framework, Model-Classes used as data sources for our UI (MLContact, MLMessage etc.) and possibly more.

Thanks again to NLNet for fund this!

So it’s that time again: we are launching a new fundraising campaign for 350 EUR to finance a new development iPhone capable of running iOS 17 and several upcoming iOS versions. Currently we are aiming for an iPhone 13.

You can view our donation options over here: Donate

The results in a nutshell: no security issues found, read the full report here: Monal IM penetration test report 2024 1.0 .

Here is the full mail, a translation of the CAC articles can be found over here for reference.

Hello,

We are writing to notify you that your application, per demand from the CAC (Cyberspace Administration of China), will be removed from the China App Store because it includes content that is illegal in China, which is not in compliance with the App Review Guidelines:

5. Legal Apps must comply with all legal requirements in any location where you make them available (if you’re not sure, check with a lawyer). We know this stuff is complicated, but it is your responsibility to understand and make sure your app conforms with all local laws, not just the guidelines below. And of course, apps that solicit, promote, or encourage criminal or clearly reckless behavior will be rejected.

According to the CAC, your app violates Articles 3 of the Provisions on the Security Assessment of Internet-based Information Services with Attribute of Public Opinions or Capable of Social Mobilization (具有舆论属性或社会动员能力的互联网信息服务安全评估规定).

If you need additional information regarding this removal or the laws and requirements in China, we encourage you to reach out directly to the CAC (Cyberspace Administration of China).

While your app has been removed from the China App Store, it is still available in the App Stores for the other territories you selected in App Store Connect. The TestFlight version of this app will also be unavailable for external and internal testing in China and all public TestFlight links will no longer be functional.

Best regards,

App Review

Enjoy watching the talk recordings and reading the slides and XEPs.

And last but not least: Happy new year!

NEW: Audio-call support (This feature will not be available to users in China and macOS users!)

Other changes:

• New Logo and new placeholder images by Ann-Sophie Zwahlen
• New “Add Contact” and “Contact Requests” UI
• Complete rewrite of OMEMO code
• Speed up app start
• Add support for SASL2 (XEP-0388)
• Implement XEP-0424: Message Retraction
• Add support for creating invitations (button only displayed if your server supports it, see https://docs.modernxmpp.org/client/invites/)
• Add timestamp when quoting older messages
• Always show a “Notes to self“ chat
• Overhaul implementation of last interaction display
• Show scroll-down button in groupchats
• OMEMO keys are copyable now (double tap)
• Add OSS based crash reporting (KSCrash), reports can be voluntarily sent via mail
• Fix logfile handling
• Add XEP-0215 (external services) to server details ui
• Only show contacts in contacts panel if they are in our roster
• Implement invitations using qr codes in addition of xmpp: uris
• Implement new image viewer compatible with iOS 17
• Implement gif support in image viewer

Bugfixes:

• Many bugfixes
• Fix bookmarks2 handling
• Fix XEP-0333 in private groups
• Fix url preview for sites not having oc: tags
• Set notifications to “mention only” when joining public channels
• Show per-resource last interaction timestamp in resource list
• Fix file uploading and sharing
• Fix timer when recording audio messages
• Fix muc avatar fetching

Other articles in this series:

• Monal Internals - XML Query Language
• Monal Internals - Serializable Promise framework

Handlers

Handlers in Monal are something like serializable callbacks. In iOS the app can get frozen or even killed any time and the push implementation requires the complete app state to frequently transition between the Main App and the Notification Service App Extension (NSE). Using normal callbacks (called blocks in ObjC) is not possible because blocks are not serializable which means they could not survive an app kill or transition to / from the NSE.

Short overview:

• Simple generalized concept usable throughout the app
• Leverages dynamic language features of ObjC
• “Serializable callbacks” to class / instance methods of ObjC classes
• Bind values (not vars) when creating a handler (e.g. to bind state to handler that can be serialized together with the handler)
• Bind vars when calling handler (overwriting creation-time bound values having the same name)
• Used in various places throughout the app like the IQ or PubSub framework (see Code examples found in the wild)
• Jump directly to list of supported data types

Usage description with examples

Define handler method

This will be a static class method and doesn’t have to be declared in any interface to be usable. The argument number or order does not matter, feel free to reorder or even remove arguments you don’t need. Arguments declared with $$-prefix are mandatory and can not be removed, though. Arguments with $_-prefix are optional and can be removed. Primitive datatypes like BOOL, int, NSINTEGER (aka $$INTEGER) etc. can not be imported as optional and are always mandatory.

$$class_handler(myHandlerName, $_ID(xmpp*, account), $$BOOL(success))
    // your code comes here
    // variables defined/imported: account (optional), success (mandatory)
$$

Instance handlers are instance methods instead of static methods. You need to specify on which instance these handlers should operate. The instance extraxtion statement (the second argument to $$instance_handler() can be everything that returns an objc object. For example: account.omemo or [account getInstanceToUse] or just account.

$$instance_handler(myHandlerName, instanceToUse, $$ID(xmpp*, account), $$BOOL(success))
    // your code comes here
    // 'self' is now the instance of the class extracted by the instanceToUse statement.
    // instead of the class instance as it would be if $$class_handler() was used instead of $$instance_handler()
    // variables defined/imported: account, success (both mandatory)
$$

Call defined handlers

Calling a defined handler is simple, just create a handler object using $newHandler() and $call() it.

MLHandler* h = $newHandler(ClassName, myHandlerName);
$call(h);

Bind variables when creating a handler

You can bind variables to MLHandler objects when creating them and when invoking them. Variables supplied on invocation overwrite variables supplied when creating the handler if the names are equal. Variables bound to the handler when creating it have to conform to the NSCoding protocol to make the handler serializable.

NSString* var1 = @"value";
MLHandler* h = $newHandler(ClassName, myHandlerName,
        $ID(var1),
        $BOOL(success, YES)
}));
xmpp* account = nil;
$call(h, $ID(account), $ID(otherAccountVarWithSameValue, account))

Usable shortcuts to create MLHandler objects:

• $newHandler(ClassName, handlerName, boundArgs...)
• $newHandlerWithInvalidation(ClassName, handlerName, invalidationHandlerName, boundArgs...)

Using invalidation handlers

You can add an invalidation method to a handler when creating the MLHandler object (after invalidating a handler you can not call or invalidate it again!). Invalidation handlers can be instance handlers or static handlers, just like with “normal” handlers:

// definition of normal handler method as instance_handler
$$instance_handler(myHandlerName, [account getInstanceToUse], $_ID(xmpp*, account), $$BOOL(success))
        // your code comes here
        // 'self' is now the instance of the class extracted by [account getInstanceToUse]
        // instead of the class instance as it would be if $$class_handler() was used instead of $$instance_handler()
$$

// definition of invalidation method
$$class_handler(myInvalidationHandlerName, $$BOOL(done), $_ID(NSString*, var1))
        // your code comes here
        // variables imported: var1, done
        // variables that could have been imported according to $newHandler and $call below: var1, success, done
$$

MLHandler* h = $newHandlerWithInvalidation(ClassName, myHandlerName, myInvalidationHandlerName,
        $ID(var1, @"value"),
        $BOOL(success, YES)
}));

// call invalidation method with "done" argument set to YES
$invalidate(h, $BOOL(done, YES))

Available data types

The following datatypes can be bound to a handler defining them using the corresponding definitions having either the $$ or $_ prefix. If the single argument binding is used, the value is bound using the same name like the var the value is coming from (in this example: name).

• define: $$ID(NSObject*, name), $_ID(NSObject*, name), bind: $ID(name), $ID(name, value)
• define: $$HANDLER(name), $_HANDLER(name), bind: $HANDLER(name), $HANDLER(name, value)
• define: $$BOOL(name), bind: $BOOL(name), $BOOL(name, value)
• define: $$INT(name), bind: $INT(name), $INT(name, value)
• define: $$DOUBLE(name), bind: $DOUBLE(name), $DOUBLE(name, value)
• define: $$INTEGER(name), bind: $INTEGER(name), $INTEGER(name, value), this corresponds to the NSInteger type alias
• define: $$UINTEGER(name), bind: $UINTEGER(name), $UINTEGER(name, value), this corresponds to the NSUInteger type alias

Code examples found in the wild

Enabling carbons

if([features containsObject:@"urn:xmpp:carbons:2"])
{
    DDLogInfo(@"got disco result with carbons ns");
    if(!account.connectionProperties.usingCarbons2)
    {
        DDLogInfo(@"enabling carbons");
        XMPPIQ* carbons = [[XMPPIQ alloc] initWithType:kiqSetType];
        [carbons addChildNode:[[MLXMLNode alloc] initWithElement:@"enable" andNamespace:@"urn:xmpp:carbons:2"]];
        [account sendIq:carbons withHandler:$newHandler(self, handleCarbonsEnabled)];
    }
}

$$class_handler(handleCarbonsEnabled, $$ID(xmpp*, account), $$ID(XMPPIQ*, iqNode))
    if([iqNode check:@"/<type=error>"])
    {
        DDLogWarn(@"carbon enable iq returned error: %@", [iqNode findFirst:@"error"]);
        [HelperTools postError:[NSString stringWithFormat:NSLocalizedString(@"Failed to enable carbons for account %@", @""), account.connectionProperties.identity.jid] withNode:iqNode andAccount:account andIsSevere:YES];
        return;
    }
    account.connectionProperties.usingCarbons2 = YES;
$$

Fetching OMEMO bundles

NSString* bundleNode = [NSString stringWithFormat:@"eu.siacs.conversations.axolotl.bundles:%@", deviceid];
[self.account.pubsub fetchNode:bundleNode from:jid withItemsList:nil andHandler:$newHandler(self, handleBundleFetchResult, $ID(rid, deviceid))];

$$instance_handler(handleBundleFetchResult, account.omemo, $$ID(xmpp*, account), $$ID(NSString*, jid), $$BOOL(success), $_ID(XMPPIQ*, errorIq), $_ID(NSString*, errorReason), $_ID((NSDictionary<NSString*, MLXMLNode*>*), data), $$ID(NSString*, rid))
    if(!success)
    {
        if(errorIq)
            DDLogError(@"Could not fetch bundle from %@: rid: %@ - %@", jid, rid, errorIq);
        else
            DDLogError(@"Could not fetch bundle from %@: rid: %@ - %@", jid, rid, errorReason);
    }
    else
    {
        // check that a corresponding buddy exists -> prevent foreign key errors
        MLXMLNode* receivedKeys = [data objectForKey:@"current"];
        if(!receivedKeys && data.count == 1)
        {
            // some clients do not use <item id="current">
            receivedKeys = [[data allValues] firstObject];
        }
        else if(!receivedKeys && data.count > 1)
        {
            DDLogWarn(@"More than one bundle item found from %@ rid: %@", jid, rid);
        }
        if(receivedKeys)
        {
            [self processOMEMOKeys:receivedKeys forJid:jid andRid:rid];
        }
    }
$$

These video tutorials can be found over here: https://www.eversten.net/videotutorials/.
Currently the audio of these videos is German only.

Monal Website
You can find our latest privacy policy for our website here: Website Privacy Policy

Monal App
Our privacy policy may differ between app versions. Before reading our privacy policy for our App you first need to find out the Monal version that you are using.

How to find out your Monal version

1. Open Monal
2. Open up the settings menu in the upper left corner (gearwheel)
3. Scroll down to the last entry “version”

Monal App Privacy Policies

Privacy Monal App ≥ 6.0.0

TLDR

• We never see your messages.
• We do not know who you are chatting with.
• We can not identify a user.
• We can see your XMPP domain and a Monal specific unique device id every time you receive a push message
• We see your IP addresses if you are on a call and your XMPP server does not provide a STUN or TURN server.
• We may see your contact’s IP address if you are using our TURN server.

Structure

The App Monal may interact with Monal servers to support Push messages or if you are establishing a call with a contact but your XMPP Server does neither provide a STUN nor a TURN server.

Our privacy details are structured as follows. First, we would like to give you a short introduction how Monal is handling push messages to ensure a pleasant user experience. We will then briefly explain VoIP calls and its privacy implications. Afterwards we like to inform you how we are using crash and usage reports, logs and GDPR Subject Access Requests (SAR).

Push

App Resources are very limited on iOS and macOS. Monal for example can only run a limited time in the background after a user either locked the screen or switched the app. Hence, apps on iOS and macOS can not simply keep a connection to your XMPP server open 24/7 to inform you about new messages. To overcome these limitations your XMPP server can request our push server to send push messages to your device through Apple. With these push messages we can request Apple to wake up Monal on your device. Once it is woken up it has about 30 seconds to connect to your XMPP server, fetch all new messages and show a push notification for these new messages, if needed.

How push works

Every time that Monal loggs in at your XMPP servers, it asks your server to inform our push server once you receive an XMPP message while Monal is closed/disconnected. To do this, we request a Monal specific push token from Apple and provide this token to your XMPP server. Using this Monal specific push token your XMPP server can instruct our push server to send push messages via Apples push system to wake up the app on your device.

Once push messages are enabled for your Monal instance on all your XMPP servers, your XMPP servers will open a encrypted server to server (s2s) connection to one of our push servers. Using this s2s connection your XMPP servers will then be able to talk to our push servers. To wake up your Monal instance your XMPP servers send us:

• Your unique Monal specific push token that was generated by Apple
• The domain of the XMPP server that you are using

Push

• We never see your messages.
• We do not know who you are chatting with.
• We could only ever track what XMPP domains a push token is/was using.
• We can not identify a user.

Push-Servers

We currently provide the following independent push server regions:

• Europe
• Alpha (based in Europe, only used for debugging with higher log levels, not for production use)

Note: Our previously used US push region was unfortunately shutdown due to fosshost ceasing operation.

How to change the push region

1. Open Monal
2. Open up the settings menu in the upper left corner (gearwheel)
3. Open the Notifications menu
4. Scroll down
5. Select a region

Push server regions

If you are an XMPP server administrator, and you restricted s2s connections, please allow s2s to all our regions.

Push server locations

VoIP (STUN / TURN)

With Monal 6.0 we introduced VoIP support. To establish the connection between you and the remote party (the remote contact) Monal utilizes STUN and TURN. In general STUN (Session Traversal Utilities for NAT) is used to allow a VoIP call even when you are behind firewalls.

Calls established using only STUN will directly exchange packets (P2P) between you and your contact. Hence, your contact may see your IP address. If you do not want your contact to see your IP Address while being on a call, disable P2P connection in Monal’s privacy settings. Once disabled Monal tries to establish the call using a TURN (Traversal Using Relays around NAT) server.

Note: Not all XMPP servers currently provide STUN and TURN servers. If your XMPP server does not provide STUN and TURN servers, Monal may use our fallback servers. These fallback servers provide both STUN and TURN. You can disable Monal to use these fallback turn servers. Please note, that we may disable our fallback STUN and TURN servers at any time, if too many users are using them.

If you use our fallback servers we will see:

• Your IP Addresses
• The IPs of your contact or the IPs of their TURN-Server
• The duration of the call

We will not see the contents of that call, because these are E2E encrypted.

Crash reports and app usage

Monal does track crashes and usage data anonymously using the tools provided by Apple. This is opt-in only and controlled by iOS and macOS global settings. If a user decides not to send any data to developers, no crash logs are sent to Monal developers.

Logs

Your local device will contain a log file with all sent and received raw XMPP messages as well as debug logs. It does contain sensitive personal data! This file will never be transferred to us, except if you explicitly (manually) send it to us (e.g., via mail).

GDPR Subject Access Requests (SAR)

European GDPR allows users to request a copy of all data retained about them. Starting with Monal 5.2.0 we no longer see your JIDs (username@domain.tld) in our push servers. We therefore are not able to send you retained data related to your JID. We furthermore are unable to provide your retained data related to your unique push token because we have no way to verify that Apple issued you a provided token. If you have questions regarding GDPR, please send us a mail to info@monal-im.org.

UPDATE: This post describes the way Monal implements push as well as the pitfalls of implementing it in another way. I tried to make this clearer in the text now.

Background knowledge: Push on iOS

On iOS there are several push modes having different properties. All modes, except VoiIP pushes, have in common, that they only provide 30 seconds of background time.

• VoIP pushes: MUST always make the device ring via Apple’s CallKit framework. You can’t use these pushes to silently retrieve messages or other XMPP stanzas in the background.
• Low priority pushes: These pushes don’t show any user-visible notification, but they can be dropped or arbitrarily delayed by Apple. They grant 30 seconds of background time.
• High priority pushes: These MUST show a user-visible notification. They grant 30 seconds of background time.
  • But: Since iOS 13.3, apps having the com.apple.developer.usernotifications.filtering entitlement are allowed to suppress the user-visible notification. To cite Apple:

    This entitlement is intended for certain types of apps — such as messaging apps or location sharing apps — that use notification service extensions to receive push notifications without delivering notifications to the user.

High priority pushes in combination with the com.apple.developer.usernotifications.filtering entitlement are not only used by Monal, but some other popular messaging apps, too (links to the source):

• WhatsApp
• Signal
• Threema
• Telegram (binary)

Monal

Monal is using this entitlement to wake up background processing without sending any message data through Apple’s servers (not even ecrypted data). When iOS grants the app it’s 30 seconds background time because of the incoming push, Monal then connects to the XMPP server over the network to retrieve the actual message and display a notification to the user, if needed.

Signal does exactly the same: It uses push only to wake up the app and then fetches the messages using a dedicated network connection: Example in Signal’s code

This way Monal avoids all of the pitfalls depicted below!

Diving deeper: Some more details about iOS push and XMPP

The following explanations and thoughts are a bit more technical and require some understanding of the XMPP protocol and it’s inner workings.

Push server implementations and iOS time limits

Pushes of all types (see above) can only wake up the iOS app for 30 seconds. But in most cases that’s more than enough to connect to the xmpp server and retrieve pending stanzas (if using the entitlement mentioned above and XEP-0198, it is even possible to get pushes for iq stanzas etc., thus behaving like being permanently connected while still sleeping most of the time because of CSI). Even if the 30 seconds don’t suffice, the client can disconnect and both, Prosody and eJabberd, will send another push if there are still unacked stanzas in the XEP-0198 queue. This will give the app another 30 seconds. Even longer catchups lasting for > 5 minutes can be done completely in the background this way (observed in the wild with Monal stable).

This means that even with the 30 second time limit in place, it usually is possible to do a longer lasting catchup completely in the background, effectively extending the time limit to several minutes. This obviously only holds, if you use the com.apple.developer.usernotifications.filtering entitlement and connect directly to the xmpp server and is hardly possible if you try to transport stanzas through the push sent through Apple (see below).

Pitfalls: Transporting (encrypted) xmpp message stanzas through push (out of band)

When using the iOS push infrastructure provided by apple for transporting (encrypted) stanzas out of band, multiple things have to be considered. First of all, pushes can get lost. That frequently happens if the device was in flight mode while the push was sent. Second, xmpp is a streaming protocol, strongly relying on the ordering of stanza (even inter-type ordering like the ordering of message and iq stanzas for mam). The order of message stanza matters for other XEPs, too (for example message retraction or last message correction). Using the iOS push service which is loosing pushes or even only sending pushes for a particular type of stanza (message stanzas having a body for example) breaks this ordering of events that every XEP implicitly relies upon. The payload sizes allowed for each push are also somewhat low (~4kb including some of the push metadata). OMEMO’s self-healing through key-transport-elements requires the client to send those key-transport message stanzas once a broken session gets detected. But using the push service for data transport is a one way channel only.

Last but not least the UX can be really degraded if a user does not open the app for some hours (while still receiving push notifications). Once they open the app, a long mam catchup has to be waited for until the ui does reflect what the user already saw in notifications. If the device has no connectivity when the user opens it, they might even be really confused why those messages they alread saw a notification for are not displayed in the app at all). Writing incoming messages to the database to solve this will only make the ordering problem depicted above harder. The client not being able to execute the OMEMO self-healing stuff mentioned above will degrade UX further.

NOT using the entitlement mentioned above will prevent the client from receiving push notifications for XEP-0333 read markers and remove already displayed notifications instead of posting a new one (because apps not having that entitlement are forced to show a notification for each incoming high priority push, even if the push was only for a XEP-0333 read marker). To be clear: without that entitlement neither message retraction nor XEP-0333 read markers can be implemented reliably!
And if an app has the entitlement: why bother delivering some stanzas out of band and running into the ordering problem if the app could as well connect to the xmpp server in the background and retrieve all pending stanzas in the right order?

Update (2023-04-21): Since I originally wrote this blog post, I’ve had the ability to discuss several of my ideas with Dave (the original author of XEP-0388 dubbed SASL2), MattJ (one of the authors of the prosody xmpp server) and others. Most, if not all, of my issues are now addressed in a bunch of updates to existing XEPs as well as new XEPs:

• Update of XEP-0388 (SASL2): https://xmpp.org/extensions/xep-0388.html
• New SASL2- and SCRAM-upgrade XEP: https://xmpp.org/extensions/xep-0480.html
• New XEP for downgrade protection when using SCRAM: https://xmpp.org/extensions/xep-0474.html
• Already existing XEP that only now gets adopted by server developers: XEP-0440

The rest of this blog post remains as is and can be used for a deeper introduction into the material and as explanation of some of the rationale behind the SASL2 updates and my ProtoXEP.

A Modern Authentication Protocol

In my opinion a modern authentication protocol should have at least the following properties. Of course that can be subject to debate, but I think most of us will agree on the following list.

1. allow for protocol agility (e.g. adding new authentication mechanisms, like adding new cipher suites in TLS)
2. prevent downgrade attacks on authentication mechanisms (we don’t want an active attacker to be able to force us to use a weak mechanism)
3. prevent storage of plaintext passwords on the server (that’s obvious: we don’t want a hacker to be able to steal our plaintext passwords once he hacked the database)
4. prevent replay attacks (we don’t want a MITM (man in the middle) be able to steal a (possibly hashed) password and use it to authenticate herself)
5. (possibly) prevent/detect MITM altogether

How SASL tries to fulfill these properties

First of all: XMPP Core mandates the use of TLS for everything, including authentication. Keep that in mind, when reading the rest of this blog post.

Property 1

SASL as defined for XMPP allows the server to present a list of authentication methods. The client then picks the one having the highest perceived strength (see XMPP-Core 6.3.3) among the ones it implements. XEP-0438: Best practices for password hashing and storage lists some common authentication methods and how they should be ordered. Currently, the methods PLAIN (plaintext password), EXTERNAL (using client certificates to authenticate the user) and SCRAM (Salted Challenge Response Authentication Mechanism) are common.

(I will not talk about SASL EXTERNAL in this blog post, because it does not use passwords and seems to be super uncommon in the XMPP world.)

This generally allows adding new mechanisms in the future, protocol agility seems to be fulfilled, right?

Property 3

Current mandatory SASL methods include SCRAM-SHA-1 and SCRAM-SHA-1-PLUS (if possible). SCRAM generally allows the server to store a salted hash instead of plaintext passwords.

Even if the client uses the PLAIN method, the server could store the password as salted hash.
And EXTERNAL usually does not use any form of password.

So that’s another property of our list that is fulfilled, right?

Property 4

Using SCRAM it is even possible to prevent replay attacks because of the used challenge-response scheme. That’s even possible if no TLS channel is used.

Cool, another property that’s fulfilled by SCRAM as well, right?

Property 5

SCRAM is even cooler because it allows for TLS channel-binding.
This allows both entities (client and server) taking part in the SCRAM authentication to bind the authentication to the TLS channel using a HMAC (Hash-Based Message Authentication Code) to sign a unique data blob tied to the TLS session in use.

If the TLS channel is intercepted by a MITM, the attacker would have to use two separate TLS channels, one to the server and one to the client.
Binding the authentication to the TLS channel allows the server and client to detect such an attacker and fail the authentication.

To indicate, that a server supports channel-binding, it appends the string -PLUS to the advertised SCRAM methods.
If the client supports channel-binding, it picks SCRAM-SHA-1-PLUS instead of SCRAM-SHA-1.
In case the client supports channel-binding, but only received methods without channel-binding, the client uses the SCRAM method without -PLUS, but also indicates that it would have used the PLUS varriant if offered by the server (SCRAM Channel Binding).
This allows for the server to detect downgrade attacks and fail the authentication.
Because there are multiple different kinds of channel-binding to TLS possible, the client also specifies which binding it uses during the SCRAM flow (protocol agility).

Channel binding prevents MITM altogether, right? Another property that’s fulfilled!

What’s broken with SASL in XMPP

Cautious readers will have noticed, that I left out property 2 (downgrade attack prevention) in my above explanation. That’s because SASL does not prevent downgrade attacks regarding the method negotiation at all. And that’s one of the main reasons why the whole SASL in XMPP stuff is so horribly broken.

Downgrade of SASL methods

The XMPP Core RFC even mentions downgrade attacks and suggests using TLS to mitigate them. But that’s not enough. If we assume the TLS channel to always be secure and MITM-free, we don’t even need SCRAM but could solely use PLAIN. The TLS channel already ensures no replay attacks can happen and storing the passwords securely using salted hashes on the server is still possible. We don’t need any channel-binding either, because that’s only needed if we assume someone has tampered with the TLS channel in the first place. In this scenario MITM-Prevention (property 5) is simply out of scope for SASL, because we assume TLS to always be MITM-free. That means our properties mentioned above are all fulfilled (property 5 by definition, the other ones by use of TLS) even if we abandon SCRAM and solely use PLAIN.

If we, on the other hand, don’t assume to always have a MITM-free TLS channel, then the above listed properties 2, 4 and 5 are all not fulfilled (that means: no downgrade prevention, no replay attack prevention and no MITM detection/prevention). The attacker could simply remove every advertised SASL method except PLAIN and thus get the plaintext password which is replayable and neither the server nor the client will detect this MITM. Supporting SCRAM in clients and servers does not help at all with this, because it simply will not be negotiated.

Well, okay, but XEP-0438: Best practices for password hashing and storage (and some RFC I don’t remember) says, we should pin the last used SASL mechanism in the client to prevent downgrade attacks in further SASL sessions. That way the client won’t use SASL mechanisms having a perceived lower strength than the pinned one. Using a stronger one is still possible. That’s right, but it makes matters worse in regard to protocol agility while still not preventing the downgrade attack for the first connection of a client to the XMPP server.

Broken protocol agility

Protocol agility means we can specify new authentication methods later on and our client will always use the best one advertised by the server. That’s important because it allows us to gradually upgrade the security strength of our authentication while still maintaining backwards compatibility with older clients, eventually removing an old authentication method once most/all clients use a newer one (like replacing DIGEST-MD5 or CRAM-MD5 with SCRAM-SHA-XXX).

SCRAM without PLUS variants

Because of SCRAM fulfilling property 3 (preventing storage of plaintext passwords on the server), we can not upgrade the stored password hashes in the server’s database to a new SCRAM-based SASL mechanism. The obvious partial solution is to store new user’s passwords using the newer SCRAM algorithm and leave the old user’s passwords like they are. That way at least some of your users get to use the new SCRAM algorithm, slowly phasing out the old one eventually. An example would be a server that previously only supported SCRAM-SHA-1 now advertising support for SCRAM-SHA-256.

Oh, no! That doesn’t work either! The current specification of SASL in XMPP does send the list of supported SASL methods to the client before knowing which username the client wants to authenticate for. That means the server will always advertise SCRAM-SHA-256, even if the hash in the database is still SHA-1. If the client supports SCRAM-SHA-256 as well, it will happily pick that one and the server, only having the SHA-1 hash at hand, won’t be able to authenticate the user. The client on the other hand will have no way to detect why the authentication failed and switch to a percieved lower strength SASL algorithm (and even if that would be possible, it could possibly be used as a downgrade vector if done wrong).

Well, method pinning to the rescue! We already talked about SASL method pinning. The client obviously knows which SASL method it used the last time it authenticated successfully and can always use just this method, no other, even if it implements some having a higher strength. That extends the pinning described in XEP-0438: Best practices for password hashing and storage to algorithms having a higher strength as well, something that, to my knowledge, isn’t specified anywhere. Additionally, that means we completely loose protocol agility after our first authentication. And newly offering SCRAM-SHA-256 on a server not storing plaintext passwords after it previously only advertised SCRAM-SHA-1 will likely still break authentication for all clients not doing this exact “always use the mechanism used on first auth” pinning.

The only way to advertise support for a new SCRAM hash algorithm and make clients use it is to upgrade your complete password database on the server by forcing a password reset upon all of your existing users. This must presumably be done out of band for XMPP. Clients implementing the strict pinning outlined above will have to reset the pinning once a new password gets entered by the user. If they don’t do so, this strict pinning to the old algorithm will still be in place and this client won’t be able to authenticate the user after the password reset.

And I’ve not even started to talk about a user having two clients, one that only supports an old SCRAM algorithm (say SCRAM-SHA-1) and one that supports a new one (say SCRAM-SHA-256). The server obviously must store both hashes in its database to allow logins for both clients.

Sounds all pretty bad, right? How come, that hasn’t been discovered yet? Well, someone already identified these problems back in 2019. See this thread on the standards@xmpp.org mailinglist. But no attempt was made to fix them. Even the new XEP-0388: Extensible SASL Profile still uses the same protocol flow with no fix, albeit allowing for additional handshakes during the authorization phase like pipelining a bind request or resumption via XEP-0198: Stream Management onto the authorization.

Sidenote: One could solve this mess by sacrificing property 3 (not storing plaintext passwords on the server). If the plaintext password is stored on the server, every SCRAM algorithm can be used by the client. But do we really want that? At least that would allow the server to advertise new SCRAM algorithms without having to force a password reset.

SCRAM with PLUS variants

When looking at the channel-binding situation we already saw that the client specifies the type of channel-binding it wants to use. That allows for protocol agility, right?

No! The client does not have any way to detect which channel-binding type the server supports (appending -PLUS to the advertised SCRAM algorithm does not in any way specify which channel-binding to use). And if the client uses the wrong channel-binding the server does not support, the server will simply fail the authentication. The client will have no way to detect if the authentication failed because of the wrong channel-binding type used or if the actual password was wrong, like with using the “wrong” SASL algorithm above.

This renders the whole channel-binding protocol agility completely useless. And protocol agility is needed even for channel-binding!

Some Solutions

Well, that’s pretty bad news, right? But I don’t want to simply rant and leave the shard for someone else to collect, but propose fixes instead. And to my knowledge some of these problems are really fixable.

In this section I want to propose fixes to at least some of these problems. These fixes are open to debate and if we come up with even better solutions during this debate, that’d be great.

Restoring protocol agility for channel-binding

The server must not use -PLUS to indicate SCRAM algorithms with channel-binding, but use the name of the concrete channel-binding type as SCRAM algorithm suffix. That way the client will be able to determine if it supports that type of channel-binding and only select those SCRAM algorithms having a channel-binding it supports. The server is only allowed to advertise channel binding methods supported by the currently used channel (e.g. don’t advertise tls-unique on a TLS 1.3 channel, but only tls-exporter (if implemented)). Clients and servers may choose to still support the -PLUS SCRAM method names in addition to these new ones containing the concrete channel-binding type, but I don’t think that gets us anywhere.

Some examples: SCRAM-SHA-1_TLS-UNIQUE or SCRAM-SHA-512_TLS-EXPORTER. That, of course, does not help at all, if channel-binding support can be rendered useless by a downgrade attack.

Preventing downgrade of SASL methods

Downgrades can be prevented by only allowing SASL EXTERNAL and SCRAM methods (or something similar to SCRAM that then mutual authenticates the whole protocol flow). The SCRAM client-final message must contain a client proof built using a HMAC not only covering the client-first-message-bare, server-first-message and client-final-message-without-proof, but also the (sorted) SASL method list. That allows the server to verify if the client used the correct list and this could not be manipulated, because it is ultimately signed with the client password, a MITM attacker can not know.

We can achieve this by adding an optional SCRAM attribute to client-first-message-bare which will be ignored by non-supporting servers, but secure the handshake against downgrades on supporting servers. This attribute (let’s name it d) will contain a base64-encoded hash of the sorted SASL method list as received by the client (using the same hash method as used in the whole SCRAM stuff). The server then checks if that hash matches the one it calculated itself by hashing the sorted SCRAM method list it advertised and fail the authentication, if these hashes don’t match.

The sorting can be done alphabetical in ascending order and the individual SASL methods be separated by < like done in XEP-0115: Entity Capabilities before the whole string is hashed.

We now have a working downgrade attack prevention that’s even backwards compatible with existing SCRAM methods and servers not supporting this new SCRAM attribute.

Restoring protocol agility for SASL methods part #1

First of all, the server will have to store SCRAM hashes (or something similar for non-scram methods) for all methods it does support. If the server operator later decides to not offer a specific SASL method anymore, they can delete the (hash) data stored for that method from their server.

Second, the server can only advertise those methods it has hashes for in its database. That means it needs to know the username before advertising which methods it supports. This requires a change in the protocol flow for mechanism negotiation, which is not codified in the SASL RFC and can be changed via XEP. A good candidate would be XEP-0388: Extensible SASL Profile (also dubbed SASL2) which is not yet in Final state and thus can be adjusted to our needs.

Let’s assume a SASL2 protocol flow by advertising support for this protocol, but not simultaneously advertising which SASL methods the server supports. The client would then first send the desired username it wants to authenticate for, and the server would respond with a list of supported SASL mechanisms for exactly this user. The username is of course not safe from a MITM attacker, but it will be included in the SCRAM authentication flow and the server will fail the authentication if that user differs from the one given earlier.

An example protocol flow using a modified SASL2 might look as follows (resembling more or less a normal SCRAM flow):

<!--
  Server sends stream features indicating support for SASL2.
-->
<stream:features>
  <authentication xmlns='urn:xmpp:sasl:1'/>
</stream:features>

<!--
  Client intiates authentication by sending the base64 encoded username it wishes to authenticate for.
-->
<request xmlns='urn:xmpp:sasl:1'>
  <user>dXNlcg==</user>
</request>

<!--
  Server sends the list of supported mechanisms for this user.
  The sorted list will be 'SCRAM-SHA-1<SCRAM-SHA-1_TLS-EXPORTER<SCRAM-SHA-256<SCRAM-SHA-256_TLS-EXPORTER',
  the corresponding base64 encoded SHA-1 hash (SHA-1 will be used because negotiated below) is: 'U3vZANxXbl1pMOMBAFPnTb5YXWk='
-->
<mechanisms xmlns='urn:xmpp:sasl:1'>
  <mechanism>SCRAM-SHA-1</mechanism>
  <mechanism>SCRAM-SHA-1_TLS-EXPORTER</mechanism>
  <mechanism>SCRAM-SHA-256</mechanism>
  <mechanism>SCRAM-SHA-256_TLS-EXPORTER</mechanism>
</mechanisms>

<!--
  Client sends the selected mechanism alogside the initial-response data.
-->
<authenticate xmlns='urn:xmpp:sasl:1' mechanism='SCRAM-SHA-1_TLS-EXPORTER'>
  <!-- Base64 of: 'p=tls-exporter,,n=user,r=12C4CD5C-E38E-4A98-8F6D-15C38F51CCC6,d=U3vZANxXbl1pMOMBAFPnTb5YXWk=' -->
  <initial-response>cD10bHMtZXhwb3J0ZXIsLG49dXNlcixyPTEyQzRDRDVDLUUzOEUtNEE5OC04RjZELTE1QzM4RjUxQ0NDNixkPVUzdlpBTnhYYmwxcE1PTUJBRlBuVGI1WVhXaz0=</initial-response>
</authenticate>

<!--
  SCRAM-SHA-1 challenge issued by the server.
  Base64 of: 'r=12C4CD5C-E38E-4A98-8F6D-15C38F51CCC6a09117a6-ac50-4f2f-93f1-93799c2bddf6,s=QSXCR+Q6sek8bf92,i=4096'
-->
<challenge xmlns='urn:xmpp:sasl:1'>
  cj0xMkM0Q0Q1Qy1FMzhFLTRBOTgtOEY2RC0xNUMzOEY1MUNDQzZhMDkxMTdhNi1hYzUwLTRmMmYtOTNmMS05Mzc5OWMyYmRkZjYscz1RU1hDUitRNnNlazhiZjkyLGk9NDA5Ng==
</challenge>

<!--
  The client responds with the base64 encoded client-final-message (password: 'pencil').
  Base64 of: 'c=cD10bHMtZXhwb3J0ZXIsLMcoQvOdBDePd4OswlmAWV3dg1a1Wh1tYPTBwVid10VU,r=12C4CD5C-E38E-4A98-8F6D-15C38F51CCC6a09117a6-ac50-4f2f-93f1-93799c2bddf6,p=icrRuoQBB0htw5+K/6RNEDJ0Q4Y='
  The c-attribute contains the GS2-header and channel-binding data blob (32 bytes).
-->
<response xmlns='urn:xmpp:sasl:1'>
  Yz1jRDEwYkhNdFpYaHdiM0owWlhJc0xNY29Rdk9kQkRlUGQ0T3N3bG1BV1YzZGcxYTFXaDF0WVBUQndWaWQxMFZVLHI9MTJDNENENUMtRTM4RS00QTk4LThGNkQtMTVDMzhGNTFDQ0M2YTA5MTE3YTYtYWM1MC00ZjJmLTkzZjEtOTM3OTljMmJkZGY2LHA9aWNyUnVvUUJCMGh0dzUrSy82Uk5FREowUTRZPQ==
</response>

<!--
  This completes, so the Server sends a success containing the base64 encoded server signature.
  A SASL2 success always includes the authorization identifier.
-->
<success xmlns='urn:xmpp:sasl:1'>
  <authorization-identifier>user@example.org</authorization-identifier>
  <!-- Base64 of: 'v=Ax+ZEP5hf90z8+KnakwspK9mEhk=' -->
  <additional-data>
    dj1BeCtaRVA1aGY5MHo4K0tuYWt3c3BLOW1FaGs9
  </additional-data>
</success>

Restoring protocol agility for SASL methods part #2

Activating a new SASL (SCRAM) method and saving new hashes for these methods for already known users on the server is a bit trickier. To solve this, the server offers new SASL mechanisms indicating that he allows for hash upgrades, if he doesn’t have all required hashes in its database yet. The ordering of these new upgrade mechanisms should use a stable sorting algorithm. First sorting by perceived strength of the algorithm updated to, then by the percieved strength of the algorithm used for authentication (e.g. use the highest strength for authentication and upgrade to the highest strength possible with this authentication).

For reference, the SCRAM flow as stated in RFC 5802 section 3 is as follows, with HMAC() and HASH() corresponding to the hash method used to authenticate (e.g. SHA-256 for SCRAM-SHA-256):

SaltedPassword  := Hi(Normalize(password), salt, i)
ClientKey       := HMAC(SaltedPassword, "Client Key")
StoredKey       := H(ClientKey)
AuthMessage     := client-first-message-bare + "," +
                   server-first-message + "," +
                   client-final-message-without-proof
ClientSignature := HMAC(StoredKey, AuthMessage)
ClientProof     := ClientKey XOR ClientSignature
ServerKey       := HMAC(SaltedPassword, "Server Key")
ServerSignature := HMAC(ServerKey, AuthMessage)

The SaltedPassword is the hash saved in the database on the server alongside the salt. Using additional SASL2 tasks we can now require the client to perform an additional task which consists of sending the SaltedPassword for the hash algorithm to upgrade to. The server just provides the required salt (that must be a new random value not equal to the one used for the old hash) and iteration count. By providing the salted hash after the successful completion of our SCRAM authentication, server and client can be sure to talk to the right one and when channel-binding is used, both can be sure no MITM is involved that could intercept the new SaltedPassword. I strongly suggest to only support password upgrades if channel-binding is used.

An example protocol flow using a modified SASL2 might look as follows (resembling more or less the SCRAM flow used in part #1 above and adding a second task for the desired hash upgrade):

<!--
  Server sends stream features indicating support for SASL2.
-->
<stream:features>
  <authentication xmlns='urn:xmpp:sasl:1'/>
</stream:features>

<!--
  Client intiates authentication by sending the base64 encoded username it wishes to authenticate for.
-->
<request xmlns='urn:xmpp:sasl:1'>
  <user>dXNlcg==</user>
</request>

<!--
  Server sends the list of supported mechanisms for this user.
  The sorted list will be 'SCRAM-SHA-1_TLS-EXPORTER<SCRAM-SHA-256_TLS-EXPORTER<UPGRADE-SCRAM-SHA-256_SCRAM-SHA-1<UPGRADE-SCRAM-SHA-256_SCRAM-SHA-1_TLS-EXPORTER',
  the corresponding base64 encoded SHA-1 hash (SHA-1 will be used because negotiated below) is: 'nKFUXQ7h9IL3eo17pKygmacnEsk='
-->
<mechanisms xmlns='urn:xmpp:sasl:1'>
  <mechanism>SCRAM-SHA-1</mechanism>
  <mechanism>SCRAM-SHA-1_TLS-EXPORTER</mechanism>
  <mechanism>UPGRADE-SCRAM-SHA-256_SCRAM-SHA-1</mechanism>
  <mechanism>UPGRADE-SCRAM-SHA-256_SCRAM-SHA-1_TLS-EXPORTER</mechanism>
</mechanisms>

<!--
  Client sends the selected mechanism alogside the initial-response data.
-->
<authenticate xmlns='urn:xmpp:sasl:1' mechanism='SCRAM-SHA-1_TLS-EXPORTER'>
  <!-- Base64 of: 'p=tls-exporter,,n=user,r=12C4CD5C-E38E-4A98-8F6D-15C38F51CCC6,d=nKFUXQ7h9IL3eo17pKygmacnEsk=' -->
  <initial-response>cD10bHMtZXhwb3J0ZXIsLG49dXNlcixyPTEyQzRDRDVDLUUzOEUtNEE5OC04RjZELTE1QzM4RjUxQ0NDNixkPVUzdlpBTnhYYmwxcE1PTUJBRlBuVGI1WVhXaz0=</initial-response>
</authenticate>

<!--
  SCRAM-SHA-1 challenge issued by the server.
  Base64 of: 'r=12C4CD5C-E38E-4A98-8F6D-15C38F51CCC6a09117a6-ac50-4f2f-93f1-93799c2bddf6,s=QSXCR+Q6sek8bf92,i=4096'
-->
<challenge xmlns='urn:xmpp:sasl:1'>
  cj0xMkM0Q0Q1Qy1FMzhFLTRBOTgtOEY2RC0xNUMzOEY1MUNDQzZhMDkxMTdhNi1hYzUwLTRmMmYtOTNmMS05Mzc5OWMyYmRkZjYscz1RU1hDUitRNnNlazhiZjkyLGk9NDA5Ng==
</challenge>

<!--
  The client responds with the base64 encoded client-final-message (password: 'pencil').
  Base64 of: 'c=cD10bHMtZXhwb3J0ZXIsLMcoQvOdBDePd4OswlmAWV3dg1a1Wh1tYPTBwVid10VU,r=12C4CD5C-E38E-4A98-8F6D-15C38F51CCC6a09117a6-ac50-4f2f-93f1-93799c2bddf6,p=UApo7xo6Pa9J+Vaejfz/dG7BomU='
  The c-attribute contains the GS2-header and channel-binding data blob (32 bytes).
-->
<response xmlns='urn:xmpp:sasl:1'>
  Yz1jRDEwYkhNdFpYaHdiM0owWlhJc0xNY29Rdk9kQkRlUGQ0T3N3bG1BV1YzZGcxYTFXaDF0WVBUQndWaWQxMFZVLHI9MTJDNENENUMtRTM4RS00QTk4LThGNkQtMTVDMzhGNTFDQ0M2YTA5MTE3YTYtYWM1MC00ZjJmLTkzZjEtOTM3OTljMmJkZGY2LHA9VUFwbzd4bzZQYTlKK1ZhZWpmei9kRzdCb21VPQ==
</response>

<!--
  This completes, so the Server sends a continue containing the base64 encoded server signature and the upgrade task to perform.
-->
<continue xmlns='urn:xmpp:sasl:1'>
  <authorization-identifier>user@example.org</authorization-identifier>
  <!-- Base64 of: 'msVHs/BzIOHDqXeVH7EmmDu9id8=' -->
  <additional-data>
    dj1tc1ZIcy9CeklPSERxWGVWSDdFbW1EdTlpZDg9
  </additional-data>
  <tasks>
    <task>UPGRADE-SCRAM-SHA-256</task>
  </tasks>
</continue>

<!--
  The client provides the SaltedPassword hash for SCRAM-SHA-256
-->
<next xmlns='urn:xmpp:sasl:1' task='UPGRADE-SCRAM-SHA-256'/>

<!--
  The server sends the required salt and iteration count encoded as base64 encoded SASL attributes.
  Base64 of: 's=A_SXCRXQ6sek8bf_Z,i=4096'
-->
<challenge xmlns='urn:xmpp:sasl:1'>
  cz1BX1NYQ1JYUTZzZWs4YmZfWixpPTQwOTY=
</challenge>

<!--
  The client responds with the base64 encoded SaltedPassword.
-->
<response xmlns='urn:xmpp:sasl:1'>
  BzOnw3Pc5H4bOLlPZ/8JAy6wnTpH05aH21KW2+Xfpaw=
</response>

<!--
  Finally, the server sends a success after adding the salted SHA-256 hash to it's database.
  A SASL2 success always includes the authorization identifier.
-->
<success xmlns='urn:xmpp:sasl:1'>
  <authorization-identifier>user@example.org</authorization-identifier>
</success>

If the server needs to upgrade to multiple new SCRAM algorithms, he can do so one at a time on every new authentication. This is no “do everything once” anyways, because not every client might support every upgrade possible.

Conclusion

To conclude this, we now identified several improvements to regain the properties listed in the beginning. These improvements can mainly be achieved by updating the SASL2 XEP (XEP-0388: Extensible SASL Profile). The downgrade prevention (the additional SCRAM attribute d) should possibly be published as RFC, but a XEP could suffice, too.

Feel free to comment on anything in here. I’m always open to feedback and improvements. Just contact me at thilo@monal-im.org.

In short this consists of the following tasks (in no special order).

1. Implement more privacy-friendly push server

The current push appserver (https://github.com/tmolitor-stud-tu/mod_push_appserver) saves more data than strictly needed to perform its task, let’s change that. On top of that, a possible HA-setup and load balancing should be strived for.

2. Implement basic Audio calls using WebRTC

Include WebRTC library into our codebase and wire it up to allow for simple audio calls not involving XMPP (maybe send SDP data through a Monal specific non-standardized XMPP stanza to make this work without hardcoded ip and port).

This is to test Audio Calls in the field first before wiring them up using correct XMPP XEPs.

3. Implement all XEPs needed for Audio Calls

To make Audio Calls for XMPP work, we need proper XMPP integration. This wires up the stuff implemented in (2) into the XMPP layer by implementing the proper XEPs listed below. This will add support for 1:1 audio calls to Conversations, Dino and many more XMPP clients.

4. Implement Video calls using WebRTC

On top of the work done in (2) and (3) we want to implement Video Call support as well. This will add support for 1:1 video calls to Conversations, Dino, and more.

5. Security quickscan

We want a security quickscan performed by Radically Open Security and implement mitigations for problems found by them.

6. Implement MUC UI management (add/remove/promote users etc.)

While Monal supports MUCs (multi user chats) in its flavours “channel” and “group”, the app is still lacking proper support for creating and managing group-type MUCs. We will implement that missing piece in this task.

7. Port addContact UI to SwiftUI

The existing UI to add a contact or show pending contact requests is not user-friendly. We will therefore port it to SwiftUI. Once contacts can be added using the new UI, a UI for creating MUCs will be added. Monal will then support the creation of new private MUCs. Additionally, the existing functionality to scan contact QR-Codes and open contact links from other apps / iOS camera will be refactored. After the refactoring the user can preview the action and select an appropriate account before adding the contact.

8. Add SASL mechanism upgrade capability to XEP-0388

XEP-0388 (“Extensible SASL Profile”) modernizes the old XMPP SASL profile defined in RFC 6120 and reduces round-trips. But neither the old SASL profile nor the new one (dubbed SASL2) solve the problem of SASL mechanism upgrades. A server that saves the hashed password for SCRAM-SHA-1 authentication has no way of upgrading to SCRAM-SHA-256. It does not know the cleartext password to calculate the hash to store itself and no way to ask the client for it. We want to take the opportunity and update the SASL2 XEP before it gets widely deployed to provide a way for a smooth upgrade path that allows the servers to store multiple hashes and get new hashes provided by clients that support the new algorithm without the need of ever knowing the cleartext password.

9. Implement SASL2 in Monal (including SCRAM SHA1, SHA256 and SHA512)

Implement the updated SASL2 XEP and the accompanying XEP-0440 (“SASL Channel-Binding Type Capability”) in Monal which serves as foundation for the implementation of XEP-0397 (“Instant Stream Resumption”).

10. Write new XEP for downgrade protection of channel-binding types and SASL mechanisms and implement it

A Man-In-The-Middle capable of impersonating the XMPP server on the TLS level can use SASL Mechanism Stripping and/or strip channel-binding types to downgrade the security. Adding an optional SCRAM attribute containing the hash of all server-advertised SASL mechanisms and channel-binding types as seen by the client will enable the server to check if any downgrade took place and to abort the authentication, if so. Because the SCRAM attribute will be part of the mutual authentication between client and server, a MITM will not be able to forge it.

As part of this transition we also deployed some new push servers to not let an old retired developer pay for the infrastructure needed for Monal.

Coming along with this transition from the old developer team to the new one is our new clean website at https://monal-im.org/. From now on, the old blog at https://monal.im/ will not be used for Monal anymore.

Many thanks to all users, contributors and followers so far.

Special thanks goes to Anu. Without him and his passion, Monal would not have been possible. He developed and maintained Monal for more than 10 years, always ensuring compatibility with the latest iOS releases.

When I (Thilo) gradually took over development, I was able to build upon an app with a decent codebase rather than writing my own app from scratch. That made it possible to improve Monal further while already being used by thousands of people. I can not stress enough how thankful I was and still am for all the work Anu put into the development of Monal. Thank you, Anu, for your wonderful work towards a modern XMPP client for iOS and macOS!

Thilo, Friedrich, Anu

Planned XEPs

Donate

Monal is developed by volunteers and community collaboration. The work which has been done is usually not paid, and the developers ask for donations to keep up service costs and development in the future! Please consider giving a little back for the hard work which has been conducted. Currently, there are three ways for financial support of the Monal development:

• Donate via GitHub Sponsors
• Donate via Libera Pay
• EU citizens can donate via SEPA, too. IBAN: DE66 5007 0371 0856 0419 01

Here you can read about further support of the development!

Find general information in the Monal Wiki.

Translations

Translations of the app itself as well as the appstore description and even the changelog displayed inside the appstore are hosted and managed via Weblate.
Feel free to translate Monal to your language!

Reporting a Vulnerability

It is highly appreciated reporting a vulnerability to the Monal developers. We ask you to not disclose it to the public until the vulnerability has been fixed. This prevents abuse and exploitation in the current published releases. Please report issues directly to info[at]monal[minus]im[dot]org.

Please try to report

• in detail what you are concerned about
• if applicable, how to reproduce
• your contact details, if the sending email is not enough. That way we can ask questions back to you.

You are also invited to make a recommendation on how to fix a potential security vulnerability.

Once a vulnerability has been announced and indicated we try our very best to provide a fix as soon as possible, at its best within days. However, dependent on the potential issue it can take longer if many code sections need to be changed.

Please be reminded that this is a non-commercial software project.

Thank you for considering reporting a security vulnerability. This improves the quality of the app significantly.

Features

Monal currently covers the following chat features:

• Decentralised and federated chat standard XMPP
• Private and group messaging
• Privacy-respecting push notifications
• Encrypted private and group chats (state-of-the-art encryption (OMEMO)
• Message history
• Free selection of your XMPP account provider
• Voice messaging
• Message archiving
• Upload of files, videos and images (HTTP Upload)
• Audio and Video calls
• Many settings and a design to offer privacy settings in the app to the need of the user
• A detailed and technical listing of your supported features (so called XMPP Extensions) can be found in our DOAP file.

Planned features

• User-interface overhaul

Implemented XEPs

You can check the complete list of supported XEPs at https://xmpp.org/software/monal-im/

Planned XEPs

Monal App
Our privacy policy may differ between app versions. Before reading our privacy policy for our App you first need to find out the Monal version that you are using.

How to find out your Monal version

1. Open Monal
2. Open up the settings menu in the upper left corner (gearwheel)
3. Scroll down to the last entry “version”

Monal App Privacy Policies

Monal Website
You can find our latest privacy policy for our website here: Website Privacy Policy

Monal App
Our privacy policy may differ between app versions. Before reading our privacy policy for our App you first need to find out the Monal version that you are using.

How to find out your Monal version

1. Open Monal
2. Open up the settings menu in the upper left corner (gearwheel)
3. Scroll down to the last entry “version”

Monal App Privacy Policies

Privacy Monal App < 5.2.0

Monal for iOS and macOS will register for APNS push notifications via a server to server (s2s) connection from your XMPP server to our push server. Your XMPP JID alongside with a push identifier and secret token from Apple, that is only valid for this app, will be saved and logged in the push-server logs. We do not intend to track you. All server logs are purged every two weeks. Our logs allow us to see the following details:

• Your JID (including your server’s hostname)
• Time when you register for push notifications
• Your apple push node and push token that was generated for Monal by Apple
• Time when your XMPP server triggered a push notification to your Monal device

To fulfill its duty, our push server has to hold some information associated with an Apple push token, until Apple marks the token a deleted, which usually means you have uninstalled the app (Info: Apple confirms if a token is still valid on every push). In detail these information consists of:

• The Apple push token
• The timestamp of the last push error
• The timestamp of the last successful push
• The timestamp of the registration of your device with Monal’s push-server
• The timestamp when the registration was renewed
• A random UUID identifying your device
• A random secret used by your XMPP server to authenticate a push

Push server locations

Crash reports and app usage

Monal does track crashes and usage data anonymously using the tools provided by Apple. This is opt-in only and controlled by iOS and macOS global settings. If a user decides not to send any data to developers, no crash logs are sent to Monal developers.

Logs

Your local device will contain a log file with all sent and received raw XMPP messages as well as debug logs. It does contain sensitive personal data! This file will never be transferred to us, except if you explicitly (manually) send it to us (e.g. via mail).

GDPR Subject Access Requests (SAR)

European GDPR allows users to request a copy of all data retained about them. Please send GDPR requests to info@monal-im.org. As by GDPR we need to validate your JID before answering to your inquiry. Therefore, we will provide you a JID you must send a confirmation to, before we can answer your request and send you all retained data related to your JID.

Monal Website
You can find our latest privacy policy for our website here: Website Privacy Policy

Monal App
Our privacy policy may differ between app versions. Before reading our privacy policy for our App you first need to find out the Monal version that you are using.

How to find out your Monal version

1. Open Monal
2. Open up the settings menu in the upper left corner (gearwheel)
3. Scroll down to the last entry “version”

Monal App Privacy Policies

Privacy Monal App 5.2.0 - 5.4.x

App Resources are very limited on iOS and macOS. Monal for example can only run a limited time in the background after a user either locked the screen or switched the app. Hence, apps can not simply keep a connection to your xmpp server open 24/7 to inform you about new messages. To overcome these limitations your XMPP-Server can can request our push server to send push messages to Apple. With these push messages we can request Apple to wake up the app on your phone. Once it is woken up it has about 30 seconds to connect to your XMPP server, fetch all new messages and show a push notification for these new messages.

How push works

Every time that Monal logins at your XMPP servers, it requests your server to inform us once your received an XMPP message while Monal was closed. We therefore requests a Monal specific push token from Apple. Using this Monal specific push token our push server can send push messages via Apples push system to wake up the app on your device.

Once push messages are enabled for your Monal instance on your XMPP servers, your XMPP servers will open a encrypted server to server (s2s) connection to one of our push servers. Using this s2s connection your XMPP servers will then request our push servers to wake up Monal every time that new messages should be processed. To wake up your instance your XMPP servers send us:

• your unique Monal specific push token that was generated by Apple
• the domain of the XMPP server that you are using.

Push

• We never see your messages.
• We do not know who you are chatting with.
• We could only ever track what XMPP domains a push token is/was using.
• We can not identify a user.

Push-Servers

We provide two independent push server regions at the moment: Europe and US. By default, each device will choose our Europe based push region unless the device local is set to the US.

How to change the push region

1. Open Monal
2. Open up the settings menu in the upper left corner (gearwheel)
3. Open the Notifications menu
4. Scroll down
5. Select a region

Push server regions

If you are an XMPP server administrator, and you restricted s2s connections, please allow s2s to all our regions.

Push server locations

Crash reports and app usage

Monal does track crashes and usage data anonymously using the tools provided by Apple. This is opt-in only and controlled by iOS and macOS global settings. If a user decides not to send any data to developers, no crash logs are sent to Monal developers.

Logs

Your local device will contain a log file with all sent and received raw XMPP messages as well as debug logs. It does contain sensitive personal data! This file will never be transferred to us, except if you explicitly (manually) send it to us (e.g. via mail).

GDPR Subject Access Requests (SAR)

European GDPR allows users to request a copy of all data retained about them. Starting with Monal 5.2.0 we no longer see your JIDs (username@domain.tld) in our push servers. We therefore are not able to send you retained data related to your JID. We furthermore are unable to provide your retained data related to your unique push token because we have no way to verify that Apple issued you a provided token. If you have questions regarding GDPR, please send us a mail to info@monal-im.org.

Monal Website
You can find our latest privacy policy for our website here: Website Privacy Policy

Monal App
Our privacy policy may differ between app versions. Before reading our privacy policy for our App you first need to find out the Monal version that you are using.

How to find out your Monal version

1. Open Monal
2. Open up the settings menu in the upper left corner (gearwheel)
3. Scroll down to the last entry “version”

Monal App Privacy Policies

Website Privacy

A user’s IP address will be logged in the HTTP server logs. All server logs are purged every two weeks and there isn’t any way for us to associate this information with any particular individual. Our websites do not use or need any cookies. And of course it doesn’t use any third party/external (JS-CSS-Font) dependencies.

TLDR:

Info: Monal will stop support for iOS 12, iOS 13 and macOS Catalina!
We are searching for a SwiftUI developer.
We need a new simplified website.
With better continuous funding, our push servers will move from the US to Europe.
We have a new support mail: info@monal-im.org

Two years ago we decided to rewrite the Monal app almost entirely and improve it gradually in the process, instead of creating another XMPP Client for iOS and macOS. We successfully managed to transform Monal from an app that had flaws and issues with many functions to a level that promotes a user-friendly experience with working features such as push notification, group chat, and partially end-to-end encryption support (OMEMO). If you are selecting an XMPP client for Apple systems we think that Monal is a great choice nowadays. We have been investing more than a thousand hours and worked hard to overcome all the flaws, the legacy app had. We invite you to give the recent beta a try!

The development of Monal has not yet finished though, and many more features are hopefully to come. But due to our time constraints, it sometimes takes a bit longer than we and the community would like. We are at most two developers at the moment using our spare time to maintain Monal and develop new features. As we are developing Monal in our spare time without decent funding, it is sometimes hard to prioritize specific features. Please give this circumstance some credits.

What should Monal look like in the future?

To give you a bit of insight knowledge of our plans we tried to summarize the most important tasks.

Interface (SwiftUI)

In the future Monal should be easy to use. Therefore, the interface requires a proper rework and while we are at it, it should be ported to SwiftUI. While we are still improving in designing with SwiftUI, we would be glad if there is a SwiftUI & Open Source enthusiast who would like to help us with this.
With this transition we would like to improve the accessibility of the app as well. If you like to support an open source project, and you would like to be part of our SwiftUI journey please contact us.

Task:

• Add new MUC creation and management UI
• Port the chat view
• Finish contact Details
  • List group chat (MUC) participants
  • Display OMEMO encryption fingerprints for verification
• Port our settings
• Port all other remaining views

Qualifications:

• General knowledge around SwiftUI (iOS and Catalyst)
• Interest in improving a (XMPP) chat client
• Optional, but preferred: Some experience with XMPP (e.g. some weeks, or maybe months of usage of Monal or any other “modern” XMPP client)
• Optional: Experience in designing inclusive / accessible UI

Website

We need a modern (simplified) Hugo based website that is easier to understand, similar to Conversations, Dino or Gajim.

If you have some spare time, and you are a skilled in creating websites with Hugo please contact us.

Requirements:

• Simple design nothing too fancy
• Privacy by design → No analytics, no external CSS, jss, … usage

OMEMO Encryption in Group Chat (MUC)

We started to integrate OMEMO for group chats (MUC) into our alpha build. The receiving and sending sides are already implemented, but there are a few more steps until we can release it into the beta.

Switching to our new Domain monal-im.org

In late 2021 we decided that we would like to have a domain with DNSSEC as our current top level domain does not support it. This domain will mainly be used for our push servers and mail servers in the future. From now on you will be able to reach us via info@monal-im.org.

Build server

Thanks to ~20 generous donors, we were able to buy a new build server that will be used to build our alpha, beta and stable releases. Furthermore, Thilo is now finally able to debug code with a proper debugger connected to his phone.

Redundant Push Servers

We are currently using an AWS US instance for our push server that is not redundant and failed in 2021 more frequently than we liked it to. For that reason we started an internal project to auto-deploy our push server with Ansible and looked into ways for running a redundant push server setup. The first tests look promising so far, but a few more things need to be sorted out before we can switch over to our new setup.

Before we can switch to the new push server setup, we need a stable funding each month. We estimate that renting a VM in Germany and one at a different hoster in Finland would cost us between 16€ to 32€ each month. Without a stable funding we might not be able to afford this new setup and our push servers would stay in the US.

Thanks to our generous build server donors, we have a few bucks left that will be used as a ~5 month buffer in case of fluctuant push server funding

Privacy improved push servers

With our current push implementation our so-called “app servers” see your JID (username + server), a unique but otherwise opaque device id and an opaque token generated by apple, as well as your interaction times (when you register for push notifications, timestamps when your XMPP server triggers a push notification device).
If you use multiple accounts on one device, the unique device id is shared across all accounts. We don’t think that this is ideal, as we know all jid’s a user is using.
In the future we want to try to reduce our knowledge by hiding your username from our app servers. If our idea works, we would only see that a device is registered on one or more domains and the timestamps that a push message was triggered from each domain that is used.

Audio and Video Calls

Many clients such as Conversations and Dino support audio and video calls, so Monal should be next 🙂

End-of-life: iOS 12, iOS 13 and macOS Catalina will not be supported anymore

Our user group on iOS 12, iOS 13 as well as macOS Catalina has decreased in last years while the resources needed to maintain these old platforms increased. We therefore decided to focus on newer iOS versions and drop the old ones. The next stable release will only be supported on iOS 14 and higher and macOS Big Sur and higher. We are still unsure how long we will support iOS 14, as most of the devices also support iOS15.

Donations and Support

Monal is developed by volunteers and community collaboration. The work which has been done is usually not paid, and the developers need to keep up service costs and development in the future! Please consider giving a bit back for the hard work which has been conducted. Currently, there are three ways to financially support the Monal development:

• Donate via GitHub Sponsors
• Donate via Libera Pay
• EU citizens can donate via SEPA, too. Just contact Thilo Molitor via mail to info@monal-im.org to get his IBAN.

Here you can read about further support of the development! Find general information in the Monal Wiki.

Translations

We host and manage translations via Weblate.

Many thanks!

Of course, thank you very much to everyone who supported us in the past two years! 🙂

You can follow us via Mastodon.

Designed for iOS and Mac

Things look and work the way you expect. iOS, iPadOS or macOS, there is a version of Monal for you.

Monal is developed under an open-source BSD license that serves the user, while not selling or tracking information for external parties (nor for anyone else). This app exists because it is key to ensure usability on all platforms and within the XMPP network with all its positives aspects when it comes to decentral communication and infrastructure.

Find the development on GitHub!

Thilo Molitor
Vogelsbergstr. 18
68642 Bürstadt
Germany

eMail: info[at]monal[minus]im[dot]org

About

Monal originates in 2002 as fork of the SworIM app. Until 2019 it has been developed by Anu Pokharel. Since then Thilo Molitor took over the development and joined in 2020 with Friedrich Altheide (until 2024). From initial rewrite of code in the backend the entire app has been upgraded with a modern XMPP engine, new features and future-proof setup. Monal challenges to be the go-to XMPP chat-app for the iOS and macOS platform.

Monal Team

Thilo Molitor - About me on wiki.xmpp.org

• Molitor, Thilo; Altheide, Friedrich. Modern XMPP - A story based on Monal Berlin XMPP Meetup, 13th April 2022, Recording, Slides
• Molitor, Thilo; Altheide, Friedrich. Privacy friendly push Berlin XMPP Meetup, 13th April 2022, Recording, Slides (starting after page 29)
• Molitor, Thilo. XMPP Push notifications on iOS
• Molitor, Thilo. Monal development recap 2019 - 2021 and open discussion Berlin XMPP Meetup, 13th October 2021, Recording

XEP publications

• XEP-0198: Stream Management
  Authors: Justin Karneges, Peter Saint-Andre, Joe Hildebrand, Fabio Forno, Dave Cridland, Matthew Wild, Thilo Molitor

  This specification defines an XMPP protocol extension for active management of an XML stream between two XMPP entities, including features for stanza acknowledgements and stream resumption.

• XEP-0353: Jingle Message Initiation
  Authors: Philipp Hancke, Peter Saint-Andre, Thilo Molitor

  This specification provides a way for the initiator of a Jingle session to propose sending an invitation in an XMPP message stanza, thus taking advantage of message delivery semantics instead of sending IQ stanzas to all of the responder’s online resources or choosing a particular online resource.

• XEP-0474: SASL SCRAM Downgrade Protection
  Authors: Thilo Molitor

  This specification provides a way to secure the SASL and SASL2 handshakes against method and channel-binding downgrades.

• XEP-0388: Extensible SASL Profile
  Authors: Dave Cridland, Thilo Molitor, Matthew Wild

  This document describes a replacement for the SASL profile documented in RFC 6120 which allows for greater extensibility.

• XEP-0480: SASL Upgrade Tasks
  Authors: Thilo Molitor

  This specification provides a way to upgrade to newer SASL mechanisms using SASL2 tasks.

• XEP-0440: SASL Channel-Binding Type Capability
  Authors: Florian Schmaus, Thilo Molitor

  This specification allows servers to annouce their supported SASL channel-binding types to clients.

• XEP-0492: Chat notification settings
  Authors: Nicolas Cedilnik, Thilo Molitor

  This document defines an XMPP protocol extension to synchronise per-chat notification settings across different clients.