Join Spam Attack Update 1

What happened? #

A “join spam” attack was started by an unknown person. This means hundreds of zombie users were created by a bot and joined to our room.

What are the implications on this room? #

Since the attack started, there is a discrepancy between the participating homeservers timeline and users for/in this room.

With your help, we assessed, the status of the timeline. So far, it looks as if it is divided into two camps. matrix.org (plus at least one more) and the rest of the homeservers. (There are currently users from 396 different servers participating in this room)

What is currently being done to restore the room to its former glory? #

We’re currently hoping, the situation resolves itself. Matrix federation should sync up the room between the participating homeservers. In addition, we contacted the matrix.org staff a few days ago and hope to get some more information, how we can speed up the process and remove the remaining “spambot” users without creating too many new state events in the room. Our previous attempt to clean up the room, before we were fully aware of the extent of the situation, did probably more harm than good. This is why we stopped removing those users, the methods used were not feasible. At least not, when the rooms are not synced up.

We now hope to find a better solution, together with the matrix.org staff, than tombstoneing the room and switching to a new one.

To prevent more “join spam” we decided to make the room temporarily “invite only” . Everyone in a room can still invite other users. Messages may appear in a wrong order, vanish, or are not federated at all. We thank you for your understanding.

Did this only happen to us? #

Probably not. We heard rumors about other rooms with similar problems, but we are not sure about that. If you know more about this, please let us know.

Why did we react so slowly/poorly? #

A series of unfortunately events lead to this situation.

TLDR; #

We didn’t know, this happened.

Long version #

When the attack started, a discrepancy between the timeline and user on many homeserver’s emerged. matrix.org didn’t sync up with the rest. The mods on matrix.org had no clue about the situation.

When I noticed something fishy on my homeserver, I contacted them. ASAP, we started digging and found the cause.

Another difficulty was, that one of the latest element updates disabled the functionality, even when enabled before, to show hidden events in the timeline. The update moved the switch to the /devtools.
I assumed, the feature was still active, which would have made it easy to spot the issue early on.

At this point, we would also like to thank the user(s), who tried to make us aware of the situation with pings. Unfortunately, we never got a notification or anything. We suspect due to the attack, the notification feature of synapse acted up (together with other parts of my homeserver, which started working again after restarting the whole thing). On matrix.org, the entire message
is still not synchronized.

How can we prevent something like this from happening again? #

We haven’t found a definitive solution for this problem yet. There are ways to make something like this a bit harder in the future, at the cost of being incompatible with some older or not as up-to-date with the spec’s matrix clients. But we try to find better solution. We keep you posted.