Implement Push Notifications (#1842)

* Add Pushserver component with Pushers API

Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Dan Peleg <dan@globekeeper.com>

* Wire Pushserver component

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Add PushGatewayClient.

The full event format is required for Sytest.

* Add a pushrules module.

* Change user API account creation to use the new pushrules module's defaults.

Introduces "scope" as required by client API, and some small field
tweaks to make some 61push Sytests pass.

* Add push rules query/put API in Pushserver.

This manipulates account data over User API, and fires sync messages
for changes. Those sync messages should, according to an existing TODO
in clientapi, be moved to userapi.

Forks clientapi/producers/syncapi.go to pushserver/ for later extension.

* Add clientapi routes for push rules to Pushserver.

A cleanup would be to move more of the name-splitting logic into
pushrules.go, to depollute routing.go.

* Output rooms.join.unread_notifications in /sync.

This is the read-side. Pushserver will be the write-side.

* Implement pushserver/storage for notifications.

* Use PushGatewayClient and the pushrules module in Pushserver's room consumer.

* Use one goroutine per user to avoid locking up the entire server for
  one bad push gateway.
* Split pushing by format.
* Send one device per push. Sytest does not support coalescing
  multiple devices into one push. Matches Synapse. Either we change
  Sytest, or remove the group-by-url-and-format logic.
* Write OutputNotificationData from push server. Sync API is already
  the consumer.

* Implement read receipt consumers in Pushserver.

Supports m.read and m.fully_read receipts.

* Add clientapi route for /unstable/notifications.

* Rename to UpsertPusher for clarity and handle pusher update

* Fix linter errors

* Ignore body.Close() error check

* Fix push server internal http wiring

* Add 40 newly passing 61push tests to whitelist

* Add next 12 newly passing 61push tests to whitelist

* Send notification data before notifying users in EDU server consumer

* NATS JetStream

* Goodbye sarama

* Fix `NewStreamTokenFromString`

* Consume on the correct topic for the roomserver

* Don't panic, NAK instead

* Move push notifications into the User API

* Don't set null values since that apparently causes Element upsetti

* Also set omitempty on conditions

* Fix bug so that we don't override the push rules unnecessarily

* Tweak defaults

* Update defaults

* More tweaks

* Move `/notifications` onto `r0`/`v3` mux

* User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly

Co-authored-by: Piotr Kozimor <p1996k@gmail.com>
Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

1 files changed, 66 insertions, 13 deletions

diff --git a/sytest-whitelist b/sytest-whitelist
index 3e38176f..602f8646 100644
--- a/sytest-whitelist
+++ b/sytest-whitelist
@@ -339,17 +339,17 @@ Existing members see new members' join events
 Inbound federation can receive events
 Inbound federation can receive redacted events
 Can logout current device
-Can send a message directly to a device using PUT /sendToDevice 
-Can recv a device message using /sync 
-Can recv device messages until they are acknowledged 
-Device messages with the same txn_id are deduplicated 
-Device messages wake up /sync 
-Can recv device messages over federation 
-Device messages over federation wake up /sync 
-Can send messages with a wildcard device id 
-Can send messages with a wildcard device id to two devices 
-Wildcard device messages wake up /sync 
-Wildcard device messages over federation wake up /sync 
+Can send a message directly to a device using PUT /sendToDevice
+Can recv a device message using /sync
+Can recv device messages until they are acknowledged
+Device messages with the same txn_id are deduplicated
+Device messages wake up /sync
+Can recv device messages over federation
+Device messages over federation wake up /sync
+Can send messages with a wildcard device id
+Can send messages with a wildcard device id to two devices
+Wildcard device messages wake up /sync
+Wildcard device messages over federation wake up /sync
 Can send a to-device message to two users which both receive it using /sync
 User can create and send/receive messages in a room with version 6
 local user can join room with version 6
@@ -477,7 +477,7 @@ Federation key API can act as a notary server via a GET request
 Inbound /make_join rejects attempts to join rooms where all users have left
 Inbound federation rejects invites which include invalid JSON for room version 6
 Inbound federation rejects invite rejections which include invalid JSON for room version 6
-GET /capabilities is present and well formed for registered user 
+GET /capabilities is present and well formed for registered user
 m.room.history_visibility == "joined" allows/forbids appropriately for Guest users
 m.room.history_visibility == "joined" allows/forbids appropriately for Real users
 POST rejects invalid utf-8 in JSON
@@ -588,6 +588,59 @@ User can invite remote user to room with version 9
 Remote user can backfill in a room with version 9
 Can reject invites over federation for rooms with version 9
 Can receive redactions from regular users over federation in room version 9
+Pushers created with a different access token are deleted on password change
+Pushers created with a the same access token are not deleted on password change
+Can fetch a user's pushers
+Can add global push rule for room
+Can add global push rule for sender
+Can add global push rule for content
+Can add global push rule for override
+Can add global push rule for underride
+Can add global push rule for content
+New rules appear before old rules by default
+Can add global push rule before an existing rule
+Can add global push rule after an existing rule
+Can delete a push rule
+Can disable a push rule
+Adding the same push rule twice is idempotent
+Can change the actions of default rules
+Can change the actions of a user specified rule
+Adding a push rule wakes up an incremental /sync
+Disabling a push rule wakes up an incremental /sync
+Enabling a push rule wakes up an incremental /sync
+Setting actions for a push rule wakes up an incremental /sync
+Can enable/disable default rules
+Trying to add push rule with missing template fails with 400
+Trying to add push rule with missing rule_id fails with 400
+Trying to add push rule with empty rule_id fails with 400
+Trying to add push rule with invalid template fails with 400
+Trying to add push rule with rule_id with slashes fails with 400
+Trying to add push rule with override rule without conditions fails with 400
+Trying to add push rule with underride rule without conditions fails with 400
+Trying to add push rule with condition without kind fails with 400
+Trying to add push rule with content rule without pattern fails with 400
+Trying to add push rule with no actions fails with 400
+Trying to add push rule with invalid action fails with 400
+Trying to add push rule with invalid attr fails with 400
+Trying to add push rule with invalid value for enabled fails with 400
+Trying to get push rules with no trailing slash fails with 400
+Trying to get push rules with scope without trailing slash fails with 400
+Trying to get push rules with template without tailing slash fails with 400
+Trying to get push rules with unknown scope fails with 400
+Trying to get push rules with unknown template fails with 400
+Trying to get push rules with unknown attribute fails with 400
+Getting push rules doesn't corrupt the cache SYN-390
+Test that a message is pushed
+Invites are pushed
+Rooms with names are correctly named in pushes
+Rooms with canonical alias are correctly named in pushed
+Rooms with many users are correctly pushed
+Don't get pushed for rooms you've muted
+Rejected events are not pushed
+Test that rejected pushers are removed.
+Notifications can be viewed with GET /notifications
+Trying to add push rule with no scope fails with 400
+Trying to add push rule with invalid scope fails with 400
 Forward extremities remain so even after the next events are populated as outliers
 If a device list update goes missing, the server resyncs on the next one
 uploading self-signing key notifies over federation
@@ -607,4 +660,4 @@ registration accepts non-ascii passwords
 registration with inhibit_login inhibits login
 The operation must be consistent through an interactive authentication session
 Multiple calls to /sync should not cause 500 errors
-
+/context/ with lazy_load_members filter works