aboutsummaryrefslogtreecommitdiff
path: root/syncapi
AgeCommit message (Collapse)Author
2022-11-01return required room_id field in /members (#2846)ash lea
### Pull Request Checklist <!-- Please read https://matrix-org.github.io/dendrite/development/contributing before submitting your pull request --> * [ ] I have added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off below using a legally identifiable name](https://matrix-org.github.io/dendrite/development/contributing#sign-off) _or_ I have already signed off privately Signed-off-by: `ash lea <example@thisismyactual.email>`
2022-11-01Try to fix a panic in the sync API PDU streamNeil Alexander
2022-10-31Tweak `"state"` and `"timeline"` filtering (#2844)Neil Alexander
This should stop state events disappearing down a gap where we'd try to separate out the sections *before* applying history visibility instead of after. This may be a better approach than #2843 but I hope @tak-hntlabs will shout if it isn't.
2022-10-28Limit recent events when going backwards (#2840)Till
If we're going backwards, we were selecting potentially thousands of events, which in turn were fed to history visibility checks, resulting in bad sync performance.
2022-10-28Simplify error checking and check the correct errorTill Faelligen
2022-10-27Fix /members (#2837)Till
Fixes a bug introduced in #2827, where the SyncAPI might not have all requested eventIDs, resulting in too few members returned.
2022-10-25Fix `Current state appears in timeline in private history with many messages ↵Till
after` (#2830) The problem was that we weren't getting enough recent events, as most of them were removed by the history visibility filter. Now we're getting all events between the given input range and re-slice the returned values after applying history visibility.
2022-10-25Return forbidden if not a member anymore (fix #2802)Till Faelligen
2022-10-25Filter `/members`, return members at given point (#2827)Till
Makes the tests ``` Can get rooms/{roomId}/members at a given point Can filter rooms/{roomId}/members ``` pass, by moving `/members` and `/joined_members` to the SyncAPI.
2022-10-24Get messages from before user left the room (#2824)Till
This is going to make `Can get rooms/{roomId}/messages for a departed room (SPEC-216)` pass, since we now only grep events from before the user left the room.
2022-10-21Add `syncapi_memberships` table tests (#2805)Till
2022-10-21Fix `invite -> leave -> join` dance when accepting invites (#2817)Till
As mentioned in https://github.com/matrix-org/dendrite/issues/2361#issuecomment-1139394565 and observed by ourselves, this should fix the odd `invite -> leave -> join` dance when accepting invites.
2022-10-19Get state deltas without filters (#2810)Till
This makes the following changes: - get state deltas without the user supplied filter, so we can actually "calculate" state transitions - closes `stmt` when using SQLite - Adds presence for users who newly joined a room, even if the syncing user already knows about the presence status (should fix https://github.com/matrix-org/complement/pull/516)
2022-10-14Always add `UnreadNotifications` to joined room reponses (#2793)Till
Fixes a minor bug, where we failed to add `UnreadNotifications` to the join response, if it wasn't in `GetUserUnreadNotificationCountsForRooms`.
2022-10-13Event relations (#2790)Neil Alexander
This adds support for tracking `m.relates_to`, as well as adding support for the various `/room/{roomID}/relations/...` endpoints to the CS API.
2022-10-11Private read receipts (#2789)Neil Alexander
Implement behaviours for `m.read.private` receipts.
2022-10-11Tweak `GetStateDeltas` behaviour (#2788)Neil Alexander
Improves the control flow of `GetStateDeltas` for clarity and possibly also fixes a bug where duplicate state delta entries could be inserted with different memberships instead of being correctly overridden by `join`.
2022-10-10Move /event to the SyncAPI (#2782)Till
This allows us to apply history visibility without having to recalculate it in the roomserver. Unblocks https://github.com/matrix-org/complement/pull/495, fix missing part of https://github.com/matrix-org/dendrite/issues/617
2022-10-07Ensure we only wake up a given user once (#2775)Neil Alexander
This ensures that the sync API notifier only wakes up a given user once for a given stream position.
2022-10-06Use the stream positions of the notifier (#2768)Till
Use the stream positions of the notifier, which might have advanced since setting it at the beginning of the loop. This possibly helps in reducing roundtrips to the SyncAPI, just because we didn't fetch the latest data. Also fixes a minor oversight in the receipts stream.
2022-10-05Remove empty fields from `/sync` response (#2755)Till
First attempt at removing empty fields from `/sync` responses. Needs https://github.com/matrix-org/sytest/pull/1298 to keep Sytest happy. Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-05Optimise `QuerySharedUsers` so that we can only work on local users (#2766)Neil Alexander
Otherwise the sync API key change consumer wastes a lot of time trying to wake up the notifiers for non-local users.
2022-10-04Add indexes that optimise `selectStateInRangeSQL` (#2764)Neil Alexander
This gets rid of some expensive scans on `add_state_ids` and `remove_state_ids`, turning them into much cheaper and faster index scans instead.
2022-10-03Fix fragility of selectEventsWithEventIDsSQL queries (#2757)Ashley Nelson
This fixes a temporary workaround with the `selectEventsWithEventIDsSQL` queries where fields need to be artificially added to the queries so the row results match the format of the `syncapi_output_room_events` table. I made similar functions that accept row results from the `syncapi_current_room_state` table and convert them into StreamEvents without the fields that are specific to output room events. There is also a unit test in the first commit to ensure the resulting behavior doesn't change from the modified queries and functions. Fixes #601. ### Pull Request Checklist <!-- Please read docs/CONTRIBUTING.md before submitting your pull request --> * [x] I have added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Ashley Nelson <fant@shley.email>` Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-03Modify sync transaction behaviour (#2758)Neil Alexander
This now uses a transaction per stream, so that errors in one stream don't propagate to another, and we therefore no longer need to do hacks to reopen a new transaction after aborting a failed one.
2022-09-30Reset transaction after a failureNeil Alexander
2022-09-30Handle case when applying history visibility failedNeil Alexander
2022-09-30Another `/sync` fixNeil Alexander
2022-09-30Fix broken `/sync` due to transaction errorNeil Alexander
2022-09-30Transactional isolation for `/sync` (#2745)Neil Alexander
This should transactional snapshot isolation for `/sync` etc requests. For now we don't use repeatable read due to some odd test failures with invites.
2022-09-28Consistent `*sql.Tx` usage across sync API (#2744)Neil Alexander
This tidies up the `storage` package so that everything takes a transaction parameter instead of something things that do and some that don't.
2022-09-28Fix for `sql: converting argument $1 type: unsupported type []interfa… (#2743)texuf
…ce {}, a slice of interface` in new notifications select The sqlite3 version was just not working, original pr here: https://github.com/matrix-org/dendrite/pull/2688 signed off by: austin ellis <austin@hntlabs.com> This doesn't fix the notification counts, they still only work about 1 out of every 5 times in my tests. I will stick with my other fix locally for reliable notification delivery: https://github.com/matrix-org/dendrite/pull/2701
2022-09-27Promote reindexing log levelNeil Alexander
2022-09-27Fulltext implementation using Bleve (#2675)Till
Based on #2480 This actually indexes events based on their event type. They are removed from the index if we receive a `m.room.redaction` event on the `OutputRoomEvent` stream. An admin endpoint is added to reindex all existing events. Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27Refactor notifications (#2688)Till
This PR changes the handling of notifications - removes the `StreamEvent` and `ReadUpdate` stream - listens on the `OutputRoomEvent` stream in the UserAPI to inform the SyncAPI about unread notifications - listens on the `OutputReceiptEvent` stream in the UserAPI to set receipts/update notifications - sets the `read_markers` directly from within the internal UserAPI Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27Improve selectRoomIDsWithAnyMembershipSQL performance (#2738)PiotrKozimor
Recently I have observed that dendrite spends a lot of time (~390s) in `selectRoomIDsWithAnyMembershipSQL` query ``` dendrite_syncapi=# select total_exec_time, left(query,100) from pg_stat_statements order by total_exec_time desc limit 5 ; total_exec_time | left --------------------+------------------------------------------------------------------------------------------------------ 747826.5800519128 | SELECT event_id, id, headered_event_json, session_id, exclude_from_sync, transaction_id, history_vis 389130.5490339942 | SELECT DISTINCT room_id, membership FROM syncapi_current_room_state WHERE type = $2 AND state_key = 376104.17514700035 | SELECT psd.datname, xact_commit, xact_rollback, blks_read, blks_hit, tup_returned, tup_fetched, tup_ 363644.164092031 | SELECT event_type_nid, event_state_key_nid, event_nid FROM roomserver_events WHERE event_nid = ANY($ 58570.48104699995 | SELECT event_id, headered_event_json FROM syncapi_current_room_state WHERE room_id = $1 AND ( $2::te (5 rows) ``` Explain analyze showed correct usage of `syncapi_room_state_unique` index: ``` dendrite_syncapi=# explain analyze SELECT distinct room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com'; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Unique (cost=2749.38..2749.56 rows=24 width=52) (actual time=2.933..2.956 rows=65 loops=1) -> Sort (cost=2749.38..2749.44 rows=24 width=52) (actual time=2.932..2.937 rows=65 loops=1) Sort Key: room_id, membership Sort Method: quicksort Memory: 34kB -> Index Scan using syncapi_room_state_unique on syncapi_current_room_state (cost=0.41..2748.83 rows=24 width=52) (actual time=0.030..2.890 rows=65 loops=1) Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Planning Time: 0.140 ms Execution Time: 2.990 ms (8 rows) ``` Multi-column indexes in Postgres shall perform well for leftmost columns, but I gave it a try and created `syncapi_current_room_state_type_state_key_idx` index. I could observe significant performance improvement. Execution time dropped from 2.9 ms to 0.24 ms: ``` explain analyze SELECT distinct room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com'; QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unique (cost=96.46..96.64 rows=24 width=52) (actual time=0.199..0.218 rows=65 loops=1) -> Sort (cost=96.46..96.52 rows=24 width=52) (actual time=0.199..0.202 rows=65 loops=1) Sort Key: room_id, membership Sort Method: quicksort Memory: 34kB -> Bitmap Heap Scan on syncapi_current_room_state (cost=4.53..95.91 rows=24 width=52) (actual time=0.048..0.139 rows=65 loops=1) Recheck Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Heap Blocks: exact=59 -> Bitmap Index Scan on syncapi_current_room_state_type_state_key_idx (cost=0.00..4.53 rows=24 width=0) (actual time=0.037..0.037 rows=65 loops=1) Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Planning Time: 0.236 ms Execution Time: 0.242 ms (11 rows) ``` Next improvement is skipping DISTINCT and rely on map assignment in `SelectRoomIDsWithAnyMembership`. Execution time drops by almost half: ``` explain analyze SELECT room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com'; QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------- Bitmap Heap Scan on syncapi_current_room_state (cost=4.53..95.91 rows=24 width=52) (actual time=0.032..0.113 rows=65 loops=1) Recheck Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Heap Blocks: exact=59 -> Bitmap Index Scan on syncapi_current_room_state_type_state_key_idx (cost=0.00..4.53 rows=24 width=0) (actual time=0.021..0.021 rows=65 loops=1) Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Planning Time: 0.087 ms Execution Time: 0.136 ms (7 rows) ``` In our env we spend only 1s on inserting to table, so the write penalty of creating an index should be small. ``` dendrite_syncapi=# select total_exec_time, left(query,100) from pg_stat_statements where query like '%INSERT%syncapi_current_room_state%' order by total_exec_time desc; total_exec_time | left --------------------+------------------------------------------------------------------------------------------------------ 1139.9057619999971 | INSERT INTO syncapi_current_room_state (room_id, event_id, type, sender, contains_url, state_key, he (1 row) ``` This PR does not require test modifications. ### Pull Request Checklist <!-- Please read docs/CONTRIBUTING.md before submitting your pull request --> * [x] I have added added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Piotr Kozimor <p1996k@gmail.com>`
2022-09-22Get the `DeviceListPosition` before anything else in complete syncs (#2733)Till
This should hopefully unflake `Can query remote device keys using POST` in Complement.
2022-09-20Remove deleted state logging lines from sync API (they are pointless)Neil Alexander
2022-09-20Mark device list as stale, if we don't have the requesting device (#2728)Till
This hopefully makes E2EE chats a little bit more reliable by re-syncing devices if we don't have the `requesting_device_id` in our database. (As seen in [Synapse](https://github.com/matrix-org/synapse/blob/c52abc1cfdd9e5480cdb4a03d626fe61cacc6573/synapse/handlers/devicemessage.py#L157-L201))
2022-09-13Tweak `LoadMembershipAtEvent` behaviour when state not known (#2716)Neil Alexander
Previously `LoadMembershipAtEvent` would fail if the state before one of the events was not known, i.e. because it was an outlier. This modifies it so that it gracefully handles not knowing the state and returns no memberships instead, so that history visibility doesn't freak out and kill `/sync` requests dead.
2022-09-13Send-to-device consumer/producer tweaks (#2713)Till
Some tweaks for the send-to-device consumers/producers: - use `json.RawMessage` without marshalling it first - try further devices (if available) if we failed to `PublishMsg` in the producers - some logging changes (to better debug E2EE issues)
2022-09-09Optimise `SharedUsers` again by using complete composite indexNeil Alexander
2022-09-09Optimise shared users query in Synx API slightly by removing a potential sortNeil Alexander
2022-09-09More writer fixes in the Sync APINeil Alexander
2022-08-31Allow batching in `JetStreamConsumer` (#2686)Neil Alexander
This allows us to receive more than one message from NATS at a time if we want.
2022-08-30Index on `syncapi_send_to_device` table (#2684)PiotrKozimor
Introduced index improves select query performance. Example execution time of `selectSendToDeviceMessagesSQL` query dropped from 80 ms to 15 ms. No sytest modifications are required. ### Pull Request Checklist * [x] I have added added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Piotr Kozimor <p1996k@gmail.com>`
2022-08-30Use existing limit instead of default limit when lazy loading members (#2682)Till
This should fix an issue where we return less than the expected membership events, when doing an initial sync. When doing an initial sync, the state limit is set to `math.MaxInt32`, while the default filter is set to 20.
2022-08-25SyncAPI tweaks/fixes (#2671)Till
- Reverts 9dc57122d991d54ea6750448ba88c8763a569830 as it was causing issues https://github.com/matrix-org/dendrite/issues/2660 - Updates the GMSL `DefaultStateFilter` to use a limit of 20 events - Uses the timeline events to determine the new position instead of the state events
2022-08-24Allow un-rejecting events on reprocessingNeil Alexander
2022-08-19Fetch more data for newly joined rooms in an incremental sync (#2657)Till
If we've joined a new room in an incremental sync, try fetching more data. This deflakes the complement server notices test (at least in my tests).
generated by cgit v1.2.3 (git 2.26.2) at 2024-11-22 19:50:02 +0000