aboutsummaryrefslogtreecommitdiff
path: root/syncapi
AgeCommit message (Collapse)Author
2022-10-05Remove empty fields from `/sync` response (#2755)Till
First attempt at removing empty fields from `/sync` responses. Needs https://github.com/matrix-org/sytest/pull/1298 to keep Sytest happy. Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-05Optimise `QuerySharedUsers` so that we can only work on local users (#2766)Neil Alexander
Otherwise the sync API key change consumer wastes a lot of time trying to wake up the notifiers for non-local users.
2022-10-04Add indexes that optimise `selectStateInRangeSQL` (#2764)Neil Alexander
This gets rid of some expensive scans on `add_state_ids` and `remove_state_ids`, turning them into much cheaper and faster index scans instead.
2022-10-03Fix fragility of selectEventsWithEventIDsSQL queries (#2757)Ashley Nelson
This fixes a temporary workaround with the `selectEventsWithEventIDsSQL` queries where fields need to be artificially added to the queries so the row results match the format of the `syncapi_output_room_events` table. I made similar functions that accept row results from the `syncapi_current_room_state` table and convert them into StreamEvents without the fields that are specific to output room events. There is also a unit test in the first commit to ensure the resulting behavior doesn't change from the modified queries and functions. Fixes #601. ### Pull Request Checklist <!-- Please read docs/CONTRIBUTING.md before submitting your pull request --> * [x] I have added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Ashley Nelson <fant@shley.email>` Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-03Modify sync transaction behaviour (#2758)Neil Alexander
This now uses a transaction per stream, so that errors in one stream don't propagate to another, and we therefore no longer need to do hacks to reopen a new transaction after aborting a failed one.
2022-09-30Reset transaction after a failureNeil Alexander
2022-09-30Handle case when applying history visibility failedNeil Alexander
2022-09-30Another `/sync` fixNeil Alexander
2022-09-30Fix broken `/sync` due to transaction errorNeil Alexander
2022-09-30Transactional isolation for `/sync` (#2745)Neil Alexander
This should transactional snapshot isolation for `/sync` etc requests. For now we don't use repeatable read due to some odd test failures with invites.
2022-09-28Consistent `*sql.Tx` usage across sync API (#2744)Neil Alexander
This tidies up the `storage` package so that everything takes a transaction parameter instead of something things that do and some that don't.
2022-09-28Fix for `sql: converting argument $1 type: unsupported type []interfa… (#2743)texuf
…ce {}, a slice of interface` in new notifications select The sqlite3 version was just not working, original pr here: https://github.com/matrix-org/dendrite/pull/2688 signed off by: austin ellis <austin@hntlabs.com> This doesn't fix the notification counts, they still only work about 1 out of every 5 times in my tests. I will stick with my other fix locally for reliable notification delivery: https://github.com/matrix-org/dendrite/pull/2701
2022-09-27Promote reindexing log levelNeil Alexander
2022-09-27Fulltext implementation using Bleve (#2675)Till
Based on #2480 This actually indexes events based on their event type. They are removed from the index if we receive a `m.room.redaction` event on the `OutputRoomEvent` stream. An admin endpoint is added to reindex all existing events. Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27Refactor notifications (#2688)Till
This PR changes the handling of notifications - removes the `StreamEvent` and `ReadUpdate` stream - listens on the `OutputRoomEvent` stream in the UserAPI to inform the SyncAPI about unread notifications - listens on the `OutputReceiptEvent` stream in the UserAPI to set receipts/update notifications - sets the `read_markers` directly from within the internal UserAPI Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27Improve selectRoomIDsWithAnyMembershipSQL performance (#2738)PiotrKozimor
Recently I have observed that dendrite spends a lot of time (~390s) in `selectRoomIDsWithAnyMembershipSQL` query ``` dendrite_syncapi=# select total_exec_time, left(query,100) from pg_stat_statements order by total_exec_time desc limit 5 ; total_exec_time | left --------------------+------------------------------------------------------------------------------------------------------ 747826.5800519128 | SELECT event_id, id, headered_event_json, session_id, exclude_from_sync, transaction_id, history_vis 389130.5490339942 | SELECT DISTINCT room_id, membership FROM syncapi_current_room_state WHERE type = $2 AND state_key = 376104.17514700035 | SELECT psd.datname, xact_commit, xact_rollback, blks_read, blks_hit, tup_returned, tup_fetched, tup_ 363644.164092031 | SELECT event_type_nid, event_state_key_nid, event_nid FROM roomserver_events WHERE event_nid = ANY($ 58570.48104699995 | SELECT event_id, headered_event_json FROM syncapi_current_room_state WHERE room_id = $1 AND ( $2::te (5 rows) ``` Explain analyze showed correct usage of `syncapi_room_state_unique` index: ``` dendrite_syncapi=# explain analyze SELECT distinct room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com'; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Unique (cost=2749.38..2749.56 rows=24 width=52) (actual time=2.933..2.956 rows=65 loops=1) -> Sort (cost=2749.38..2749.44 rows=24 width=52) (actual time=2.932..2.937 rows=65 loops=1) Sort Key: room_id, membership Sort Method: quicksort Memory: 34kB -> Index Scan using syncapi_room_state_unique on syncapi_current_room_state (cost=0.41..2748.83 rows=24 width=52) (actual time=0.030..2.890 rows=65 loops=1) Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Planning Time: 0.140 ms Execution Time: 2.990 ms (8 rows) ``` Multi-column indexes in Postgres shall perform well for leftmost columns, but I gave it a try and created `syncapi_current_room_state_type_state_key_idx` index. I could observe significant performance improvement. Execution time dropped from 2.9 ms to 0.24 ms: ``` explain analyze SELECT distinct room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com'; QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unique (cost=96.46..96.64 rows=24 width=52) (actual time=0.199..0.218 rows=65 loops=1) -> Sort (cost=96.46..96.52 rows=24 width=52) (actual time=0.199..0.202 rows=65 loops=1) Sort Key: room_id, membership Sort Method: quicksort Memory: 34kB -> Bitmap Heap Scan on syncapi_current_room_state (cost=4.53..95.91 rows=24 width=52) (actual time=0.048..0.139 rows=65 loops=1) Recheck Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Heap Blocks: exact=59 -> Bitmap Index Scan on syncapi_current_room_state_type_state_key_idx (cost=0.00..4.53 rows=24 width=0) (actual time=0.037..0.037 rows=65 loops=1) Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Planning Time: 0.236 ms Execution Time: 0.242 ms (11 rows) ``` Next improvement is skipping DISTINCT and rely on map assignment in `SelectRoomIDsWithAnyMembership`. Execution time drops by almost half: ``` explain analyze SELECT room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com'; QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------- Bitmap Heap Scan on syncapi_current_room_state (cost=4.53..95.91 rows=24 width=52) (actual time=0.032..0.113 rows=65 loops=1) Recheck Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Heap Blocks: exact=59 -> Bitmap Index Scan on syncapi_current_room_state_type_state_key_idx (cost=0.00..4.53 rows=24 width=0) (actual time=0.021..0.021 rows=65 loops=1) Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Planning Time: 0.087 ms Execution Time: 0.136 ms (7 rows) ``` In our env we spend only 1s on inserting to table, so the write penalty of creating an index should be small. ``` dendrite_syncapi=# select total_exec_time, left(query,100) from pg_stat_statements where query like '%INSERT%syncapi_current_room_state%' order by total_exec_time desc; total_exec_time | left --------------------+------------------------------------------------------------------------------------------------------ 1139.9057619999971 | INSERT INTO syncapi_current_room_state (room_id, event_id, type, sender, contains_url, state_key, he (1 row) ``` This PR does not require test modifications. ### Pull Request Checklist <!-- Please read docs/CONTRIBUTING.md before submitting your pull request --> * [x] I have added added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Piotr Kozimor <p1996k@gmail.com>`
2022-09-22Get the `DeviceListPosition` before anything else in complete syncs (#2733)Till
This should hopefully unflake `Can query remote device keys using POST` in Complement.
2022-09-20Remove deleted state logging lines from sync API (they are pointless)Neil Alexander
2022-09-20Mark device list as stale, if we don't have the requesting device (#2728)Till
This hopefully makes E2EE chats a little bit more reliable by re-syncing devices if we don't have the `requesting_device_id` in our database. (As seen in [Synapse](https://github.com/matrix-org/synapse/blob/c52abc1cfdd9e5480cdb4a03d626fe61cacc6573/synapse/handlers/devicemessage.py#L157-L201))
2022-09-13Tweak `LoadMembershipAtEvent` behaviour when state not known (#2716)Neil Alexander
Previously `LoadMembershipAtEvent` would fail if the state before one of the events was not known, i.e. because it was an outlier. This modifies it so that it gracefully handles not knowing the state and returns no memberships instead, so that history visibility doesn't freak out and kill `/sync` requests dead.
2022-09-13Send-to-device consumer/producer tweaks (#2713)Till
Some tweaks for the send-to-device consumers/producers: - use `json.RawMessage` without marshalling it first - try further devices (if available) if we failed to `PublishMsg` in the producers - some logging changes (to better debug E2EE issues)
2022-09-09Optimise `SharedUsers` again by using complete composite indexNeil Alexander
2022-09-09Optimise shared users query in Synx API slightly by removing a potential sortNeil Alexander
2022-09-09More writer fixes in the Sync APINeil Alexander
2022-08-31Allow batching in `JetStreamConsumer` (#2686)Neil Alexander
This allows us to receive more than one message from NATS at a time if we want.
2022-08-30Index on `syncapi_send_to_device` table (#2684)PiotrKozimor
Introduced index improves select query performance. Example execution time of `selectSendToDeviceMessagesSQL` query dropped from 80 ms to 15 ms. No sytest modifications are required. ### Pull Request Checklist * [x] I have added added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Piotr Kozimor <p1996k@gmail.com>`
2022-08-30Use existing limit instead of default limit when lazy loading members (#2682)Till
This should fix an issue where we return less than the expected membership events, when doing an initial sync. When doing an initial sync, the state limit is set to `math.MaxInt32`, while the default filter is set to 20.
2022-08-25SyncAPI tweaks/fixes (#2671)Till
- Reverts 9dc57122d991d54ea6750448ba88c8763a569830 as it was causing issues https://github.com/matrix-org/dendrite/issues/2660 - Updates the GMSL `DefaultStateFilter` to use a limit of 20 events - Uses the timeline events to determine the new position instead of the state events
2022-08-24Allow un-rejecting events on reprocessingNeil Alexander
2022-08-19Fetch more data for newly joined rooms in an incremental sync (#2657)Till
If we've joined a new room in an incremental sync, try fetching more data. This deflakes the complement server notices test (at least in my tests).
2022-08-19Set historyVisibility for backfilled events over federation (#2656)Till
This should hopefully deflake Backfill works correctly with history visibility set to joined as we were using the default shared visibility, even if the events are set to joined (or something else)
2022-08-19Make SyncAPI unit tests more reliable (#2655)Till
This should hopefully make some SyncAPI tests more reliable
2022-08-18Add a delay after sending events to the roomserverTill Faelligen
2022-08-16Stop spamming the logs with `StateBetween: ignoring deleted state` event IDsNeil Alexander
2022-08-16Lazy-loading fixes (#2646)Neil Alexander
* Use existing current room state if we have it * Don't dedupe before applying the history vis filter * Revert "Don't dedupe before applying the history vis filter" This reverts commit d27c4a0874dabb77c2eda6b23eb7c00478bc9e90. * Revert "Use existing current room state if we have it" This reverts commit 5819b4a7ce511204c4fb48d3c4741612b136e2ea. * Tweaks
2022-08-16Only return non-retired invites (#2643)Till
* Only return non-retired invites * Revert "Only return non-retired invites" This reverts commit 1150aa7f385b7d7cf5378297f3e17566d5aabcc6. * Check if we're doing an initial sync in the stream
2022-08-11Implement history visibility on `/messages`, `/context`, `/sync` (#2511)Till
* Add possibility to set history_visibility and user AccountType * Add new DB queries * Add actual history_visibility changes for /messages * Add passing tests * Extract check function * Cleanup * Cleanup * Fix build on 386 * Move ApplyHistoryVisibilityFilter to internal * Move queries to topology table * Add filtering to /sync and /context Some cleanup * Add passing tests; Remove failing tests :( * Re-add passing tests * Move filtering to own function to avoid duplication * Re-add passing test * Use newly added GMSL HistoryVisibility * Update gomatrixserverlib * Set the visibility when creating events * Default to shared history visibility * Remove unused query * Update history visibility checks to use gmsl Update tests * Remove unused statement * Update migrations to set "correct" history visibility * Add method to fetch the membership at a given event * Tweaks and logging * Use actual internal rsAPI, default to shared visibility in tests * Revert "Move queries to topology table" This reverts commit 4f0d41be9c194a46379796435ce73e79203edbd6. * Remove noise/unneeded code * More cleanup * Try to optimize database requests * Fix imports * PR peview fixes/changes * Move setting history visibility to own migration, be more restrictive * Fix unit tests * Lint * Fix missing entries * Tweaks for incremental syncs * Adapt generic changes Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> Co-authored-by: kegsay <kegan@matrix.org>
2022-08-11Generic-based internal HTTP API (#2626)Neil Alexander
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API) * Add `PerformInvite` * More tweaks * Fix metric name * Fix LookupStateIDs * Lots of changes to clients * Some serverside stuff * Some error handling * Use paths as metric names * Revert "Use paths as metric names" This reverts commit a9323a6a343f5ce6461a2e5bd570fe06465f1b15. * Namespace metric names * Remove duplicate entry * Remove another duplicate entry * Tweak error handling * Some more tweaks * Update error behaviour * Some more error tweaking * Fix API path for `PerformDeleteKeys` * Fix another path * Tweak federation client proxying * Fix another path * Don't return typed nils * Some more tweaks, not that it makes any difference * Tweak federation client proxying * Maybe fix the key backup test
2022-08-09Send-to-device/sync tweaks (#2630)Till
* Always delete send to device messages * Omit empty to_device * Tweak /sync response to omit empty values
2022-08-05Don't panic if we fail to upsert account dataTill Faelligen
2022-08-05Invalidate lazyLoadCache if we're doing an initial sync (#2623)Till
* Bypass lazyLoadCache if we're doing an initial sync * Make the linter happy again? * Revert "Make the linter happy again?" This reverts commit 52a5691ba3c17c05698bcc6a13092090f27ace63. * Try that again * Invalidate LazyLoadCache on initial syncs * Remove unneeded check * Add TODO * Rename Invalite -> InvalidateLazyLoadedUser * Thanks IDE
2022-08-05Partly fix notification counts (#2621)Till
* Fix notification query * Also for SQLite * Move tests to whitelist * Revert "Move tests to whitelist" This reverts commit a7d0120019a111ce45a447ba40233d9c101e6e9b.
2022-08-05Do not use `ioutil` as it is deprecated (#2625)Neil Alexander
2022-08-05Fix linter issues (#2624)Till
* Try that again * All hail the mighty linter? * And once again * goimport all the things
2022-08-05Add race testing to tests, and fix a few small race conditions in the tests ↵Brian Meek
(#2587) * Add race testing to tests, and fix a few small race conditions in the tests * Enable run-sytest on MacOS * Remove deadlock detecting mutex, per code review feedback * Remove autoformatting related changes and a closure that is not needed * Adjust to importing nats client as 'natsclient' Signed-off-by: Brian Meek <brian@hntlabs.com> * Clarify the use of gooseMutex to proect goose internal state Signed-off-by: Brian Meek <brian@hntlabs.com> * Remove no longer needed mutex for guarding goose Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-08-03Fix syncapi shared users query & device lists (#2614)Till
* Fix query issue, only add "changed" users if we actually share a room * Avoid log spam if context is done * Undo changes to filterSharedUsers * Add logging again.. * Fix SQLite shared users query * Change query to include invited users
2022-08-02Delete correct Send-to-Device messages (#2608)Till
* Add send-to-device tests * Update tests, fix message deletion * PR comments
2022-08-02Fix issue with sync API not advancing. (#2603)sergekh2
Issue: During conversation, under some conditions, sync cookie is not advanced, and, as a result, client loops on the same sync API call creating high traffic and CPU load. Fix: pdu component of cookie was updated incorrectly.
2022-08-01De-race `CompleteSync` (#2601)Neil Alexander
The `err` was coming from outside of the goroutine and being written to by concurrent goroutines.
2022-07-25Update database migrations, remove goose (#2264)Till
* Add new db migration * Update migrations Remove goose * Add possibility to test direct upgrades * Try to fix WASM test * Add checks for specific migrations * Remove AddMigration Use WithTransaction Add Dendrite version to table * Fix linter issues * Update tests * Update comments, outdent if * Namespace migrations * Add direct upgrade tests, skipping over one version * Split migrations * Update go version in CI * Fix copy&paste mistake * Use contexts in migrations Co-authored-by: kegsay <kegan@matrix.org> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-18 15:37:50 +0000