aboutsummaryrefslogtreecommitdiff
path: root/syncapi/storage/postgres
AgeCommit message (Collapse)Author
2022-09-09Optimise `SharedUsers` again by using complete composite indexNeil Alexander
2022-09-09Optimise shared users query in Synx API slightly by removing a potential sortNeil Alexander
2022-09-09More writer fixes in the Sync APINeil Alexander
2022-08-30Index on `syncapi_send_to_device` table (#2684)PiotrKozimor
Introduced index improves select query performance. Example execution time of `selectSendToDeviceMessagesSQL` query dropped from 80 ms to 15 ms. No sytest modifications are required. ### Pull Request Checklist * [x] I have added added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Piotr Kozimor <p1996k@gmail.com>`
2022-08-16Stop spamming the logs with `StateBetween: ignoring deleted state` event IDsNeil Alexander
2022-08-11Implement history visibility on `/messages`, `/context`, `/sync` (#2511)Till
* Add possibility to set history_visibility and user AccountType * Add new DB queries * Add actual history_visibility changes for /messages * Add passing tests * Extract check function * Cleanup * Cleanup * Fix build on 386 * Move ApplyHistoryVisibilityFilter to internal * Move queries to topology table * Add filtering to /sync and /context Some cleanup * Add passing tests; Remove failing tests :( * Re-add passing tests * Move filtering to own function to avoid duplication * Re-add passing test * Use newly added GMSL HistoryVisibility * Update gomatrixserverlib * Set the visibility when creating events * Default to shared history visibility * Remove unused query * Update history visibility checks to use gmsl Update tests * Remove unused statement * Update migrations to set "correct" history visibility * Add method to fetch the membership at a given event * Tweaks and logging * Use actual internal rsAPI, default to shared visibility in tests * Revert "Move queries to topology table" This reverts commit 4f0d41be9c194a46379796435ce73e79203edbd6. * Remove noise/unneeded code * More cleanup * Try to optimize database requests * Fix imports * PR peview fixes/changes * Move setting history visibility to own migration, be more restrictive * Fix unit tests * Lint * Fix missing entries * Tweaks for incremental syncs * Adapt generic changes Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> Co-authored-by: kegsay <kegan@matrix.org>
2022-08-05Partly fix notification counts (#2621)Till
* Fix notification query * Also for SQLite * Move tests to whitelist * Revert "Move tests to whitelist" This reverts commit a7d0120019a111ce45a447ba40233d9c101e6e9b.
2022-08-03Fix syncapi shared users query & device lists (#2614)Till
* Fix query issue, only add "changed" users if we actually share a room * Avoid log spam if context is done * Undo changes to filterSharedUsers * Add logging again.. * Fix SQLite shared users query * Change query to include invited users
2022-08-02Delete correct Send-to-Device messages (#2608)Till
* Add send-to-device tests * Update tests, fix message deletion * PR comments
2022-07-25Update database migrations, remove goose (#2264)Till
* Add new db migration * Update migrations Remove goose * Add possibility to test direct upgrades * Try to fix WASM test * Add checks for specific migrations * Remove AddMigration Use WithTransaction Add Dendrite version to table * Fix linter issues * Update tests * Update comments, outdent if * Namespace migrations * Add direct upgrade tests, skipping over one version * Split migrations * Update go version in CI * Fix copy&paste mistake * Use contexts in migrations Co-authored-by: kegsay <kegan@matrix.org> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-18Set historyVisibility in rowsToStreamEventsTill Faelligen
2022-07-18History visibility database changes (#2533)Till
* Add new history_visibility column * Update SQL queries to include history_visibility * Store the history visibilty calculated by the roomserver * Update GMSL * Update migrations * Fix migration * Update GMSL * Fix `go.sum` * Update GMSL to use sql.Scanner & sql.Valuer * Re-order migration/table creation * Update gomatrixserverlib * Add history_visibility column to current_room_state * Fix migrations * Return error instead of Fatal log Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-15Use sync API database in `filterSharedUsers` (#2572)Neil Alexander
* Add function to the sync API storage package for filtering shared users * Use the database instead of asking the RS API * Fix unit tests * Fix map handling in `filterSharedUsers`
2022-07-12Minor SendToDevice fix (#2565)Till
* Avoid unnecessary marshalling if sending to the local server * Fix ordering of ToDevice messages * Revive SendToDevice test
2022-05-10Add indexes to `syncapi_output_room_events` table that satisfy the filters ↵Neil Alexander
(#2446)
2022-05-03Global database connection pool (for monolith mode) (#2411)Neil Alexander
* Allow monolith components to share a single database pool * Don't yell about missing connection strings * Rename field * Setup tweaks * Fix panic * Improve configuration checks * Update config * Fix lint errors * Update comments
2022-04-28Only load members of newly joined rooms (#2389)Till
* Only load members of newly joined rooms * Comment that the query is prepared at runtime Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-28Limit presence in `/sync` responses (#2394)Till
* Use filter and limit presence count * More limiting * More limiting * Fix unit test * Also limit presence by last_active_ts * Update query, use "from" as the initial lastPos * Get 1000 presence events, they are filtered later Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-27Fix #2390 (#2392)Till
Fix duplicate heroes in `/sync` response.
2022-04-27Try that againNeil Alexander
2022-04-27Correct account data position mappingNeil Alexander
2022-04-27Don't create fictitious presence entries (#2381)Neil Alexander
* Don't create fictitious presence entries for users that don't have any * Update whitelist, since that test probably shouldn't be passing * Fix panics
2022-04-26Send all account data on complete sync by defaultNeil Alexander
Squashed commit of the following: commit 0ec8de57261d573a5f88577aa9d7a1174d3999b9 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Apr 26 16:56:30 2022 +0100 Select filter onto provided target filter commit da40b6fffbf5737864b223f49900048f557941f9 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Apr 26 16:48:00 2022 +0100 Specify other field too commit ffc0b0801f63bb4d3061b6813e3ce5f3b4c8fbcb Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Apr 26 16:45:44 2022 +0100 Send as much account data as possible during complete sync
2022-04-26Fix account data positionNeil Alexander
2022-04-26Start account data ID from `from`Neil Alexander
2022-04-26Fix account_data not correctly send in a complete sync (#2379)Till
* Return the StreamPosition from the database and not the latest * Fix linter issue
2022-04-26Add heroes to the room summary (#2373)Till
* Implement room summary heroes * Add passing tests * Move MembershipCount to addRoomSummary * Add comments, close Statement
2022-04-13Make `/messages` filterable (#2347)Till
* Make /messages filterable Fix bug when determining if an event contains an URL * Add newly passing test * Fix test
2022-04-11Correctly use provided filters (#2339)Till
* Apply filters correctly * Fix issues; Use prepareWithFilters * Update gmsl & tests * go.mod.. * PR comments
2022-04-09Remove dead code in the sync api (#2341)kegsay
2022-04-08syncapi: add more tests; fix more bugs (#2338)kegsay
* syncapi: add more tests; fix more bugs bugfixes: - The postgres impl of TopologyTable.SelectEventIDsInRange did not use the provided txn - The postgres impl of EventsTable.SelectEvents did not preserve the ordering of the input event IDs in the output events slice - The sqlite impl of EventsTable.SelectEvents did not use a bulk `IN ($1)` query. Added tests: - `TestGetEventsInRangeWithTopologyToken` - `TestOutputRoomEventsTable` - `TestTopologyTable` * -p 1 for now
2022-04-07Add possibility to ignore users (#2329)Till
* Add ignore users * Ignore users in pushrules Add passing tests * Update sytest lists * Store ignore knowledge in the sync API * Fix copyrights Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-06Add presence module V2 (#2312)Till
* Syncapi presence * Clientapi http presence handler * Why is this here? * Missing files * FederationAPI presence implementation * Add new presence stream * Pinecone update * Pinecone update * Add passing tests * Make linter happy * Add presence producer * Add presence config option * Set user to unavailable after x minutes * Only set currently_active if online Avoid unneeded presence updates when syncing * Tweaks * Query devices for last_active_ts Fixes & tweaks * Export SharedUsers/SharedUsers * Presence stream in MemoryStorage * Remove status_msg_nil * Fix sytest crashes * Make presence types const and use stringer for it * Change options to allow inbound/outbound presence * Fix option & typo * Update configs Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-01Include joined and invite member counts in room summary (#2315)Neil Alexander
* Include joined and invite member counts in room summary This should fix #2314 and also fix the problem where some clients like Element Android, Fluffychat etc would display the wrong member count for a given room. * Improve SQLite query precision * Check existence of state key for membership events
2022-03-29Remove eduserver (#2306)S7evinK
* Move receipt sending to own JetStream producer * Move SendToDevice to producer * Remove most parts of the EDU server * Fix SendToDevice & copyrights * Move structs, cleanup EDU Server traces * Use HeadersOnly subscription * Missing file * Fix linter issues * Move consumers to own files * Rename durable consumer; Consumer cleanup * Docs/config cleanup
2022-03-24Fix lint errorsNeil Alexander
2022-03-21Remove unused partition_offset_table (#2288)S7evinK
2022-03-18Better mapping of stream positions to topological positions in `/messages` ↵Neil Alexander
(#2263) * Convert stream positions into topological positions for both `from` and `to` in `/messages` * Hopefully it works now * Remove unnecessary logging * Return sane values if `StreamToTopologicalPosition` can't work out the right thing to do * Revert logging change * tweaks * Fix `selectEventIDsInRangeASCSQL` * Test `Getting messages going forward is limited for a departed room (SPEC-216)` was passing incorrectly so un-whitelist it
2022-03-17Various bug fixes and tweaks around invites and membershipNeil Alexander
2022-03-16Fix SQL statement for PurgeRoomState (#2280)S7evinK
2022-03-11Don't range entire state for `/sync` (#2270)Neil Alexander
* Don't range entire state for rooms the user has no reason to care about * Remove unnecessary db field in postgresql
2022-03-03Implement Push Notifications (#1842)Dan
* Add Pushserver component with Pushers API Co-authored-by: Tommie Gannert <tommie@gannert.se> Co-authored-by: Dan Peleg <dan@globekeeper.com> * Wire Pushserver component Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> * Add PushGatewayClient. The full event format is required for Sytest. * Add a pushrules module. * Change user API account creation to use the new pushrules module's defaults. Introduces "scope" as required by client API, and some small field tweaks to make some 61push Sytests pass. * Add push rules query/put API in Pushserver. This manipulates account data over User API, and fires sync messages for changes. Those sync messages should, according to an existing TODO in clientapi, be moved to userapi. Forks clientapi/producers/syncapi.go to pushserver/ for later extension. * Add clientapi routes for push rules to Pushserver. A cleanup would be to move more of the name-splitting logic into pushrules.go, to depollute routing.go. * Output rooms.join.unread_notifications in /sync. This is the read-side. Pushserver will be the write-side. * Implement pushserver/storage for notifications. * Use PushGatewayClient and the pushrules module in Pushserver's room consumer. * Use one goroutine per user to avoid locking up the entire server for one bad push gateway. * Split pushing by format. * Send one device per push. Sytest does not support coalescing multiple devices into one push. Matches Synapse. Either we change Sytest, or remove the group-by-url-and-format logic. * Write OutputNotificationData from push server. Sync API is already the consumer. * Implement read receipt consumers in Pushserver. Supports m.read and m.fully_read receipts. * Add clientapi route for /unstable/notifications. * Rename to UpsertPusher for clarity and handle pusher update * Fix linter errors * Ignore body.Close() error check * Fix push server internal http wiring * Add 40 newly passing 61push tests to whitelist * Add next 12 newly passing 61push tests to whitelist * Send notification data before notifying users in EDU server consumer * NATS JetStream * Goodbye sarama * Fix `NewStreamTokenFromString` * Consume on the correct topic for the roomserver * Don't panic, NAK instead * Move push notifications into the User API * Don't set null values since that apparently causes Element upsetti * Also set omitempty on conditions * Fix bug so that we don't override the push rules unnecessarily * Tweak defaults * Update defaults * More tweaks * Move `/notifications` onto `r0`/`v3` mux * User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly Co-authored-by: Piotr Kozimor <p1996k@gmail.com> Co-authored-by: Tommie Gannert <tommie@gannert.se> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-01Hopefully fix read receipts (#2241)Neil Alexander
2022-02-21Implement `/context` (#2207)S7evinK
* Add QueryEventsAfter * Add /context * Make all tests pass on sqlite * Add queries to get the events for /context requests * Move /context to the syncapi * Revert "Add QueryEventsAfter" This reverts commit 440a771d10632622e8c65d35fe90f0804bc98862. * Simplify getting the required events * Apply RoomEventFilter when getting events * Add passing tests * Remove logging * Remove unused SQL statements Update comments & add TODO
2021-11-03Reduce CPU usage of SelectStateInRange (#2038)Neil Alexander
2021-03-03Increase gocyclo complexity to 25 (and remove all but 2 golint directives ↵Neil Alexander
related to it) (#1783)
2021-02-17Don't exclude an event from sync if it was previously not excluded (#1767)Neil Alexander
2021-02-04Don't re-request state events that are already in the timeline (#1739)Neil Alexander
* Don't request state events if we already have the timeline events (Postgres only) * Rename variable * nocyclo * Add SQLite * Tweaks * Revert query change * Don't dedupe if asking for full state * Update query
2021-02-04Fix ON CONFLICT on sync API account data (#1745) (#1750)Neil Alexander
2021-01-20Add sync API memberships table (#1726)Neil Alexander
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-23 09:50:05 +0000