aboutsummaryrefslogtreecommitdiff
path: root/syncapi/streams
AgeCommit message (Collapse)Author
2022-09-13Tweak `LoadMembershipAtEvent` behaviour when state not known (#2716)Neil Alexander
Previously `LoadMembershipAtEvent` would fail if the state before one of the events was not known, i.e. because it was an outlier. This modifies it so that it gracefully handles not knowing the state and returns no memberships instead, so that history visibility doesn't freak out and kill `/sync` requests dead.
2022-08-30Use existing limit instead of default limit when lazy loading members (#2682)Till
This should fix an issue where we return less than the expected membership events, when doing an initial sync. When doing an initial sync, the state limit is set to `math.MaxInt32`, while the default filter is set to 20.
2022-08-25SyncAPI tweaks/fixes (#2671)Till
- Reverts 9dc57122d991d54ea6750448ba88c8763a569830 as it was causing issues https://github.com/matrix-org/dendrite/issues/2660 - Updates the GMSL `DefaultStateFilter` to use a limit of 20 events - Uses the timeline events to determine the new position instead of the state events
2022-08-19Fetch more data for newly joined rooms in an incremental sync (#2657)Till
If we've joined a new room in an incremental sync, try fetching more data. This deflakes the complement server notices test (at least in my tests).
2022-08-16Lazy-loading fixes (#2646)Neil Alexander
* Use existing current room state if we have it * Don't dedupe before applying the history vis filter * Revert "Don't dedupe before applying the history vis filter" This reverts commit d27c4a0874dabb77c2eda6b23eb7c00478bc9e90. * Revert "Use existing current room state if we have it" This reverts commit 5819b4a7ce511204c4fb48d3c4741612b136e2ea. * Tweaks
2022-08-16Only return non-retired invites (#2643)Till
* Only return non-retired invites * Revert "Only return non-retired invites" This reverts commit 1150aa7f385b7d7cf5378297f3e17566d5aabcc6. * Check if we're doing an initial sync in the stream
2022-08-11Implement history visibility on `/messages`, `/context`, `/sync` (#2511)Till
* Add possibility to set history_visibility and user AccountType * Add new DB queries * Add actual history_visibility changes for /messages * Add passing tests * Extract check function * Cleanup * Cleanup * Fix build on 386 * Move ApplyHistoryVisibilityFilter to internal * Move queries to topology table * Add filtering to /sync and /context Some cleanup * Add passing tests; Remove failing tests :( * Re-add passing tests * Move filtering to own function to avoid duplication * Re-add passing test * Use newly added GMSL HistoryVisibility * Update gomatrixserverlib * Set the visibility when creating events * Default to shared history visibility * Remove unused query * Update history visibility checks to use gmsl Update tests * Remove unused statement * Update migrations to set "correct" history visibility * Add method to fetch the membership at a given event * Tweaks and logging * Use actual internal rsAPI, default to shared visibility in tests * Revert "Move queries to topology table" This reverts commit 4f0d41be9c194a46379796435ce73e79203edbd6. * Remove noise/unneeded code * More cleanup * Try to optimize database requests * Fix imports * PR peview fixes/changes * Move setting history visibility to own migration, be more restrictive * Fix unit tests * Lint * Fix missing entries * Tweaks for incremental syncs * Adapt generic changes Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> Co-authored-by: kegsay <kegan@matrix.org>
2022-08-09Send-to-device/sync tweaks (#2630)Till
* Always delete send to device messages * Omit empty to_device * Tweak /sync response to omit empty values
2022-08-05Invalidate lazyLoadCache if we're doing an initial sync (#2623)Till
* Bypass lazyLoadCache if we're doing an initial sync * Make the linter happy again? * Revert "Make the linter happy again?" This reverts commit 52a5691ba3c17c05698bcc6a13092090f27ace63. * Try that again * Invalidate LazyLoadCache on initial syncs * Remove unneeded check * Add TODO * Rename Invalite -> InvalidateLazyLoadedUser * Thanks IDE
2022-08-02Fix issue with sync API not advancing. (#2603)sergekh2
Issue: During conversation, under some conditions, sync cookie is not advanced, and, as a result, client loops on the same sync API call creating high traffic and CPU load. Fix: pdu component of cookie was updated incorrectly.
2022-08-01De-race `CompleteSync` (#2601)Neil Alexander
The `err` was coming from outside of the goroutine and being written to by concurrent goroutines.
2022-07-15Use sync API database in `filterSharedUsers` (#2572)Neil Alexander
* Add function to the sync API storage package for filtering shared users * Use the database instead of asking the RS API * Fix unit tests * Fix map handling in `filterSharedUsers`
2022-06-30Silence presence logs (#2547)Till
2022-05-06Move LL cache (#2429)Till
2022-05-05syncapi: define specific interfaces for internal HTTP communications (#2416)kegsay
* syncapi: use finer-grained interfaces when making the syncapi * Use specific interfaces for syncapi-roomserver interactions * Define query access token api for shared http auth code
2022-04-30Return "to", if we didn't return any presence events (#2407)Till
Return correct stream position, if we didn't return any presence events
2022-04-28Only load members of newly joined rooms (#2389)Till
* Only load members of newly joined rooms * Comment that the query is prepared at runtime Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-28Limit presence in `/sync` responses (#2394)Till
* Use filter and limit presence count * More limiting * More limiting * Fix unit test * Also limit presence by last_active_ts * Update query, use "from" as the initial lastPos * Get 1000 presence events, they are filtered later Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-27Don't send account data or receipts for left/forgotten rooms (#2382)Neil Alexander
* Only include account data and receipts for rooms in a complete sync that we care about * Fix global account data
2022-04-27Don't create fictitious presence entries (#2381)Neil Alexander
* Don't create fictitious presence entries for users that don't have any * Update whitelist, since that test probably shouldn't be passing * Fix panics
2022-04-26Fix account_data not correctly send in a complete sync (#2379)Till
* Return the StreamPosition from the database and not the latest * Fix linter issue
2022-04-26Add heroes to the room summary (#2373)Till
* Implement room summary heroes * Add passing tests * Move MembershipCount to addRoomSummary * Add comments, close Statement
2022-04-25Use provided filter for account_data (#2372)Till
* Reuse IncrementalSync, use provided filter * Inform SyncAPI about newly created push_rules
2022-04-20Eliminate more SQL no row errors in sync API (#2363)Neil Alexander
* Handle `sql.ErrNoRows` in main `/sync` codepaths * Catch more
2022-04-20Lazy loading fixes (#2362)Neil Alexander
* Return some more usefully wrapped errors when doing sync * Remove unnecessary error check * Couple of guards around `sql.ErrNoRows` * Nolint
2022-04-19Implement lazy loading on `/sync` (#2346)Till
* Initial work on lazyloading * Partially implement lazy loading on /sync * Rename methods * Make missing tests pass * Preallocate slice, even if it will end up with fewer values * Let the cache handle the user mapping * Linter * Cap cache growth
2022-04-11Correctly use provided filters (#2339)Till
* Apply filters correctly * Fix issues; Use prepareWithFilters * Update gmsl & tests * go.mod.. * PR comments
2022-04-07Add possibility to ignore users (#2329)Till
* Add ignore users * Ignore users in pushrules Add passing tests * Update sytest lists * Store ignore knowledge in the sync API * Fix copyrights Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-07Simplify presence stringification (should help with ↵Neil Alexander
vector-im/element-android#5712)
2022-04-06Reduce allocations in `/sync` presence stream (#2326)Neil Alexander
* Reduce allocations on presence * Try to reduce allocations further * Tweak `IsSharedUser` some more * Take map lock
2022-04-06Add presence module V2 (#2312)Till
* Syncapi presence * Clientapi http presence handler * Why is this here? * Missing files * FederationAPI presence implementation * Add new presence stream * Pinecone update * Pinecone update * Add passing tests * Make linter happy * Add presence producer * Add presence config option * Set user to unavailable after x minutes * Only set currently_active if online Avoid unneeded presence updates when syncing * Tweaks * Query devices for last_active_ts Fixes & tweaks * Export SharedUsers/SharedUsers * Presence stream in MemoryStorage * Remove status_msg_nil * Fix sytest crashes * Make presence types const and use stringer for it * Change options to allow inbound/outbound presence * Fix option & typo * Update configs Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-01Include joined and invite member counts in room summary (#2315)Neil Alexander
* Include joined and invite member counts in room summary This should fix #2314 and also fix the problem where some clients like Element Android, Fluffychat etc would display the wrong member count for a given room. * Improve SQLite query precision * Check existence of state key for membership events
2022-03-29Remove eduserver (#2306)S7evinK
* Move receipt sending to own JetStream producer * Move SendToDevice to producer * Remove most parts of the EDU server * Fix SendToDevice & copyrights * Move structs, cleanup EDU Server traces * Use HeadersOnly subscription * Missing file * Fix linter issues * Move consumers to own files * Rename durable consumer; Consumer cleanup * Docs/config cleanup
2022-03-25Use most recent event in response to get latest stream position in ↵Neil Alexander
incremental sync (#2302) * Use latest event position in response for advancing the stream position in an incremental sync * Create some calm * Use To in worst case * Don't waste CPU cycles on an empty response after all * Bug fixes * Fix another bug
2022-03-16Truncate `recentStreamEvents` before working out which event IDs to exclude ↵Neil Alexander
from `stateEvents` (#2281)
2022-03-03Implement Push Notifications (#1842)Dan
* Add Pushserver component with Pushers API Co-authored-by: Tommie Gannert <tommie@gannert.se> Co-authored-by: Dan Peleg <dan@globekeeper.com> * Wire Pushserver component Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> * Add PushGatewayClient. The full event format is required for Sytest. * Add a pushrules module. * Change user API account creation to use the new pushrules module's defaults. Introduces "scope" as required by client API, and some small field tweaks to make some 61push Sytests pass. * Add push rules query/put API in Pushserver. This manipulates account data over User API, and fires sync messages for changes. Those sync messages should, according to an existing TODO in clientapi, be moved to userapi. Forks clientapi/producers/syncapi.go to pushserver/ for later extension. * Add clientapi routes for push rules to Pushserver. A cleanup would be to move more of the name-splitting logic into pushrules.go, to depollute routing.go. * Output rooms.join.unread_notifications in /sync. This is the read-side. Pushserver will be the write-side. * Implement pushserver/storage for notifications. * Use PushGatewayClient and the pushrules module in Pushserver's room consumer. * Use one goroutine per user to avoid locking up the entire server for one bad push gateway. * Split pushing by format. * Send one device per push. Sytest does not support coalescing multiple devices into one push. Matches Synapse. Either we change Sytest, or remove the group-by-url-and-format logic. * Write OutputNotificationData from push server. Sync API is already the consumer. * Implement read receipt consumers in Pushserver. Supports m.read and m.fully_read receipts. * Add clientapi route for /unstable/notifications. * Rename to UpsertPusher for clarity and handle pusher update * Fix linter errors * Ignore body.Close() error check * Fix push server internal http wiring * Add 40 newly passing 61push tests to whitelist * Add next 12 newly passing 61push tests to whitelist * Send notification data before notifying users in EDU server consumer * NATS JetStream * Goodbye sarama * Fix `NewStreamTokenFromString` * Consume on the correct topic for the roomserver * Don't panic, NAK instead * Move push notifications into the User API * Don't set null values since that apparently causes Element upsetti * Also set omitempty on conditions * Fix bug so that we don't override the push rules unnecessarily * Tweak defaults * Update defaults * More tweaks * Move `/notifications` onto `r0`/`v3` mux * User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly Co-authored-by: Piotr Kozimor <p1996k@gmail.com> Co-authored-by: Tommie Gannert <tommie@gannert.se> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-01Use correct stream provider in `Latest` for `ReceiptPosition`Neil Alexander
2022-03-01Hopefully fix read receipts (#2241)Neil Alexander
2022-01-20BREAKING: Remove Partitioned Stream Positions (#2096)kegsay
* go mod tidy * Break complement to check it fails CI * Remove partitioned stream positions This was used by the device list stream position. The device list position now corresponds to the `Offset`, and the partition is always 0, in prep for removing reliance on Kafka topics for device list changes. * Linting * Migrate old style tokens to new style because element-web doesn't soft-logoout on 4xx errors on /sync
2021-07-14bugfix: retire invites even when we cannot talk to the remote server to ↵kegsay
make/send_leave (#1918) * bugfix: retire invites even when we cannot talk to the remote server to make/send_leave Also modify the leave response in /sync to include a fake event as this is ultimately what clients (and sytest) will use to determine leave-ness. * hash the event ID * Base64 not hex
2021-03-03Increase gocyclo complexity to 25 (and remove all but 2 golint directives ↵Neil Alexander
related to it) (#1783)
2021-02-04Don't re-request state events that are already in the timeline (#1739)Neil Alexander
* Don't request state events if we already have the timeline events (Postgres only) * Rename variable * nocyclo * Add SQLite * Tweaks * Revert query change * Don't dedupe if asking for full state * Update query
2021-01-29Complete sync performance (#1741)Neil Alexander
* Parallelise PDU stream fetching for complete sync * Fixes * Fixes * Worker queue * Workers * Don't populate device list changes on complete sync * Don't fast-forward typing notifications either on complete sync * Revert "Don't fast-forward typing notifications either on complete sync" This reverts commit 01471f78431cdd840915111f71bd2b5176e584a8. * Comments
2021-01-19Basic sync filtering (#1721)Neil Alexander
* Add some filtering (postgres only for now) * Fix build error * Try to use request filter * Use default filter as a template when retrieving from the database * Remove unused strut * Update sytest-whitelist * Add filtering to SelectEarlyEvents * Fix Postgres selectEarlyEvents query * Attempt filtering on SQLite * Test limit, set field for limit/order in prepareWithFilters * Remove debug logging, add comments * Tweaks, debug logging * Separate SQLite stream IDs * Fix filtering in current state table * Fix lock issues * More tweaks * Current state requires room ID * Review comments
2021-01-13Simplify send-to-device messaging (#1702)Neil Alexander
* Simplify send-to-device messaging * Don't return error if there's no work to do * Remove SQLite migrations for now * Tweak Postgres migrations * Tweaks * Fixes * Cleanup separately * Fix SQLite migration
2021-01-13Sync fixes (#1709)Neil Alexander
* omitempty some fields in sync * Add a few more * Don't send push rules over and over again in incremental sync * Further tweaks
2021-01-08Sync refactor — Part 1 (#1688)Neil Alexander
* It's half-alive * Wakeups largely working * Other tweaks, typing works * Fix bugs, add receipt stream * Delete notifier, other tweaks * Dedupe a bit, add a template for the invite stream * Clean up, add templates for other streams * Don't leak channels * Bring forward some more PDU logic, clean up other places * Add some more wakeups * Use addRoomDeltaToResponse * Log tweaks, typing fixed? * Fix timed out syncs * Don't reset next batch position on timeout * Add account data stream/position * End of day * Fix complete sync for receipt, typing * Streams package * Clean up a bit * Complete sync send-to-device * Don't drop errors * More lightweight notifications * Fix typing positions * Don't advance position on remove again unless needed * Device list updates * Advance account data position * Use limit for incremental sync * Limit fixes, amongst other things * Remove some fmt.Println * Tweaks * Re-add notifier * Fix invite position * Fixes * Notify account data without advancing PDU position in notifier * Apply account data position * Get initial position for account data * Fix position update * Fix complete sync positions * Review comments @Kegsay * Room consumer parameters
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-15 18:47:28 +0000