aboutsummaryrefslogtreecommitdiff
path: root/syncapi
AgeCommit message (Collapse)Author
2022-07-22Membership updater refactoring (#2541)Neil Alexander
* Membership updater refactoring * Pass in membership state * Use membership check rather than referring to state directly * Delete irrelevant membership states * We don't need the leave event after all * Tweaks * Put a log entry in that I might stand a chance of finding * Be less panicky * Tweak invite handling * Don't freak if we can't find the event NID * Use event NID from `types.Event` * Clean up * Better invite handling * Placate the almighty linter * Blacklist a Sytest which is otherwise fine under Complement for reasons I don't understand * Fix the sytest after all (thanks @S7evinK for the spot)
2022-07-18Set historyVisibility in rowsToStreamEventsTill Faelligen
2022-07-18History visibility database changes (#2533)Till
* Add new history_visibility column * Update SQL queries to include history_visibility * Store the history visibilty calculated by the roomserver * Update GMSL * Update migrations * Fix migration * Update GMSL * Fix `go.sum` * Update GMSL to use sql.Scanner & sql.Valuer * Re-order migration/table creation * Update gomatrixserverlib * Add history_visibility column to current_room_state * Fix migrations * Return error instead of Fatal log Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-15Use sync API database in `filterSharedUsers` (#2572)Neil Alexander
* Add function to the sync API storage package for filtering shared users * Use the database instead of asking the RS API * Fix unit tests * Fix map handling in `filterSharedUsers`
2022-07-12Minor SendToDevice fix (#2565)Till
* Avoid unnecessary marshalling if sending to the local server * Fix ordering of ToDevice messages * Revive SendToDevice test
2022-07-11Ristretto cache (#2563)Neil Alexander
* Try Ristretto cache * Tweak * It's beautiful * Update GMSL * More strict keyable interface * Fix that some more * Make less panicky * Don't enforce mutability checks for now * Determine mutability using deep equality * Tweaks * Namespace keys * Make federation caches mutable * Update cost estimation, add metric * Update GMSL * Estimate cost for metrics better * Reduce counters a bit * Try caching events * Some guards * Try again * Try this * Use separate caches for hopefully better hash distribution * Fix bug with admitting events into cache * Try to fix bugs * Check nil * Try that again * Preserve order jeezo this is messy * thanks VS Code for doing exactly the wrong thing * Try this again * Be more specific * aaaaargh * One more time * That might be better * Stronger sorting * Cache expiries, async publishing of EDUs * Put it back * Use a shared cache again * Cost estimation fixes * Update ristretto * Reduce counters a bit * Clean up a bit * Update GMSL * 1GB * Configurable cache sizees * Tweaks * Add `config.DataUnit` for specifying friendly cache sizes * Various tweaks * Update GMSL * Add back some lazy loading caching * Include key in cost * Include key in cost * Tweak max age handling, config key name * Only register prometheus metrics if requested * Review comments @S7evinK * Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere) * Review comments * Update sample configs * Update GHA Workflow * Update Complement images to Go 1.18 * Remove the cache test from the federation API as we no longer guarantee immediate cache admission * Don't check the caches in the renewal test * Possibly fix the upgrade tests * Update to matrix-org/gomatrixserverlib#322 * Update documentation to refer to Go 1.18
2022-07-07Fix issue `uint64 values with high bit are not supported` in presence (#2562)Till
* Fix issue #2528 * Use gomatrixserverlib.Timestamp * Use ParseUint instead of ParseInt
2022-07-05Hopefully fix read receipts timestamps (#2557)Neil Alexander
This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.
2022-07-01Try to fix backfilling (#2548)Till
* Try to fix backfilling * Return start/end to not confuse clients * Update GMSL * Update GMSL
2022-06-30Silence presence logs (#2547)Till
2022-06-29Don't return `end` if there are not more messages (#2542)Till
* Be more spec compliant * Move lazyLoadMembers to own method
2022-06-15Add `InputDeviceListUpdate` to the keyserver, remove old input API (#2536)Neil Alexander
* Add `InputDeviceListUpdate` to the keyserver, remove old input API * Fix copyright * Log more information when a device list update fails
2022-06-13Handle state before, send history visibility in output (#2532)Neil Alexander
* Check state before event * Tweaks * Refactor a bit, include in output events * Don't waste time if soft failed either * Tweak control flow, comments, use GMSL history visibility type
2022-05-19syncapi: don't return early for no-op incremental syncs (#2473)kegsay
* syncapi: don't return early for no-op incremental syncs Comments explain why, but basically it's an inefficient use of bandwidth and some sytests rely on /sync to block. * Honour timeouts * Actually return a response with timeout=0
2022-05-17bugfix: fix race condition when updating presence via /sync (#2470)kegsay
* bugfix: fix race condition when updating presence via /sync Previously when presence is updated via /sync, we would send the presence update asyncly via NATS. This created a race condition: - If the presence update is processed quickly, the /sync which triggered the presence update would see an online presence. - If the presence update was processed slowly, the /sync which triggered the presence update would see an offline presence. This is the root cause behind the flakey sytest: 'User sees their own presence in a sync'. The fix is to ensure we update the database/advance the stream position synchronously for local users. * Bugfix for test
2022-05-17bugfix: E2EE device keys could sometimes not be sent to remote servers (#2466)kegsay
* Fix flakey sytest 'Local device key changes get to remote servers' * Debug logs * Remove internal/test and use /test only Remove a lot of ancient code too. * Use FederationRoomserverAPI in more places * Use more interfaces in federationapi; begin adding regression test * Linting * Add regression test * Unbreak tests * ALL THE LOGS * Fix a race condition which could cause events to not be sent to servers If a new room event which rewrites state arrives, we remove all joined hosts then re-calculate them. This wasn't done in a transaction so for a brief period we would have no joined hosts. During this interim, key change events which arrive would not be sent to destination servers. This would sporadically fail on sytest. * Unbreak new tests * Linting
2022-05-13Only try to get OTKs if the context isn't done yetTill Faelligen
2022-05-12Wait 100ms for events to be processed by syncapiKegan Dougal
2022-05-11Fix OTK upload spam (#2448)Till
* Fix OTK spam * Update comment * Optimize selectKeysCountSQL to only return max 100 keys * Return CurrentPosition if the request timed out * Revert "Return CurrentPosition if the request timed out" This reverts commit 7dbdda964189f5542048c06ce5ffc6d4da1814e6. Co-authored-by: kegsay <kegan@matrix.org>
2022-05-11More syncapi tests (#2451)kegsay
* WIP tests for flakey create event * Uncomment all database test
2022-05-11Add RoomExists flag to QueryMembershipForUser (#2450)kegsay
Fixes https://github.com/matrix-org/complement/pull/369
2022-05-10Add indexes to `syncapi_output_room_events` table that satisfy the filters ↵Neil Alexander
(#2446)
2022-05-09Begin adding syncapi component tests (#2442)kegsay
* Add very basic syncapi tests * Add a way to inject jetstream messages * implement add_state_ids * bugfixes * Unbreak tests * Remove now un-needed API call * Linting
2022-05-09Don't store invites in sync API that aren't relevant to local users (#2439)Neil Alexander
2022-05-09Federation consumer `adds_state_event_ids` tweak (#2441)Neil Alexander
* Don't ask roomserver for events we already have in federation API * Check number of events returned is as expected * Preallocate array * Improve shape a bit
2022-05-09Don't try to re-fetch the event if it is listed in `adds_state_event_ids` ↵Neil Alexander
(#2437) * Don't try to re-fetch the event in the output message * Try that again * Add the initial event into the set
2022-05-09One NATS instance per `BaseDendrite` (#2438)Neil Alexander
* One NATS instance per `BaseDendrite` * Fix roomserver
2022-05-06Move LL cache (#2429)Till
2022-05-05syncapi: define specific interfaces for internal HTTP communications (#2416)kegsay
* syncapi: use finer-grained interfaces when making the syncapi * Use specific interfaces for syncapi-roomserver interactions * Define query access token api for shared http auth code
2022-05-04Add opt-in anonymous stats reporting (#2249)Till
* Initial phone home stats queries * Add userAgent to UpdateDeviceLastSeen Add new Table for tracking daily user vists * Add user_daily_visits table * Fix queries * userapi stats tables & queries * userapi interface and internal api * sycnapi stats queries * testing phone home stats * Add complete config to syncapi * add missing files * Fix queries * Send empty request * Add version & monolith stats * Add configuration for phone home stats * Move WASM to its own file, add config and comments * Add tracing methods * Add total rooms * Add more fields, actually send data somewhere * Move stats to the userapi * Move phone home stats to util package * Cleanup * Linter & parts of GH comments * More GH comments changes - Move comments to SQL statements - Shrink interface, add struct for stats - No fatal errors, use defaults * Be more explicit when querying * Fix wrong calculation & wrong query params Add tests * Add Windows stats * ADd build constraint * Use new testing structure Fix issues with getting values when using SQLite Fix wrong AddDate value Export UpdateUserDailyVisits * Fix query params * Fix test * Add comment about countR30UsersSQL and countR30UsersV2SQL; fix test * Update config * Also update example config file * Use OS level proxy, update logging Co-authored-by: kegsay <kegan@matrix.org>
2022-05-03Tidy up `AddPublicRoutes` (#2412)Neil Alexander
* Simplify federation API `AddPublicRoutes` * Simplify client API `AddPublicRoutes` * Simplify media API `AddPublicRoutes` * Simplify sync API `AddPublicRoutes` * Simplify `AddAllPublicRoutes`
2022-05-03Global database connection pool (for monolith mode) (#2411)Neil Alexander
* Allow monolith components to share a single database pool * Don't yell about missing connection strings * Rename field * Setup tweaks * Fix panic * Improve configuration checks * Update config * Fix lint errors * Update comments
2022-04-30Return "to", if we didn't return any presence events (#2407)Till
Return correct stream position, if we didn't return any presence events
2022-04-28Only load members of newly joined rooms (#2389)Till
* Only load members of newly joined rooms * Comment that the query is prepared at runtime Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-28Limit presence in `/sync` responses (#2394)Till
* Use filter and limit presence count * More limiting * More limiting * Fix unit test * Also limit presence by last_active_ts * Update query, use "from" as the initial lastPos * Get 1000 presence events, they are filtered later Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-27Fix #2390 (#2392)Till
Fix duplicate heroes in `/sync` response.
2022-04-27Try that againNeil Alexander
2022-04-27Correct account data position mappingNeil Alexander
2022-04-27Don't send account data or receipts for left/forgotten rooms (#2382)Neil Alexander
* Only include account data and receipts for rooms in a complete sync that we care about * Fix global account data
2022-04-27Don't create fictitious presence entries (#2381)Neil Alexander
* Don't create fictitious presence entries for users that don't have any * Update whitelist, since that test probably shouldn't be passing * Fix panics
2022-04-26Use a value that is Go 1.16-friendlyNeil Alexander
2022-04-26Send all account data on complete sync by defaultNeil Alexander
Squashed commit of the following: commit 0ec8de57261d573a5f88577aa9d7a1174d3999b9 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Apr 26 16:56:30 2022 +0100 Select filter onto provided target filter commit da40b6fffbf5737864b223f49900048f557941f9 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Apr 26 16:48:00 2022 +0100 Specify other field too commit ffc0b0801f63bb4d3061b6813e3ce5f3b4c8fbcb Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Apr 26 16:45:44 2022 +0100 Send as much account data as possible during complete sync
2022-04-26Fix account data positionNeil Alexander
2022-04-26Start account data ID from `from`Neil Alexander
2022-04-26Fix account_data not correctly send in a complete sync (#2379)Till
* Return the StreamPosition from the database and not the latest * Fix linter issue
2022-04-26Add heroes to the room summary (#2373)Till
* Implement room summary heroes * Add passing tests * Move MembershipCount to addRoomSummary * Add comments, close Statement
2022-04-25Use provided filter for account_data (#2372)Till
* Reuse IncrementalSync, use provided filter * Inform SyncAPI about newly created push_rules
2022-04-22Reuse the existing lazyload cache on /context and /messages (#2367)Till
2022-04-20Eliminate more SQL no row errors in sync API (#2363)Neil Alexander
* Handle `sql.ErrNoRows` in main `/sync` codepaths * Catch more
2022-04-20Lazy loading fixes (#2362)Neil Alexander
* Return some more usefully wrapped errors when doing sync * Remove unnecessary error check * Couple of guards around `sql.ErrNoRows` * Nolint
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-19 08:45:19 +0000