aboutsummaryrefslogtreecommitdiff
path: root/federationapi
AgeCommit message (Collapse)Author
2022-09-26Remove `origin` field from PDUs (#2737)Neil Alexander
This nukes the `origin` field from PDUs as per matrix-org/matrix-spec#998, matrix-org/gomatrixserverlib#341.
2022-09-22Tweak `InsertMigration` to avoid logging (#2720)Till
We'd still produce logs in Postgres when trying to insert a migration we already ran. This should stop us from creating those log entries.
2022-09-20Update dependencies (#2729)Neil Alexander
This updates Dendrite dependencies.
2022-09-16Fix origin on device list update EDUsTill Faelligen
2022-09-13Send-to-device consumer/producer tweaks (#2713)Till
Some tweaks for the send-to-device consumers/producers: - use `json.RawMessage` without marshalling it first - try further devices (if available) if we failed to `PublishMsg` in the producers - some logging changes (to better debug E2EE issues)
2022-09-08Fix issue with stale device lists (#2702)Till
We were only sending the last entry to the worker, so most likely missed updates.
2022-09-07Add HTTP status code to FederationClientError (#2699)Till
Also ensures we wait on more HTTP status codes.
2022-09-07Avoid unneeded JSON operations (#2698)Till
We were `json.Unmarshal`ing the EDU and `json.Marshal`ing right before sending the EDU to the stream. Those are now removed and the consumer does `json.Unmarshal` once.
2022-09-07Add a SigningKeyUpdate producer (#2697)Till
This adds a new stream for signing key updates, this should ensure we don't lose any updates over federation.
2022-09-07Handle errors differently in the `DeviceListUpdater` (#2695)Till
`If a device list update goes missing, the server resyncs on the next one` was failing because a previous test would receive a `waitTime` of 1h, resulting in the test timing out. This now tries to handle the returned errors differently, e.g. by using the default `waitTime` of 2s. Also doesn't try further users in the list, if one of the errors would cause a longer `waitTime`.
2022-09-01Configuration tweaks (#2567)Neil Alexander
This makes the following changes: * The various `Defaults` functions are now responsible for setting sane defaults if `generate` is specified, rather than hiding them in `generate-config` * Some configuration options have been marked as `omitempty` so that they don't appear in generated configs unnecessarily (monolith-specific vs. polylith-specific options) * A new option `-polylith` has been added to `generate-config` to create a config that makes sense for polylith deployments (i.e. including the internal/external API listeners and per-component database sections) * A new option `-normalise` has been added to `generate-config` to take an existing file and add any missing options and/or defaults
2022-08-31Allow batching in `JetStreamConsumer` (#2686)Neil Alexander
This allows us to receive more than one message from NATS at a time if we want.
2022-08-25Fix 500s on `/state`, `/state_ids` when state not known (#2672)Neil Alexander
This was due to bad error bubbling.
2022-08-23Only set backOffStarted to false if until is not zero (#2669)Till
2022-08-19De-race `TestExpireEDUs` (#2654)Winter
In some conditions (fast CPUs), this test would race the clock for EDU expiration when all we want to make sure of is that the expired EDUs are properly deleted. Given this, we set the expiry time to 0 so the specified EDUs are always deleted when DeleteExpiredEDUs is called. Fixes #2650. Signed-off-by: Winter <winter@winter.cafe>
2022-08-18Actually store EDUs once we retrieved from the database (#2651)Till
We now actually cache the EDUs once we got them from the database and ensures we only evict them if we successfully deleted them.
2022-08-11Generic-based internal HTTP API (#2626)Neil Alexander
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API) * Add `PerformInvite` * More tweaks * Fix metric name * Fix LookupStateIDs * Lots of changes to clients * Some serverside stuff * Some error handling * Use paths as metric names * Revert "Use paths as metric names" This reverts commit a9323a6a343f5ce6461a2e5bd570fe06465f1b15. * Namespace metric names * Remove duplicate entry * Remove another duplicate entry * Tweak error handling * Some more tweaks * Update error behaviour * Some more error tweaking * Fix API path for `PerformDeleteKeys` * Fix another path * Tweak federation client proxying * Fix another path * Don't return typed nils * Some more tweaks, not that it makes any difference * Tweak federation client proxying * Maybe fix the key backup test
2022-08-09Add housekeeping function to delete old/expired EDUs (#2399)Till
* Add housekeeping function to delete old/expired EDUs * Add migrations * Evict EDUs from cache * Fix queries * Fix upgrade * Use map[string]time.Duration to specify different expiry times * Fix copy & paste mistake * Set expires_at to tomorrow * Don't allow NULL * Add comment * Add tests * Use new testrig package * Fix migrations * Never expire m.direct_to_device Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> Co-authored-by: kegsay <kegan@matrix.org>
2022-08-05Do not use `ioutil` as it is deprecated (#2625)Neil Alexander
2022-08-05Fix linter issues (#2624)Till
* Try that again * All hail the mighty linter? * And once again * goimport all the things
2022-08-05Add race testing to tests, and fix a few small race conditions in the tests ↵Brian Meek
(#2587) * Add race testing to tests, and fix a few small race conditions in the tests * Enable run-sytest on MacOS * Remove deadlock detecting mutex, per code review feedback * Remove autoformatting related changes and a closure that is not needed * Adjust to importing nats client as 'natsclient' Signed-off-by: Brian Meek <brian@hntlabs.com> * Clarify the use of gooseMutex to proect goose internal state Signed-off-by: Brian Meek <brian@hntlabs.com> * Remove no longer needed mutex for guarding goose Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-08-05Only create a new destinationQueue if we don't have one (#2620)Till
2022-08-02Stronger checks for `/send_join` (#2604)Neil Alexander
2022-07-25Update database migrations, remove goose (#2264)Till
* Add new db migration * Update migrations Remove goose * Add possibility to test direct upgrades * Try to fix WASM test * Add checks for specific migrations * Remove AddMigration Use WithTransaction Add Dendrite version to table * Fix linter issues * Update tests * Update comments, outdent if * Namespace migrations * Add direct upgrade tests, skipping over one version * Split migrations * Update go version in CI * Fix copy&paste mistake * Use contexts in migrations Co-authored-by: kegsay <kegan@matrix.org> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-22Membership updater refactoring (#2541)Neil Alexander
* Membership updater refactoring * Pass in membership state * Use membership check rather than referring to state directly * Delete irrelevant membership states * We don't need the leave event after all * Tweaks * Put a log entry in that I might stand a chance of finding * Be less panicky * Tweak invite handling * Don't freak if we can't find the event NID * Use event NID from `types.Event` * Clean up * Better invite handling * Placate the almighty linter * Blacklist a Sytest which is otherwise fine under Complement for reasons I don't understand * Fix the sytest after all (thanks @S7evinK for the spot)
2022-07-12Minor SendToDevice fix (#2565)Till
* Avoid unnecessary marshalling if sending to the local server * Fix ordering of ToDevice messages * Revive SendToDevice test
2022-07-11Ristretto cache (#2563)Neil Alexander
* Try Ristretto cache * Tweak * It's beautiful * Update GMSL * More strict keyable interface * Fix that some more * Make less panicky * Don't enforce mutability checks for now * Determine mutability using deep equality * Tweaks * Namespace keys * Make federation caches mutable * Update cost estimation, add metric * Update GMSL * Estimate cost for metrics better * Reduce counters a bit * Try caching events * Some guards * Try again * Try this * Use separate caches for hopefully better hash distribution * Fix bug with admitting events into cache * Try to fix bugs * Check nil * Try that again * Preserve order jeezo this is messy * thanks VS Code for doing exactly the wrong thing * Try this again * Be more specific * aaaaargh * One more time * That might be better * Stronger sorting * Cache expiries, async publishing of EDUs * Put it back * Use a shared cache again * Cost estimation fixes * Update ristretto * Reduce counters a bit * Clean up a bit * Update GMSL * 1GB * Configurable cache sizees * Tweaks * Add `config.DataUnit` for specifying friendly cache sizes * Various tweaks * Update GMSL * Add back some lazy loading caching * Include key in cost * Include key in cost * Tweak max age handling, config key name * Only register prometheus metrics if requested * Review comments @S7evinK * Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere) * Review comments * Update sample configs * Update GHA Workflow * Update Complement images to Go 1.18 * Remove the cache test from the federation API as we no longer guarantee immediate cache admission * Don't check the caches in the renewal test * Possibly fix the upgrade tests * Update to matrix-org/gomatrixserverlib#322 * Update documentation to refer to Go 1.18
2022-07-05Hopefully fix read receipts timestamps (#2557)Neil Alexander
This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.
2022-06-30Silence presence logs (#2547)Till
2022-06-15Add `InputDeviceListUpdate` to the keyserver, remove old input API (#2536)Neil Alexander
* Add `InputDeviceListUpdate` to the keyserver, remove old input API * Fix copyright * Log more information when a device list update fails
2022-06-15Fix rare panic when returning user devices over federation (#2534)Till
2022-05-30Add missing indexes to federation API destination queues (#2502)Neil Alexander
* Add missing indexes to federation API destination queues * Fix duplicate name
2022-05-26Add support for MSC3787 and `org.matrix.msc3787` room version (update to ↵Neil Alexander
matrix-org/gomatrixserverlib#310)
2022-05-25Fix using the remote provided event from `/send_join` (#2490)Neil Alexander
2022-05-25Restricted join support on `/make_join`, `/send_join` (#2478)Neil Alexander
* Add `QueryRestrictedJoinAllowed` * Add `Resident` flag to `QueryRestrictedJoinAllowedResponse` * Check restricted joins on federation API * Return `Restricted` to determine if the room was restricted or not * Populate `AuthorisedVia` properly * Sign the event on `/send_join`, return it in the `/send_join` response in the `"event"` key * Kick back joins with invalid authorising user IDs, use event from `"event"` key if returned in `RespSendJoin` * Use invite helper in `QueryRestrictedJoinAllowed` * Only use users with the power to invite, change error bubbling a bit * Placate the almighty linter One day I will nuke `gocyclo` from orbit and everything in the world will be much better for it. * Review comments
2022-05-23Preserve content from `/make_join` as well as supplied content in the ↵Neil Alexander
request (#2481)
2022-05-23Allow defers to run in `TestMain` in federation API testsNeil Alexander
2022-05-17Remove debug loggingKegan Dougal
2022-05-17bugfix: E2EE device keys could sometimes not be sent to remote servers (#2466)kegsay
* Fix flakey sytest 'Local device key changes get to remote servers' * Debug logs * Remove internal/test and use /test only Remove a lot of ancient code too. * Use FederationRoomserverAPI in more places * Use more interfaces in federationapi; begin adding regression test * Linting * Add regression test * Unbreak tests * ALL THE LOGS * Fix a race condition which could cause events to not be sent to servers If a new room event which rewrites state arrives, we remove all joined hosts then re-calculate them. This wasn't done in a transaction so for a brief period we would have no joined hosts. During this interim, key change events which arrive would not be sent to destination servers. This would sporadically fail on sytest. * Unbreak new tests * Linting
2022-05-09Begin adding syncapi component tests (#2442)kegsay
* Add very basic syncapi tests * Add a way to inject jetstream messages * implement add_state_ids * bugfixes * Unbreak tests * Remove now un-needed API call * Linting
2022-05-09Federation consumer `adds_state_event_ids` tweak (#2441)Neil Alexander
* Don't ask roomserver for events we already have in federation API * Check number of events returned is as expected * Preallocate array * Improve shape a bit
2022-05-09One NATS instance per `BaseDendrite` (#2438)Neil Alexander
* One NATS instance per `BaseDendrite` * Fix roomserver
2022-05-06Clean up interface definitions (#2427)kegsay
* tidy up interfaces * remove unused GetCreatorIDForAlias * Add RoomserverUserAPI interface * Define more interfaces * Use AppServiceInternalAPI for consistent naming * clean up federationapi constructor a bit * Fix monolith in -http mode
2022-05-05Define component interfaces based on consumers (2/2) (#2425)kegsay
* convert remaining interfaces * Tidy up the userapi interfaces
2022-05-05Define component interfaces based on consumers (1/2) (#2423)kegsay
* Specify interfaces used by appservice, do half of clientapi * convert more deps of clientapi to finer-grained interfaces * Convert mediaapi and rest of clientapi * Somehow this got missed
2022-05-03Tidy up `AddPublicRoutes` (#2412)Neil Alexander
* Simplify federation API `AddPublicRoutes` * Simplify client API `AddPublicRoutes` * Simplify media API `AddPublicRoutes` * Simplify sync API `AddPublicRoutes` * Simplify `AddAllPublicRoutes`
2022-05-03Global database connection pool (for monolith mode) (#2411)Neil Alexander
* Allow monolith components to share a single database pool * Don't yell about missing connection strings * Rename field * Setup tweaks * Fix panic * Improve configuration checks * Update config * Fix lint errors * Update comments
2022-04-29Device list display name fixes (#2405)Neil Alexander
* Get device names from `unsigned` in `/user/devices` * Fix display name updates * Fix bug * Fix another bug
2022-04-28Don't answer expensive federation requests for rooms we no longer belong to ↵Neil Alexander
(#2398) This includes `/state`, `/state_ids`, `/get_missing_events` and `/backfill`. This should fix #2396.
2022-04-27Fix graceful shutdownNeil Alexander
generated by cgit v1.2.3 (git 2.26.2) at 2024-11-25 10:40:13 +0000