aboutsummaryrefslogtreecommitdiff
path: root/federationapi/queue
AgeCommit message (Collapse)Author
2024-10-18Update the license headers to mention the commercial licenseQuentin Gliech
2024-10-17Rename the go packageQuentin Gliech
github.com/matrix-org/dendrite to github.com/element-hq/dendrite
2024-10-17Update the copyright headers in all filesQuentin Gliech
2024-05-01Take advantage of changes in recent Go versions (#3361)0x1a8510f2
Given that #2714 wasn't merged but we are now at a minimum supported Go version of 1.20 (soon to be 1.21), I wanted to carry over some of the changes. Namely: - Fix the log typo - Simplify build constraints for unix - Use stdlib atomic package ### Pull Request Checklist <!-- Please read https://matrix-org.github.io/dendrite/development/contributing before submitting your pull request --> * [x] I have added Go unit tests or [Complement integration tests](https://github.com/matrix-org/complement) for this PR _or_ I have justified why this PR doesn't need tests * [x] Pull request includes a [sign off below using a legally identifiable name](https://matrix-org.github.io/dendrite/development/contributing#sign-off) _or_ I have already signed off privately Signed-off-by: `0x1a8510f2 <admin@0x1a8510f2.space>` --------- Co-authored-by: devonh <devon.dmytro@gmail.com>
2024-04-09Don't attempt to send transactions if Dendrite is shutting down (#3356)Till
This should avoid confusions with logs like: ``` time="2024-04-08T08:38:45.104235081Z" level=error msg="Failed to set \"scs.ems.host\" as assumed offline" func="github.com/matrix-org/dendrite/federationapi/statistics.(*ServerStatistics).Failure" file="github.com/matrix-org/dendrite/federationapi/statistics/statistics.go:204" error="sqlutil.WithTransaction.Begin: sql: database is closed" time="2024-04-08T08:38:45.104239201Z" level=error msg="Failed to set \"obermui.de\" as assumed offline" func="github.com/matrix-org/dendrite/federationapi/statistics.(*ServerStatistics).Failure" file="github.com/matrix-org/dendrite/federationapi/statistics/statistics.go:204" error="sqlutil.WithTransaction.Begin: sql: database is closed" ``` or ``` time="2024-04-08T08:38:45.105235411Z" level=error msg="Failed to get pending EDUs for \"retro76.net\"" func="github.com/matrix-org/dendrite/federationapi/queue.(*destinationQueue).getPendingFromDatabase" file="github.com/matrix-org/dendritefederationapi/queue/destinationqueue.go:258" error="sqlutil.WithTransaction.Begin: sql: database is closed" ``` [skip ci]
2024-02-28Make usage of relays optional, avoid DB roundtrips (#3337)Till
This should avoid 2 additional DB roundtrips if we don't want to use relays. So instead of possibly doing roughly 20k trips to the DB, we are now "only" doing ~6600. --------- Co-authored-by: devonh <devon.dmytro@gmail.com>
2023-11-22Update ACLs when received as outliers (#3008)Till
This should fix #3004 by making sure we also update our in-memory ACLs after joining a new room. Also makes use of more caching in `GetStateEvent` Bonus: Adds some tests, as I was about to use `GetBulkStateContent`, but turns out that `GetStateEvent` is basically doing the same, just that it only gets the `eventTypeNID`/`eventStateKeyNID` once and not for every call.
2023-09-15Update gmsl to use new validated RoomID on PDUs (#3200)devonh
GMSL returns a `spec.RoomID` when calling `PDU.RoomID()`
2023-05-02Use PDU not *Event in HeaderedEvent (#3073)kegsay
Requires https://github.com/matrix-org/gomatrixserverlib/pull/376 This has numerous upsides: - Less type casting to `*Event` is required. - Making Dendrite work with `PDU` interfaces means we can swap out Event impls more easily. - Tests which represent weird event shapes are easier to write. Part of a series of refactors on GMSL.
2023-04-27Remove gmsl.HeaderedEvent (#3068)kegsay
Replaced with types.HeaderedEvent _for now_. In reality we want to move them all to gmsl.Event and only use HeaderedEvent when we _need_ to bundle the version/event ID with the event (seriailsation boundaries, and even then only when we don't have the room version). Requires https://github.com/matrix-org/gomatrixserverlib/pull/373
2023-04-24Move fedclient interface over to gmsl (#3061)devonh
Companion PR: https://github.com/matrix-org/gomatrixserverlib/pull/366
2023-04-21Implement new RoomVersionImpl API (#3062)kegsay
As outlined in https://github.com/matrix-org/gomatrixserverlib/pull/368 The main change Dendrite side is that `RoomVersion` no longer has any methods on it. Instead, you need to bounce via `gmsl.GetRoomVersion`. It's very interesting to see where exactly Dendrite cares about this. For some places it's creating events (fine) but others are way more specific. Those areas will need to migrate to GMSL at some point.
2023-04-20refactor: funnel event creation through room versions (#3060)kegsay
In preparation of interfacing up the room version value.
2023-04-19refactor: update GMSL (#3058)kegsay
Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/364 Read this commit by commit to avoid going insane.
2023-04-06refactor: use latest GMSL which splits fed client from matrix room logic (#3051)kegsay
Part of a series of refactors on GMSL.
2023-03-22Remove `BaseDendrite` (#3023)Till
Removes `BaseDendrite` to, hopefully, make testing and composing of components easier in the future.
2023-03-17Preparations for removing `BaseDendrite` (#3016)Till
Preparations to actually remove/replace `BaseDendrite`. Quite a few changes: - SyncAPI accepts an `fulltext.Indexer` interface (fulltext is removed from `BaseDendrite`) - Caches are removed from `BaseDendrite` - Introduces a `Router` struct (likely to change) - also fixes #2903 - Introduces a `sqlutil.ConnectionManager`, which should remove `base.DatabaseConnection` later on - probably more
2023-01-28Relay integration to pinecone demos (#2955)devonh
This extends the dendrite monolith for pinecone to integrate the s&f features into the mobile apps. Also makes a few tweaks to federation queueing/statistics to make some edge cases more robust.
2023-01-23Initial Store & Forward Implementation (#2917)devonh
This adds store & forward relays into dendrite for p2p. A few things have changed: - new relay api serves new http endpoints for s&f federation - updated outbound federation queueing which will attempt to forward using s&f if appropriate - database entries to track s&f relays for other nodes
2022-12-22Add federation peeking table tests (#2920)Till
As the title says, adds tests for inbound/outbound peeking federation table tests. Also removes some unused code
2022-11-18Add p2p wakeup broadcast handling to pinecone demos (#2841)devonh
Adds wakeup broadcast handling to the pinecone demos. This will reset their blacklist status and interrupt any ongoing federation queue backoffs currently in progress for this peer. The end result is that any queued events will quickly be sent to the peer if they had disconnected while attempting to send events to them.
2022-11-15Federation fixes for virtual hostingNeil Alexander
2022-11-11Virtual hosting schema and logic changes (#2876)Neil Alexander
Note that virtual users cannot federate correctly yet.
2022-10-26Associate events in db before queueing them to send (#2833)devonh
Fixes a race condition between sending federation events and having them fully associated in the database.
2022-10-26Initial support for multiple server names (#2829)Neil Alexander
This PR is the first step towards virtual hosting by laying the groundwork for multiple server names being configured.
2022-10-21Optimize inserting pending PDUs/EDUs (#2821)Till
This optimizes the association of PDUs/EDUs to their destination by inserting all destinations in one transaction.
2022-10-20Fix flakey queue test (#2818)devonh
Ensure both events are added to the database, even if the destination is already blacklisted.
2022-10-20Fix race conditionTill Faelligen
2022-10-19Refactor Federation Destination Queues (#2807)devonh
This is a refactor of the federation destination queues. It fixes a few things, namely: - actually retry outgoing events with backoff behaviour - obtain enough events from the database to fill messages as much as possible - minimize the amount of running goroutines - use pure timers for backoff - don't restart queue unless necessary - close the background task when backing off - increase max edus in a transaction to match the spec - cleanup timers more aggresively to reduce memory usage - add jitter to backoff timers to reduce resource spikes - add a bunch of tests (with real and fake databases) to ensure everything is working
2022-10-13Revert "Federation backoff fixes and tests (#2792)"Neil Alexander
This reverts commit dcedd1b6bf1e890ff425bdf1fcd8a2e0850778b5.
2022-10-13Federation backoff fixes and tests (#2792)devonh
This fixes some edge cases where federation queue backoffs and blacklisting weren't behaving as expected. It also adds new tests for the federation queues to ensure their behaviour continues to work correctly.
2022-10-10Send E2EE related errors to sentry (#2784)Till
Only sends errors if we're not retrying them in NATS. Not sure if those should be scoped/tagged with something like "E2EE".
2022-09-20Update dependencies (#2729)Neil Alexander
This updates Dendrite dependencies.
2022-08-09Add housekeeping function to delete old/expired EDUs (#2399)Till
* Add housekeeping function to delete old/expired EDUs * Add migrations * Evict EDUs from cache * Fix queries * Fix upgrade * Use map[string]time.Duration to specify different expiry times * Fix copy & paste mistake * Set expires_at to tomorrow * Don't allow NULL * Add comment * Add tests * Use new testrig package * Fix migrations * Never expire m.direct_to_device Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> Co-authored-by: kegsay <kegan@matrix.org>
2022-08-05Only create a new destinationQueue if we don't have one (#2620)Till
2022-05-17bugfix: E2EE device keys could sometimes not be sent to remote servers (#2466)kegsay
* Fix flakey sytest 'Local device key changes get to remote servers' * Debug logs * Remove internal/test and use /test only Remove a lot of ancient code too. * Use FederationRoomserverAPI in more places * Use more interfaces in federationapi; begin adding regression test * Linting * Add regression test * Unbreak tests * ALL THE LOGS * Fix a race condition which could cause events to not be sent to servers If a new room event which rewrites state arrives, we remove all joined hosts then re-calculate them. This wasn't done in a transaction so for a brief period we would have no joined hosts. During this interim, key change events which arrive would not be sent to destination servers. This would sporadically fail on sytest. * Unbreak new tests * Linting
2022-04-27Fix graceful shutdownNeil Alexander
2022-04-25Only call key update process functions if there are updates, don't send ↵Neil Alexander
things to ourselves over federation
2022-04-25Store the EDU type in the database (#2370)Till
2022-04-04Slower federation warm-up (#2320)Neil Alexander
* Wake destination queues gradually, rather than all at once * Delay device list updates too * Maximum two minute warmup period
2022-02-11Remove FederationDisabled error type (#2174)S7evinK
2022-02-08Demote logging entry for backoffNeil Alexander
2021-11-24Merge `federationapi`, `federationsender`, `signingkeyserver` components (#2055)Neil Alexander
* Initial federation sender -> federation API refactoring * Move base into own package, avoids import cycle * Fix build errors * Fix tests * Add signing key server tables * Try to fold signing key server into federation API * Fix dendritejs builds * Update embedded interfaces * Fix panic, fix lint error * Update configs, docker * Rename some things * Reuse same keyring on the implementing side * Fix federation tests, `NewBaseDendrite` can accept freeform options * Fix build * Update create_db, configs * Name tables back * Don't rename federationsender consumer for now
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-18 08:30:28 +0000