aboutsummaryrefslogtreecommitdiff
path: root/federationsender
AgeCommit message (Collapse)Author
2020-09-04Remove QueryRoomsForUser from current state server (#1398)Kegsay
2020-09-04Remove ServerACLs from the current state server (#1390)Kegsay
* Remove ServerACLs from the current state server Functionality moved to roomserver * Nothing to see here, move along
2020-09-03Defer keyserver and federationsender wakeups to give HTTP listeners time to ↵Neil Alexander
start (#1389)
2020-09-02Use federation sender for backfill/getting missing events (#1379)Neil Alexander
* Use federation sender for backfill and getting missing events * Fix internal URL paths * Update go.mod/go.sum for matrix-org/gomatrixserverlib#218 * Add missing server implementations in HTTP interface
2020-09-01Report which component failed to consume (#1375)Neil Alexander
2020-08-27Try to protect GetNextTransactionPDUs (#1350)Neil Alexander
2020-08-27Fix lock errors in federation sender (#1347)Neil Alexander
* Fix lock errors in federation sender * Additional fix to writers
2020-08-21Component-wide TransactionWriters (#1290)Neil Alexander
* Offset updates take place using TransactionWriter * Refactor TransactionWriter in current state server * Refactor TransactionWriter in federation sender * Refactor TransactionWriter in key server * Refactor TransactionWriter in media API * Refactor TransactionWriter in server key API * Refactor TransactionWriter in sync API * Refactor TransactionWriter in user API * Fix deadlocking Sync API tests * Un-deadlock device database * Fix appservice API * Rename TransactionWriters to Writers * Move writers up a layer in sync API * Document sqlutil.Writer interface * Add note to Writer documentation
2020-08-20Add FederationClient interface to federationsender (#1284)Kegsay
* Add FederationClient interface to federationsender - Use a shim struct in HTTP mode to keep the same API as `FederationClient`. - Use `federationsender` instead of `FederationClient` in `keyserver`. * Pointers not values * Review comments * Fix unit tests * Rejig backoff * Unbreak test * Remove debug logs * Review comments and linting
2020-08-20Change backoff behaviour so that Failure returns planned end time (#1288)Neil Alexander
2020-08-19Transaction writer changes, move roomserver writers (#1285)Neil Alexander
* Updated TransactionWriters, moved locks in roomserver, various other tweaks * Fix redaction deadlocks * Fix lint issue * Rename SQLiteTransactionWriter to ExclusiveTransactionWriter * Fix us not sending transactions through in latest events updater
2020-08-17Synchronous invites (#1273)Neil Alexander
* Refactor invites to be synchronous * Fix synchronous invites * Fix client API return type for send invite error * Linter * Restore PerformError on rsAPI.PerformInvite * Update sytest-whitelist * Don't override PerformError with normal errors * Fix error passing * Un-whitelist a couple of tests * Update sytest-whitelist * Try to handle multiple invite rejections better * nolint * Update gomatrixserverlib * Fix /v1/invite test * Remove replace from go.mod
2020-08-13Don't send to ACL'd servers (#1267)Neil Alexander
* Don't send to ACL'd servers * Use gjson to look for room_id in EDU
2020-08-13API setup refactoring (#1266)Neil Alexander
* Start HTTP endpoint refactoring * Update SetupAndServeHTTP * Fix builds * Don't set up external listener if no address configured * TLS HTTP setup * Break apart client/federation/key/media muxes * Tweaks * Fix P2P demos * Fix media API routing * Review comments @Kegsay * Update sample config * Fix gobind build * Fix External -> Public in federation API test
2020-08-10Prefix-defined Kafka topics (#1254)Neil Alexander
* Prefix-defined Kafka topics * Fix current state server test
2020-08-10Configuration format v1 (#1230)Neil Alexander
* Initial pass at refactoring config (not finished) * Don't forget current state and EDU servers * More shifting around * Update server key API tests * Fix roomserver test * Fix more tests * Further tweaks * Fix current state server test (sort of) * Maybe fix appservices * Fix client API test * Include database connection string in database options * Fix sync API build * Update config test * Fix unit tests * Fix federation sender build * Fix gobind build * Set Listen address for all services in HTTP monolith mode * Validate config, reinstate appservice derived in directory, tweaks * Tweak federation API test * Set MaxOpenConnections/MaxIdleConnections to previous values * Update generate-config
2020-08-07Backoff fixes (#1250)Neil Alexander
* Backoff fixes * Update comments * Fix destination queue * Log why we're blacklisting * Fix logic fail * Logging level * Fix bug * Maybe fix that bug after all * Fix debug output * Fix tests
2020-08-07Tweak log levels of some federation logging (#1248)Neil Alexander
* Tweak log levels of some federation logging * Update go.mod/go.sum for matrix-org/util#22 and matrix-org/gomatrixserverlib#215
2020-08-06Yggdrasil demo updates (#1241)Neil Alexander
* PerformServersAlive in PerformBroadcastEDU * Don't double-pointer * More reliable QUIC session handling * Direct peer lookup, other tweaks * Tweaks * Try to wake up queues on incoming QUIC session * Set session callbak on gobind build * Fix incoming session storage * Stateless reset, other tweaks * Reset sessions when coordinates change * Disable HTTP connection reuse, tweak timeouts
2020-08-05SelectJoinedHostsForRooms should use QueryVariadic on SQLite (#1238)Neil Alexander
* SelectJoinedHostsForRooms should use QueryVariadic on SQLite * Fix strings.Replace * Fix statement
2020-08-04Unbreak postgresKegan Dougal
2020-08-04Send device list updates to servers (outbound only) (#1237)Kegsay
* Add QueryDeviceMessages to serve up device keys and stream IDs * Consume key change events in fedsender Don't yet send them to destinations as we haven't worked them out yet * Send device list updates to all required servers * Glue it all together
2020-07-23Add UNIQUE constraint to blacklist table (#1216)Neil Alexander
2020-07-22Persistent federation sender blacklist (#1214)Neil Alexander
* Initial persistence of blacklists * Move statistics folder * Make MaxFederationRetries configurable * Set lower failure thresholds for Yggdrasil demos * Still write events into database for blacklisted hosts (they can be tidied up later) * Review comments
2020-07-20Remove debug linesNeil Alexander
2020-07-20Deduplicate FS database, EDU persistence table (#1207)Neil Alexander
* Deduplicate FS database, add some EDU persistence groundwork * Extend TransactionWriter to use optional existing transaction, use that for FS SQLite database writes * Fix build due to bad keyserver import * Working EDU persistence * gocyclo, unsurprisingly * Remove unused * Update copyright notices
2020-07-16Yggdrasil demo updates ("Bare QUIC")Neil Alexander
Squashed commit of the following: commit 86c2388e13ffdbabdd50cea205652dccc40e1860 Merge: b0a3ee6c f5e7e751 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Thu Jul 16 13:47:10 2020 +0100 Merge branch 'master' into neilalexander/yggbarequic commit b0a3ee6c5c063962384bb91c59ec753ddc8cfe5f Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Thu Jul 16 13:42:22 2020 +0100 Add support for broadcasting wake-up EDUs to known hosts commit 8a5c2020b3a4b705b5d5686a9e71990a49e6d471 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Thu Jul 16 13:42:10 2020 +0100 Bare QUIC demo working commit d3939b3d6568cf4262c0391486a5203873b68bfc Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Wed Jul 15 11:42:43 2020 +0100 Support bare Yggdrasil sessions with encrypted QUIC
2020-07-14Send-to-device messages over federation (#1198)Neil Alexander
* Initial work to send send-to-device messages over federation * Wire up send-to-device consumer, message formatting * Generate random message ID * Review comments, update sytest whitelist
2020-07-10Yggdrasil demo updatesNeil Alexander
Squashed commit of the following: commit 6c2c48f862c1b6f8e741c57804282eceffe02487 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Fri Jul 10 16:28:09 2020 +0100 Add README.md commit 5eeefdadf8e3881dd7a32559a92be49bd7ddaf47 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Fri Jul 10 10:18:50 2020 +0100 Fix wedge in federation sender commit e2ebffbfba25cf82378393940a613ec32bfb909f Merge: 0883ef88 abf26c12 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Fri Jul 10 09:51:23 2020 +0100 Merge branch 'master' into neilalexander/yggdrasil commit 0883ef8870e340f2ae9a0c37ed939dc2ab9911f6 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Fri Jul 10 09:51:06 2020 +0100 Adjust timeouts commit ba2d53199910f13b60cc892debe96a962e8c9acb Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Thu Jul 9 16:34:40 2020 +0100 Try to wake up from peers/sessions properly commit 73f42eb494741ba5b0e0cef43654708e3c8eb399 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Thu Jul 9 15:43:38 2020 +0100 Use TransactionWriter to reduce database lock issues on SQLite commit 08bfe63241a18c58c539c91b9f52edccda63a611 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Thu Jul 9 12:38:02 2020 +0100 Un-wedge federation Squashed commit of the following: commit aee933f8785e7a7998105f6090f514d18051a1bd Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Thu Jul 9 12:22:41 2020 +0100 Un-goroutine the goroutines commit 478374e5d18a3056cac6682ef9095d41352d1295 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Thu Jul 9 12:09:31 2020 +0100 Reduce federation sender wedges commit 40cc62c54d9e3a863868214c48b7c18e522a4772 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Thu Jul 9 10:02:52 2020 +0100 Handle switching in/out background more reliably
2020-07-09Use TransactionWriter to reduce database lock issues on SQLite (#1192)Neil Alexander
2020-07-09Reduce federation sender wedges (#1191)Neil Alexander
* Reduce federation sender wedges * Un-goroutine the goroutines
2020-07-08Handle case where pendingPDUs might get out of sync for some reasonNeil Alexander
2020-07-08Squashed commit of the following:Neil Alexander
commit b4cb47aa1329d2ada10ae6426fd9d2a69f47536a Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Wed Jul 8 14:13:27 2020 +0100 Restrict transaction send context time commit 7c28205cdb5d842071d46b1ec599d09cca708e57 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Wed Jul 8 14:00:06 2020 +0100 Add to gobind build commit d9e2c72e0576a2eb0ce6ac48eed6cc9d4761a0ea Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Wed Jul 8 13:43:21 2020 +0100 Wake up destination queues for new sessions/links commit 21766c6c52bd00511d28981457e9034358c32a8d Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Wed Jul 8 13:17:18 2020 +0100 Tweak QUIC parameters
2020-07-07Fix buildNeil Alexander
2020-07-07Fix error handling in federationsenderNeil Alexander
2020-07-07 bugfix: continue sending PDUs if ones are added whilst sending another PDU ↵Kegsay
(#1187) * Add a bit more logging to the fedsender * bugfix: continue sending PDUs if ones are added whilst sending another PDU Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't fire because `pendingPDUs` is already > 0. This should fix a flakey sytest. * Break if no txn is sent * Tweak federation sender wake-ups * Update comments * Remove break or that'll kill the parent loop Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-07-03Wake up destination queues more aggressively (#1183)Neil Alexander
* Wake up destination queues more aggressively * We don't really need Ch here do we
2020-07-03Hydrate destination queues at startup (#1179)Neil Alexander
* Hydrate destination queues at startup * Review comments
2020-07-02Yggdrasil+QUIC demo, federation sender tweaks (#1177)Neil Alexander
* Initial QUIC work * Update Yggdrasil demo * Make sure that the federation sender knows how many pending events are in the database when the worker starts * QUIC tunables * pprof * Don't spin * Set build info for Yggdrasil
2020-07-01Persistent federation sender queues (PDUs) (#1173)Neil Alexander
* Initial work on persistent queues * Update index for event ID and server name * Put things into database (postgres for now) * Duplicate postgres code into sqlite for now just to stop build errors, will fix SQLite soon * Fix table name * Fix index * Fix table name * Use RETURNING because LastInsertID is not supported by postgres * Use functions * Marshal headered event * Don't error on now rows * Don't block if there are PDUs waiting * Try to tidy up JSON * Debug logging * Fix query, use transactions in postgres * Clean up * Rehydrate more opportunistically * Fix SQLite * remove unused types * Review comments * Shuffle things around a bit * Clean up transaction properly * Don't send empty transactions * Reduce unnecessary retries * Count PDUs to make more resilient * Don't stop when there is work to be done * Try to limit wakeups * well this is tedious * Fix race in incomplete transactions * Thread safety on transaction ID/count
2020-06-25Return remote errors from FS.PerformJoin (#1164)Kegsay
* Return remote errors from FS.PerformJoin Follows the same pattern as PerformJoin on roomserver (no error return). Also return the right format for incompatible room version errors. Makes a bunch of tests pass! * Handle network errors better when returning remote HTTP errors * Linting * Fix tests * Update whitelist, pass network errors through in API=1 mode
2020-06-12Minor perf/debugging improvements (#1121)Kegsay
* Minor perf/debugging improvements - publicroomsapi: Don't call QueryEventsByID with no event IDs - appservice: Consume only if there are 1 or more ASes - roomserver: don't keep a copy of the request "for debugging" - we trace now * fedsender: return early if we have no destinations * Unbreak tests
2020-06-12Rehuffle where things are in the internal package (#1122)Kegsay
renamed: internal/eventcontent.go -> internal/eventutil/eventcontent.go renamed: internal/events.go -> internal/eventutil/events.go renamed: internal/types.go -> internal/eventutil/types.go renamed: internal/http/http.go -> internal/httputil/http.go renamed: internal/httpapi.go -> internal/httputil/httpapi.go renamed: internal/httpapi_test.go -> internal/httputil/httpapi_test.go renamed: internal/httpapis/paths.go -> internal/httputil/paths.go renamed: internal/routing.go -> internal/httputil/routing.go renamed: internal/basecomponent/base.go -> internal/setup/base.go renamed: internal/basecomponent/flags.go -> internal/setup/flags.go renamed: internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go renamed: internal/postgres.go -> internal/sqlutil/postgres.go renamed: internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go renamed: internal/sql.go -> internal/sqlutil/sql.go
2020-06-12Add trace logging to RoomserverInternalAPI (#1120)Kegsay
This is a wrapper around whatever impl we have which then logs the function name/request/response/error. Also tweak when we log on kafka streams: only log on the producer side not the consumer side: we've never had issues with comms and having 1 message rather than N would be nice.
2020-06-11Roomserver API changes (#1118)Kegsay
* s/QueryBackfill/PerformBackfill/g * OutputEvent now includes AddStateEvents which contain the full event of extra state events * Only include adds not the current event * Get adding state right
2020-06-11Remove unused internal APIs (#1117)Kegsay
2020-06-10Remove federationsender producer, which in fact was not a producer (#1115)Kegsay
* Remove federationsender producer, which in fact was not a producer * Set the signing struct
2020-06-08Split out SetupFooComponent (#1106)Kegsay
* Split out adding HTTP routes from making internal APIs for clarity * Split out more components * Split out more things * Finish converting * internal mux for internal routes
2020-06-05Use AuthChainProvider to try and speed up federated joins (#1100)Neil Alexander
* Use MissingAuthEventHandler on performjoin to try and speed up cases where we have missing events * Update gomatrixserverlib * Use supplied room version * Use AuthChainProvider * Tweaks * Update gomatrixserverlib * Signature checks
2020-06-04Refactor how federationsender gets created (#1095)Kegsay
* Refactor how federationsender gets created * s/httpint/inthttp/ for alphabetical niceness with internal package
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-19 09:31:13 +0000