aboutsummaryrefslogtreecommitdiff
path: root/federationsender
AgeCommit message (Collapse)Author
2020-06-01Add a way to force federationsender to retry sending transactions (#1077)Kegsay
* Add a way to force federationsender to retry sending transactions And use it in P2P mode when we pick up new nodes. * Linting * Use atomic bool to stop us blocking on the channel
2020-05-29fedsender: de-duplicate without sorting server names (#1073)Kegsay
2020-05-27Read batches from incoming channels (#1067)Neil Alexander
2020-05-26Miscellaneous fixes (#1060)Neil Alexander
* Add missing routing for PerformDirectoryLookupRequest * Tweak output * Fix some bugs in devices * Don't default to federated room joins in response to invite * Update sytest-whitelist * Update comments * Return correct room ID from PerformJoin * Fix appservice and EDU server API setup, update sytest-whitelist * Update sytest-whitelist
2020-05-22Fix API pathsNeil Alexander
2020-05-22dendritejs tweaks for persisting sqlite DBs (#1058)Kegsay
* Use uri.path so we don't have file: in the filename * New go-sqlite-js version
2020-05-22Separate muxes for public and internal APIs (#1056)Neil Alexander
* Separate muxes for public and internal APIs * Update client-api-proxy and federation-api-proxy so they don't add /api to the path * Tidy up * Consistent HTTP setup * Set up prefixes properly
2020-05-21Fix #897 and shuffle directory around (#1054)Kegsay
* Fix #897 and shuffle directory around * Update find-lint * goimports Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-18Add -api flag to monolith (#1044)Neil Alexander
* Add flag for enabling HTTP APIs in monolith mode * Flag -api * Only start HTTP APIs if needed
2020-05-15Don't prematurely stop trying to join using servers (#1041)Neil Alexander
* Don't prematurely stop trying to join using servers * Factor out performJoinUsingServer
2020-05-07Fix 'input to Unique() must be sorted' panicNeil Alexander
2020-05-07Improve federation sender performance, implement backoff and blacklisting, ↵Neil Alexander
fix up invites a bit (#1007) * Improve federation sender performance and behaviour, add backoff * Tweaks * Tweaks * Tweaks * Take copies of events before passing to destination queues * Don't accidentally drop queued messages * Don't take copies again * Tidy up a bit * Break out statistics (tracked component-wide), report success and failures from Perform actions * Fix comment, use atomic add * Improve logic a bit, don't block on wakeup, move idle check * Don't retry sucessful invites, don't dispatch sendEvent, sendInvite etc * Dedupe destinations, fix other bug hopefully * Dispatch sends again * Federation sender to ignore invites that are destined locally * Loopback invite events * Remodel a bit with channels * Linter * Only loopback invite event if we know the room * We should tell other resident servers about the invite if we know about the room * Correct invite signing * Fix invite loopback * Check HTTP response codes, push new invites to front of queue * Review comments
2020-05-04Roomserver perform leave (#1004)Neil Alexander
* First pass at PerformLeave * Fix SQLite bulkSelectEventStateKey * Update gomatrixserverlib * Fix bugs * Tidy a bit * Satisfy King Linter * Review comments * Review comments * Fix constants in SQLite event state keys table
2020-05-04Roomserver perform join (#1001)Neil Alexander
* Add PerformJoin template * Try roomserver perform join * Send correct server name to FS API * Pass through content, try to handle multiple server names * Fix local server checks * Don't refer to non-existent error * Add directory lookups of aliases * Remove unneeded parameters * Don't repeat join events into the roomserver * Unmarshal the content, that would help * Check if the user is already in the room in the fedeationapi too * Return incompatible room version error * Use Membership, don't try more servers than needed * Review comments, make FS API take list of servernames, dedupe them, break out of loop properly on success * Tweaks
2020-05-01Limit database connections (#980, #564) (#998)Neil Alexander
* Limit database connections (#564) - Add new options to the config file database: max_open_conns: 100 max_idle_conns: 2 conn_max_lifetime: -1 - Implement connection parameter setup on the *DB (database/sql) in internal/sqlutil/trace.go:Open() - Propagate the values in the form of DbProperties interface via all the Open() and NewDatabase() functions Signed-off-by: Tomas Jirka <tomas.jirka@email.cz> * Fix wasm builds * Remove file accidentally added from working tree Co-authored-by: Tomas Jirka <tomas.jirka@email.cz>
2020-05-01Rename FS queue package to internal (#997)Neil Alexander
2020-05-01Consolidation of roomserver APIs (#994)Neil Alexander
* Consolidation of roomserver APIs * Comment out alias tests for now, they are broken * Wire AS API into roomserver again * Roomserver didn't take asAPI param before so return to that * Prevent roomserver asking AS API for alias info * Rename some files * Remove alias_test, incoherent tests and unwanted appservice integration * Remove FS API inject on syncapi component
2020-04-29Join room support in federation sender (#989)Neil Alexander
* Implement PerformJoinRequest * Rename perform functions * Check send join response * Temporary wiring to test federation sender room joins * Actually pass through the config * Make sure membership content shows join
2020-04-29Federation sender API remodel (#988)Neil Alexander
* Define an input API for the federationsender * Wiring for rooomserver input API and federation sender input API * Whoops, commit common too * Merge input API into query API * Rename FederationSenderQueryAPI to FederationSenderInternalAPI * Fix dendritejs * Rename Input to Perform * Fix a couple of inputs -> performs * Remove needless storage interface, add comments
2020-04-28Loopback event from invite response (#982)Neil Alexander
* Working invite v2 support * Fix copyright notice * Update gomatrixserverlib * Add fallthrough * Add missing continue * Update sytest-whitelist, gomatrixserverlib * Update gomatrixserverlib to test matrix-org/gomatrixserverlib#181 * Update gomatrixserverlib
2020-04-24More invite support (#979)Neil Alexander
* Update gomatixserverlib * Try to build invite stripped state if not given to us * SendInvite improvements * Transpose invite_room_state into invite_state.events for sync API * Remove syncapi debugging output * Use RespInviteV2 * Update gomatrixserverlib * Send the invite event as a normal roomserver event too, for incorporating into room (should this be done by the roomserver automatically for invite inputs?) * Federation sender use invite_room_state, room server try to insert membership state * Check supported room versions on the invite endpoint * Prevent roomserver query API from trying to handle requests for stub rooms * Adding a nolint * Replace IsRoomStub with RoomNIDExcludingStubs, fix query API to use that instead * Review comments
2020-04-22Fix sarama import URLs (#856)Neil Alexander
* Fix sarama import URLs * Update gomatrixserverlib * Update naffka * Update naffka * Update in kafka-producer
2020-04-16Add SQL tracing via DENDRITE_TRACE_SQL (#968)Kegsay
* Add SQL tracing via DENDRITE_TRACE_SQL Add this to `internal/sqlutil` in preparation for #897 * Not entirely
2020-04-03Invites v2 endpoint (#952)Neil Alexander
* Start converting v1 invite endpoint to v2 * Update gomatrixserverlib * Early federationsender code for sending invites * Sending invites sorta happens now * Populate invite request with stripped state * Remodel a bit, don't reflect received invites * Handle invite_room_state * Handle room versions a bit better * Update gomatrixserverlib * Tweak order in destinationQueue.next * Revert check in processMessage * Tweak federation sender destination queue code a bit * Add comments
2020-04-03use custom http client instead of the http DefaultClient (#823)Ben B
This commit replaces the default client from the http lib with a custom one. The previously used default client doesn't come with a timeout. This could cause unwanted locks. That solution chosen here creates a http client in the base component dendrite with a constant timeout of 30 seconds. If it should be necessary to overwrite this, we could include the timeout in the dendrite configuration. Here it would be a good idea to extend the type "Address" by a timeout and create an http client for each service. Closes #820 Signed-off-by: Benedikt Bongartz <benne@klimlive.de> Co-authored-by: Kegsay <kegan@matrix.org>
2020-03-30Implement typing over federation (#949)Kegsay
Also fix a pet peeve of mine: not putting units on things!!! Manually tested on p2p and works well, with some fudge factor delay.
2020-03-30Rename the typing server to EDU server (#948)Kegsay
* Blunt move and sed rename * Rename common/ refs to typing * Rename internal stuff in eduserver * Rename docs and scripts * Rename constants/filenames, goimports everything to re-order imports
2020-03-27Further room version wiring (#936)Neil Alexander
* Room version 2 by default, other wiring updates, update gomatrixserverlib * Fix nil pointer exception * Fix some more nil pointer exceptions hopefully * Update gomatrixserverlib * Send all room versions when joining, not just stable ones * Remove room version cquery * Get room version when getting events from the roomserver database * Reset default back to room version 2 * Don't generate event IDs unless needed * Revert "Remove room version cquery" This reverts commit a170d5873360dd059614460acc8b21ab2cda9767. * Query room version in federation API, client API as needed * Improvements to make_join send_join dance * Make room server producers use headered events * Lint tweaks * Update gomatrixserverlib * Versioned SendJoin * Query room version in syncapi backfill * Handle transaction marshalling/unmarshalling within Dendrite * Sorta fix federation (kinda) * whoops commit federation API too * Use NewEventFromTrustedJSON when getting events from the database * Update gomatrixserverlib * Strip headers on federationapi endpoints * Fix bug in clientapi profile room version query * Update gomatrixserverlib * Return more useful error if room version query doesn't find the room * Update gomatrixserverlib * Update gomatrixserverlib * Maybe fix federation * Fix formatting directive * Update sytest whitelist and blacklist * Temporarily disable room versions 3 and 4 until gmsl is fixed * Fix count of EDUs in logging * Update gomatrixserverlib * Update gomatrixserverlib * Update gomatrixserverlib * Rely on EventBuilder in gmsl to generate the event IDs for us * Some review comments fixed * Move function out of common and into gmsl * Comment in federationsender destinationqueue * Update gomatrixserverlib
2020-03-18Log errors from rows.Close (#920)Prateek Sachan
* Log errors from rows.Close * fixed imports * Added contextual messages * fixed review changes
2020-03-17Implement gomatrixserverlib.HeaderedEvent in roomserver Kafka output (#914)Neil Alexander
* Use Event.Headered * Use HeaderedEvent in roomserver kafka output * Fix syncserver-integration-tests * Update producers to roomserver inputs * Update gomatrixserverlib * Update gomatrixserverlib * Update gomatrixserverlib * Update gomatrixserverlib * Update gomatrixserverlib * Update gomatrixserverlib
2020-03-16Implement gomatrixserverlib.HeaderedEvent in roomserver query API (#912)Neil Alexander
* Implement gomatrixserverlib.HeaderedEvent, which should allow us to store room version headers along with the event across API boundaries and consumers/producers, and intercept unmarshalling to get the event structure right * Add federationsender to previous
2020-03-06Add peer-to-peer support into Dendrite via libp2p and fetch (#880)Kegsay
* Use a fork of pq which supports userCurrent on wasm * Use sqlite3_js driver when running in JS * Add cmd/dendritejs to pull in sqlite3_js driver for wasm only * Update to latest go-sqlite-js version * Replace prometheus with a stub. sigh * Hard-code a config and don't use opentracing * Latest go-sqlite3-js version * Generate a key for now * Listen for fetch traffic rather than HTTP * Latest hacks for js * libp2p support * More libp2p * Fork gjson to allow us to enforce auth checks as before Previously, all events would come down redacted because the hash checks would fail. They would fail because sjson.DeleteBytes didn't remove keys not used for hashing. This didn't work because of a build tag which included a file which no-oped the index returned. See https://github.com/tidwall/gjson/issues/157 When it's resolved, let's go back to mainline. * Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157 * Use latest gomatrixserverlib for sig checks * Fix a bug which could cause exclude_from_sync to not be set Caused when sending events over federation. * Use query variadic to make lookups actually work! * Latest gomatrixserverlib * Add notes on getting p2p up and running Partly so I don't forget myself! * refactor: Move p2p specific stuff to cmd/dendritejs This is important or else the normal build of dendrite will fail because the p2p libraries depend on syscall/js which doesn't work on normal builds. Also, clean up main.go to read a bit better. * Update ho-http-js-libp2p to return errors from RoundTrip * Add an LRU cache around the key DB We actually need this for P2P because otherwise we can *segfault* with things like: "runtime: unexpected return pc for runtime.handleEvent" where the event is a `syscall/js` event, caused by spamming sql.js caused by "Checking event signatures for 14 events of room state" which hammers the key DB repeatedly in quick succession. Using a cache fixes this, though the underlying cause is probably a bug in the version of Go I'm on (1.13.7) * breaking: Add Tracing.Enabled to toggle whether we do opentracing Defaults to false, which is why this is a breaking change. We need this flag because WASM builds cannot do opentracing. * Start adding conditional builds for wasm to handle lib/pq The general idea here is to have the wasm build have a `NewXXXDatabase` that doesn't import any postgres package and hence we never import `lib/pq`, which doesn't work under WASM (undefined `userCurrent`). * Remove lib/pq for wasm for syncapi * Add conditional building to remaining storage APIs * Update build script to set env vars correctly for dendritejs * sqlite bug fixes * Docs * Add a no-op main for dendritejs when not building under wasm * Use the real prometheus, even for WASM Instead, the dendrite-sw.js must mock out `process.pid` and `fs.stat` - which must invoke the callback with an error (e.g `EINVAL`) in order for it to work: ``` global.process = { pid: 1, }; global.fs.stat = function(path, cb) { cb({ code: "EINVAL", }); } ``` * Linting
2020-02-28Make sure PDUs and EDUs in transaction don't marshal to null (#876)Neil Alexander
2020-02-13Support sqlite in addition to postgres (#869)Kegsay
* Move current work into single branch * Initial massaging of clientapi etc (not working yet) * Interfaces for accounts/devices databases * Duplicate postgres package for sqlite3 (no changes made to it yet) * Some keydb, accountdb, devicedb, common partition fixes, some more syncapi tweaking * Fix accounts DB, device DB * Update naffka dependency for SQLite * Naffka SQLite * Update naffka to latest master * SQLite support for federationsender * Mostly not-bad support for SQLite in syncapi (although there are problems where lots of events get classed incorrectly as backward extremities, probably because of IN/ANY clauses that are badly supported) * Update Dockerfile -> Go 1.13.7, add build-base (as gcc and friends are needed for SQLite) * Implement GET endpoints for account_data in clientapi * Nuke filtering for now... * Revert "Implement GET endpoints for account_data in clientapi" This reverts commit 4d80dff4583d278620d9b3ed437e9fcd8d4674ee. * Implement GET endpoints for account_data in clientapi (#861) * Implement GET endpoints for account_data in clientapi * Fix accountDB parameter * Remove fmt.Println * Fix insertAccountData SQLite query * Fix accountDB storage interfaces * Add empty push rules into account data on account creation (#862) * Put SaveAccountData into the right function this time * Not sure if roomserver is better or worse now * sqlite work * Allow empty last sent ID for the first event * sqlite: room creation works * Support sending messages * Nuke fmt.println * Move QueryVariadic etc into common, other device fixes * Fix some linter issues * Fix bugs * Fix some linting errors * Fix errcheck lint errors * Make naffka use postgres as fallback, fix couple of compile errors * What on earth happened to the /rooms/{roomID}/send/{eventType} routing Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-02-11Always defer *sql.Rows.Close and consult with Err (#844)Kiril Vladimiroff
* Always defer *sql.Rows.Close and consult with Err database/sql.Rows.Next() makes sure to call Close only after exhausting result rows which would NOT happen when returning early from a bad Scan. Close being idempotent makes it a great candidate to get always deferred regardless of what happens later on the result set. This change also makes sure call Err() after exhausting Next() and propagate non-nil results from it as the documentation advises. Closes #764 Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org> * Override named result parameters in last returns Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org> * Do the same over new changes that got merged Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-01-09Fix #842 - again...Neil Alexander
2020-01-09Fix #842Neil Alexander
2020-01-09Fall back to postgres when database connection string parsing fails (#842)Neil Alexander
* Fall back to postgres when parsing the database connection string for a URI schema fails * Fix behaviour so that it really tries postgres when URL parsing fails and it complains about unknown schema if it succeeds
2020-01-03Implement storage interfaces (#841)Neil Alexander
* Implement interfaces for federationsender storage * Implement interfaces for mediaapi storage * Implement interfaces for publicroomsapi storage * Implement interfaces for roomserver storage * Implement interfaces for syncapi storage * Implement interfaces for keydb storage * common.PartitionStorer in publicroomsapi interface * Update copyright notices
2019-12-20Return server names (#833)Neil Alexander
* Remove unnecessary map->array processing * Return server names in room federation directory query * Knock off a TODO
2019-08-24Set up queryAPI in federationsender (#791)Alex Chen
2019-08-22Add joined hosts query APIs (#781)Alex Chen
This adds two joined hosts query APIs to the federation sender for use of other components.
2019-08-06Replace membership and visibility values with constants (#774)Alex Chen
Signed-off-by: Alex Chen <minecnly@gmail.com>
2019-07-01Don't print Sending EDU if there is noone to send to (#721)Andrew Morgan
The logs had a lot of: ``` Sending EDU event destinations="[]" edu_type=m.typing ``` Which is useless if it isn't actually sending the event anywhere (destinations is empty).
2019-05-21use go module for dependencies (#594)ruben
generated by cgit v1.2.3 (git 2.26.2) at 2024-12-02 21:51:43 +0000