aboutsummaryrefslogtreecommitdiff
path: root/cmd/dendrite-monolith-server
AgeCommit message (Collapse)Author
2020-08-20Add FederationClient interface to federationsender (#1284)Kegsay
* Add FederationClient interface to federationsender - Use a shim struct in HTTP mode to keep the same API as `FederationClient`. - Use `federationsender` instead of `FederationClient` in `keyserver`. * Pointers not values * Review comments * Fix unit tests * Rejig backoff * Unbreak test * Remove debug logs * Review comments and linting
2020-08-13API setup refactoring (#1266)Neil Alexander
* Start HTTP endpoint refactoring * Update SetupAndServeHTTP * Fix builds * Don't set up external listener if no address configured * TLS HTTP setup * Break apart client/federation/key/media muxes * Tweaks * Fix P2P demos * Fix media API routing * Review comments @Kegsay * Update sample config * Fix gobind build * Fix External -> Public in federation API test
2020-08-10Configuration format v1 (#1230)Neil Alexander
* Initial pass at refactoring config (not finished) * Don't forget current state and EDU servers * More shifting around * Update server key API tests * Fix roomserver test * Fix more tests * Further tweaks * Fix current state server test (sort of) * Maybe fix appservices * Fix client API test * Include database connection string in database options * Fix sync API build * Update config test * Fix unit tests * Fix federation sender build * Fix gobind build * Set Listen address for all services in HTTP monolith mode * Validate config, reinstate appservice derived in directory, tweaks * Tweak federation API test * Set MaxOpenConnections/MaxIdleConnections to previous values * Update generate-config
2020-08-07Allow enforcing X.509 certificate validity (MSC1711) (#1249)Neil Alexander
* Configurable X.509 certificate validation * Fix dendritejs * Update go.mod/go.sum for matrix-org/gomatrixserverlib#214 * Update sample config
2020-08-04Send device list updates to servers (outbound only) (#1237)Kegsay
* Add QueryDeviceMessages to serve up device keys and stream IDs * Consume key change events in fedsender Don't yet send them to destinations as we haven't worked them out yet * Send device list updates to all required servers * Glue it all together
2020-07-30Notify clients when devices are deleted (#1233)Kegsay
* Recheck device lists when join/leave events come in * Add PerformDeviceDeletion * Notify clients when devices are deleted * Unbreak things * Remove debug logging
2020-07-23keyserver: produce key change events (#1218)Kegsay
* Produce kafka events when keys are added * Consume key changes in syncapi with TODO markers for handling them and catching up * unbreak tests * Linting
2020-07-22Handle inbound federation E2E key queries/claims (#1215)Kegsay
* Handle inbound /keys/claim and /keys/query requests * Add display names to device key responses * Linting
2020-07-21Perform outbound federation hits for querying/claiming E2E keys (#1212)Kegsay
* Perform outbound federation hits for querying/claiming E2E keys Untested currently because we need the receiving end to work before sytest will be happy. * Linting
2020-07-15Implement key uploads (#1202)Kegsay
* Add storage layer for postgres/sqlite * Return OTK counts when inserting new keys * Hook up the key DB and make a test pass * Convert postgres queries to be sqlite queries * Blacklist test due to requiring rejected events * Unbreak tests * Update blacklist
2020-07-13Add boilerplate for key server APIs (#1196)Kegsay
Also add a README which outilnes how things will work.
2020-07-02Remove publicroomsapi (#1176)Kegsay
* Remove all of publicroomsapi * Remove references to publicroomsapi * Remove doc references to publicroomsapi
2020-06-30Remove membership table from account DB (#1172)Kegsay
* Remove membership table from account DB And make code which needs that data use the currentstate server * Unbreak tests; use a membership enum for space
2020-06-23LintingKegan Dougal
2020-06-23Add instrumented main for coverageKegan Dougal
2020-06-16BREAKING: Make eduserver/appservice use userapi (#1138)Kegsay
* BREAKING: Make eduserver/appservice use userapi This is a breaking change because this PR restructures how the AS API tracks its position in Kafka streams. Previously, it used the account DB to store partition offsets. However, this is also being used by `clientapi` for the same purpose, which is bad (each component needs to store offsets independently or else you might lose messages across restarts). This PR changes this behaviour to now store partition offsets in the `appservice` database. This means that: - Upon restart, the `appservice` component will attempt to replay all room events from the beginning of time. - An additional table will be created in the appservice database, which in and of itself is backwards compatible. * Return ErrorConflict
2020-06-16Fix media API for demos and possibly Synapse (#1134)Neil Alexander
* Fix media API for demos and possibly Synapse * User API * goimports
2020-06-16Make userapi responsible for checking access tokens (#1133)Kegsay
* Make userapi responsible for checking access tokens There's still plenty of dependencies on account/device DBs, but this is a start. This is a breaking change as it adds a required config value `listen.user_api`. * Cleanup * Review comments and test fix
2020-06-15Unbreak HTTP mode (#1131)Kegsay
2020-06-15Fix rooms v3 url paths for good - with tests (#1130)Kegsay
* Fix rooms v3 url paths for good - with tests - Add a test rig around `federationapi` to test routing. - Use `JSONVerifier` over `KeyRing` so we can stub things out more easily. - Add `test.NopJSONVerifier` which verifies nothing. - Add `base.BaseMux` which is the original `mux.Router` used to spawn public/internal routers. - Listen on `base.BaseMux` and not the default serve mux as it cleans paths which we don't want. - Factor out `ListenAndServe` to `test.ListenAndServe` and add flag for listening on TLS. * Fix comments * Linting
2020-06-12Rehuffle where things are in the internal package (#1122)Kegsay
renamed: internal/eventcontent.go -> internal/eventutil/eventcontent.go renamed: internal/events.go -> internal/eventutil/events.go renamed: internal/types.go -> internal/eventutil/types.go renamed: internal/http/http.go -> internal/httputil/http.go renamed: internal/httpapi.go -> internal/httputil/httpapi.go renamed: internal/httpapi_test.go -> internal/httputil/httpapi_test.go renamed: internal/httpapis/paths.go -> internal/httputil/paths.go renamed: internal/routing.go -> internal/httputil/routing.go renamed: internal/basecomponent/base.go -> internal/setup/base.go renamed: internal/basecomponent/flags.go -> internal/setup/flags.go renamed: internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go renamed: internal/postgres.go -> internal/sqlutil/postgres.go renamed: internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go renamed: internal/sql.go -> internal/sqlutil/sql.go
2020-06-12Add trace logging to RoomserverInternalAPI (#1120)Kegsay
This is a wrapper around whatever impl we have which then logs the function name/request/response/error. Also tweak when we log on kafka streams: only log on the producer side not the consumer side: we've never had issues with comms and having 1 message rather than N would be nice.
2020-06-10Remove clientapi producers which aren't actually producers (#1111)Kegsay
* Remove clientapi producers which aren't actually producers They are actually just convenience wrappers around the internal APIs for roomserver/eduserver. Move their logic to their respective `api` packages and call them directly. * Remove TODO * unbreak ygg
2020-06-09Factor out how monolith routes get added (#1107)Kegsay
Previously we had 3 monoliths: - dendrite-monolith-server - dendrite-demo-libp2p - dendritejs which all had their own of setting up public routes. Factor this out into a new `setup.Monolith` struct which gets all dependencies set as fields. This is different to `basecomponent.Base` which doesn't provide any way to set configured deps (e.g public rooms db) Part of a larger process to clean up how we initialise Dendrite.
2020-06-08Split out SetupFooComponent (#1106)Kegsay
* Split out adding HTTP routes from making internal APIs for clarity * Split out more components * Split out more things * Finish converting * internal mux for internal routes
2020-06-05General cleanup when making components (#1098)Kegsay
* Remove ParseMonolith/LoadMonolith * cleanup which components need to be made
2020-06-04Convert serverkeys to inthttp (#1097)Kegsay
2020-06-04Convert everything but serverkeyapi to inthttp (#1096)Kegsay
* Convert roomserver to new inthttp format * Convert eduserver to new inthttp format * Convert appservice to new inthttp format
2020-06-04Refactor how federationsender gets created (#1095)Kegsay
* Refactor how federationsender gets created * s/httpint/inthttp/ for alphabetical niceness with internal package
2020-06-02Only store our own aliases in publicroomsapi (#1081)Kegsay
Otherwise we just store the latest aliases submitted from a server, which is not what we want.
2020-06-01Send-to-device support (#1072)Neil Alexander
* Groundwork for send-to-device messaging * Update sample config * Add unstable routing for now * Send to device consumer in sync API * Start the send-to-device consumer * fix indentation in dendrite-config.yaml * Create send-to-device database tables, other tweaks * Add some logic for send-to-device messages, add them into sync stream * Handle incoming send-to-device messages, count them with EDU stream pos * Undo changes to test * pq.Array * Fix sync * Logging * Fix a couple of transaction things, fix client API * Add send-to-device test, hopefully fix bugs * Comments * Refactor a bit * Fix schema * Fix queries * Debug logging * Fix storing and retrieving of send-to-device messages * Try to avoid database locks * Update sync position * Use latest sync position * Jiggle about sync a bit * Fix tests * Break out the retrieval from the update/delete behaviour * Comments * nolint on getResponseWithPDUsForCompleteSync * Try to line up sync tokens again * Implement wildcard * Add all send-to-device tests to whitelist, what could possibly go wrong? * Only care about wildcard when targeted locally * Deduplicate transactions * Handle tokens properly, return immediately if waiting send-to-device messages * Fix sync * Update sytest-whitelist * Fix copyright notice (need to do more of this) * Comments, copyrights * Return errors from Do, fix dendritejs * Review comments * Comments * Constructor for TransactionWriter * defletions * Update gomatrixserverlib, sytest-blacklist
2020-05-27Server key component (#1050)Neil Alexander
* Server key API (works for monolith but not for polylith yet) * Re-enable caching on server key API component * Groundwork for HTTP APIs for server key API * Hopefully implement HTTP for server key API * Simplify public key request marshalling from map keys * Update gomatrixserverlib * go mod tidy * Common -> internal * remove keyring.go * Update Docker Hub for server key API * YAML is funny about indentation * Wire in new server key API into hybrid monolith mode * Create maps * Route server key API endpoints on internal API mux * Fix server key API URLs * Add fetcher behaviour into server key API implementation * Return error if we failed to fetch some keys * Return results anyway * Move things about a bit * Remove unused code * Fix comments, don't use federation sender URL in polylith mode * Add server_key_api to sample config * Review comments * HTTP API to cache keys that have been requested * Overwrite server_key_api listen in monolith hybrid mode
2020-05-22Fix monolith room server-federation sender connectionNeil Alexander
2020-05-22Use HTTP APIs when -api specified (#1057)Neil Alexander
2020-05-22Separate muxes for public and internal APIs (#1056)Neil Alexander
* Separate muxes for public and internal APIs * Update client-api-proxy and federation-api-proxy so they don't add /api to the path * Tidy up * Consistent HTTP setup * Set up prefixes properly
2020-05-21Fix #897 and shuffle directory around (#1054)Kegsay
* Fix #897 and shuffle directory around * Update find-lint * goimports Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-18Add -api flag to monolith (#1044)Neil Alexander
* Add flag for enabling HTTP APIs in monolith mode * Flag -api * Only start HTTP APIs if needed
2020-05-14Keyserver skeleton (#1032)Neil Alexander
* Keyserver skeleton * Indentation
2020-05-01Limit database connections (#980, #564) (#998)Neil Alexander
* Limit database connections (#564) - Add new options to the config file database: max_open_conns: 100 max_idle_conns: 2 conn_max_lifetime: -1 - Implement connection parameter setup on the *DB (database/sql) in internal/sqlutil/trace.go:Open() - Propagate the values in the form of DbProperties interface via all the Open() and NewDatabase() functions Signed-off-by: Tomas Jirka <tomas.jirka@email.cz> * Fix wasm builds * Remove file accidentally added from working tree Co-authored-by: Tomas Jirka <tomas.jirka@email.cz>
2020-05-01Consolidation of roomserver APIs (#994)Neil Alexander
* Consolidation of roomserver APIs * Comment out alias tests for now, they are broken * Wire AS API into roomserver again * Roomserver didn't take asAPI param before so return to that * Prevent roomserver asking AS API for alias info * Rename some files * Remove alias_test, incoherent tests and unwanted appservice integration * Remove FS API inject on syncapi component
2020-04-29Join room support in federation sender (#989)Neil Alexander
* Implement PerformJoinRequest * Rename perform functions * Check send join response * Temporary wiring to test federation sender room joins * Actually pass through the config * Make sure membership content shows join
2020-04-29Federation sender API remodel (#988)Neil Alexander
* Define an input API for the federationsender * Wiring for rooomserver input API and federation sender input API * Whoops, commit common too * Merge input API into query API * Rename FederationSenderQueryAPI to FederationSenderInternalAPI * Fix dendritejs * Rename Input to Perform * Fix a couple of inputs -> performs * Remove needless storage interface, add comments
2020-04-28Dependency inject the federation client so p2p binaries work as expected (#987)Kegsay
2020-04-28Implement backfill in the roomserver (#983)Kegsay
* Initial cut for backfilling The syncserver now asks the roomserver via QueryBackfill (which already existed to *handle* backfill requests) which then makes federation requests via gomatrixserverlib.RequestBackfill. Currently, tests fail on subsequent /messages requests because we don't know which servers are in the room, because we are unable to get state snapshots from a backfilled event because that code doesn't exist yet. * WIP backfill, doesn't work * Make initial backfill pass checks * Persist backfilled events with state snapshots * Remove debug lines * Linting * Review comments
2020-04-28Loopback event from invite response (#982)Neil Alexander
* Working invite v2 support * Fix copyright notice * Update gomatrixserverlib * Add fallthrough * Add missing continue * Update sytest-whitelist, gomatrixserverlib * Update gomatrixserverlib to test matrix-org/gomatrixserverlib#181 * Update gomatrixserverlib
2020-04-20Perspective key fetching, some federation room join fixes (#975)Neil Alexander
* Update gomatrixserverlib * Test matrix.org as perspective key server * Base64 decode better * Optional strict validity checking in gmsl * Update gomatrixserverlib * Attempt to find missing auth events over federation (this shouldn't happen but I am guessing there is a synapse bug involved where we don't get all of the auth events) * Update gomatrixserverlib, debug logging * Remove debugging output * More verbose debugging * Print outliers * Increase timeouts for testing, observe contexts before trying to join over more servers * Don't block on roomserver (experimental) * Don't block on roomserver * Update gomatrixserverlib * Update gomatrixserverlib * Configurable perspective key fetchers * Output number of configured keys for perspective * Example perspective config included * Undo debug stack trace * Undo debug stack trace * Restore original HTTP listener in monolith * Fix lint * Review comments * Set default HTTP server timeout to 5 minutes now, block again when joining * Don't use HTTP address for HTTPS whoops * Update gomatrixserverlib * Update gomatrixserverlib * Update gomatrixserverlib * Actually add perspectives * Actually add perspectives * Update gomatrixserverlib
2020-04-14WIP: Add libp2p-go (#956)Hilmar Gústafsson
* Add libp2p-go * Some tweaks, tidying up (cherry picked from commit 1a5bb121f8121c4f68a27abbf25a9a35a1b7c63e) * Move p2p dockerfile (cherry picked from commit 8d3bf44ea1bf37f950034e73bcdc315afdabe79a) * Remove containsBackwardsExtremity * Fix some linter errors, update some libp2p packages/calls, other tidying up * Add -port for dendrite-p2p-demo * Use instance name as key ID * Remove P2P demo docker stuff, no longer needed now that we have SQLite * Remove Dockerfile-p2p too * Remove p2p logic from dendrite-monolith-server * Inject publicRoomsDB in publicroomsapi Inject publicRoomsDB instead of switching on base.libP2P. See: https://github.com/matrix-org/dendrite/pull/956/files?file-filters%5B%5D=.go#r406276914 * Fix lint warning * Extract mDNSListener from base.go * Extract CreateFederationClient into demo * Create P2PDendrite from BaseDendrite Extract logic specific to P2PDendrite from base.go * Set base.go to upstream/master * Move pubsub to demo cmd * Move PostgreswithDHT to cmd * Remove unstable features * Add copyrights * Move libp2pvalidator into p2pdendrite * Rename dendrite-p2p-demo -> dendrite-demo-libp2p * Update copyrights * go mod tidy Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-04-14S7evinK: basicauth metrics (#961)Kegsay
* Add setting to enable/disable metrics (#461) Add basic auth to /metric handlers Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Add warning message if metrics are exposed without protection * Remove redundant type conversion Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * SetBasicAuth per test case * Update warning message and change loglevel to warn * Update common/config/config.go * Update dendrite-config.yaml Co-authored-by: Till Faelligen <tfaelligen@gmail.com> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-03-30Implement typing over federation (#949)Kegsay
Also fix a pet peeve of mine: not putting units on things!!! Manually tested on p2p and works well, with some fudge factor delay.
2020-03-30Rename the typing server to EDU server (#948)Kegsay
* Blunt move and sed rename * Rename common/ refs to typing * Rename internal stuff in eduserver * Rename docs and scripts * Rename constants/filenames, goimports everything to re-order imports
generated by cgit v1.2.3 (git 2.26.2) at 2024-12-20 06:55:25 +0000