aboutsummaryrefslogtreecommitdiff
path: root/common
AgeCommit message (Collapse)Author
2020-05-21Fix #897 and shuffle directory around (#1054)Kegsay
* Fix #897 and shuffle directory around * Update find-lint * goimports Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-18Add -api flag to monolith (#1044)Neil Alexander
* Add flag for enabling HTTP APIs in monolith mode * Flag -api * Only start HTTP APIs if needed
2020-05-15Only keep keys valid for 100 years rather than the heat death of the universeKegan Dougal
WASM builds really dislike thinking that far into the future, causing WASM to trap with "float unrepresentable in integer range", which corrupts the Go stack in wasm_exec which then leads to a segfault and the program exiting.
2020-05-15Set canonical alias on room creation (#1042)Kegsay
2020-05-15Prometheus metrics for LRU cache (#1039)Neil Alexander
* Add prom metrics for the in-memory LRU cache * Increase cache sizes
2020-05-15Implement key caching directly (#1038)Neil Alexander
* Use gomatrixserverlib key caching * Implement key caching wrapper * Add caching wrapper in BaseComponent * Review comments
2020-05-14Keyserver skeleton (#1032)Neil Alexander
* Keyserver skeleton * Indentation
2020-05-11Add PPROFLISTEN (#1019)Neil Alexander
* Add PPROFLISTEN env var * Direct logging to more useful places * Space
2020-05-01Limit database connections (#980, #564) (#998)Neil Alexander
* Limit database connections (#564) - Add new options to the config file database: max_open_conns: 100 max_idle_conns: 2 conn_max_lifetime: -1 - Implement connection parameter setup on the *DB (database/sql) in internal/sqlutil/trace.go:Open() - Propagate the values in the form of DbProperties interface via all the Open() and NewDatabase() functions Signed-off-by: Tomas Jirka <tomas.jirka@email.cz> * Fix wasm builds * Remove file accidentally added from working tree Co-authored-by: Tomas Jirka <tomas.jirka@email.cz>
2020-05-01Consolidation of roomserver APIs (#994)Neil Alexander
* Consolidation of roomserver APIs * Comment out alias tests for now, they are broken * Wire AS API into roomserver again * Roomserver didn't take asAPI param before so return to that * Prevent roomserver asking AS API for alias info * Rename some files * Remove alias_test, incoherent tests and unwanted appservice integration * Remove FS API inject on syncapi component
2020-04-29Federation sender API remodel (#988)Neil Alexander
* Define an input API for the federationsender * Wiring for rooomserver input API and federation sender input API * Whoops, commit common too * Merge input API into query API * Rename FederationSenderQueryAPI to FederationSenderInternalAPI * Fix dendritejs * Rename Input to Perform * Fix a couple of inputs -> performs * Remove needless storage interface, add comments
2020-04-28Fix URLDecodeMapValues (#984)Neil Alexander
* Update gomatrixserverlib * Fix URLDecodeMapValues * Update gomatrixserverlib
2020-04-22Fix sarama import URLs (#856)Neil Alexander
* Fix sarama import URLs * Update gomatrixserverlib * Update naffka * Update naffka * Update in kafka-producer
2020-04-22LRU cache for room versions in RS query API (#976)Neil Alexander
* Experimental LRU cache for room versions * Don't accidentally try to type-assert nil * Also reduce hits on query API * Use hashicorp implementation which mutexes for us * Define const for max cache entries * Rename to be specifically immutable, panic if we try to mutate a cache entry * Review comments * Remove nil guards, give roomserver integration test a cache * go mod tidy
2020-04-20Perspective key fetching, some federation room join fixes (#975)Neil Alexander
* Update gomatrixserverlib * Test matrix.org as perspective key server * Base64 decode better * Optional strict validity checking in gmsl * Update gomatrixserverlib * Attempt to find missing auth events over federation (this shouldn't happen but I am guessing there is a synapse bug involved where we don't get all of the auth events) * Update gomatrixserverlib, debug logging * Remove debugging output * More verbose debugging * Print outliers * Increase timeouts for testing, observe contexts before trying to join over more servers * Don't block on roomserver (experimental) * Don't block on roomserver * Update gomatrixserverlib * Update gomatrixserverlib * Configurable perspective key fetchers * Output number of configured keys for perspective * Example perspective config included * Undo debug stack trace * Undo debug stack trace * Restore original HTTP listener in monolith * Fix lint * Review comments * Set default HTTP server timeout to 5 minutes now, block again when joining * Don't use HTTP address for HTTPS whoops * Update gomatrixserverlib * Update gomatrixserverlib * Update gomatrixserverlib * Actually add perspectives * Actually add perspectives * Update gomatrixserverlib
2020-04-19Fix nil http clients provided to components in polylith mode (#973)Alex Chen
Signed-off-by: Alex Chen <minecnly@gmail.com>
2020-04-17Allow anyone to publish rooms (#971)Kegsay
* Allow anyone to publish rooms * Formatting and unused var * Review comments
2020-04-16Add SQL tracing via DENDRITE_TRACE_SQL (#968)Kegsay
* Add SQL tracing via DENDRITE_TRACE_SQL Add this to `internal/sqlutil` in preparation for #897 * Not entirely
2020-04-15Add HTTP trace logging (#965)Kegsay
* Dump all requests/response server-side * Wrap in DENDRITE_TRACE * DENDRITE_TRACE_HTTP is better * Bugfix for response body and linting * False is true and true is false * Linting * How did this get missed * More linting
2020-04-14Move /room/{roomID}/state endpoints into client API (#606) (#962)Neil Alexander
* Move /room/{roomID}/state endpoints into client API (#606) * Update sytest-whitelist * Blacklist tests which rely on endpoints we don't implement
2020-04-14S7evinK: basicauth metrics (#961)Kegsay
* Add setting to enable/disable metrics (#461) Add basic auth to /metric handlers Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Add warning message if metrics are exposed without protection * Remove redundant type conversion Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * SetBasicAuth per test case * Update warning message and change loglevel to warn * Update common/config/config.go * Update dendrite-config.yaml Co-authored-by: Till Faelligen <tfaelligen@gmail.com> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-04-14support non-exclusive namespaces for AS (#828)mohit kumar singh
Signed-off-by: MohitKS5 <mohitkumarsingh907@gmail.com> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> Co-authored-by: Kegsay <kegan@matrix.org>
2020-04-09Federation for v3/v4 rooms (#954)Neil Alexander
* Update gomatrixserverlib * Default to room version 4 * Update gomatrixserverlib * Limit prev_events and auth_events * Fix auth_events, prev_events * Fix linter issues * Update gomatrixserverlib * Fix getState * Update sytest-whitelist * Squashed commit of the following: commit 067b87506357c996fd6ddb11271db9469ad4ce80 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Fri Apr 3 14:29:06 2020 +0100 Invites v2 endpoint (#952) * Start converting v1 invite endpoint to v2 * Update gomatrixserverlib * Early federationsender code for sending invites * Sending invites sorta happens now * Populate invite request with stripped state * Remodel a bit, don't reflect received invites * Handle invite_room_state * Handle room versions a bit better * Update gomatrixserverlib * Tweak order in destinationQueue.next * Revert check in processMessage * Tweak federation sender destination queue code a bit * Add comments commit 955244c09298d0e6c870377dad3af2ffa1f5e578 Author: Ben B <benne@klimlive.de> Date: Fri Apr 3 12:40:50 2020 +0200 use custom http client instead of the http DefaultClient (#823) This commit replaces the default client from the http lib with a custom one. The previously used default client doesn't come with a timeout. This could cause unwanted locks. That solution chosen here creates a http client in the base component dendrite with a constant timeout of 30 seconds. If it should be necessary to overwrite this, we could include the timeout in the dendrite configuration. Here it would be a good idea to extend the type "Address" by a timeout and create an http client for each service. Closes #820 Signed-off-by: Benedikt Bongartz <benne@klimlive.de> Co-authored-by: Kegsay <kegan@matrix.org> * Update sytest-whitelist, sytest-blacklist * Update go.mod/go.sum * Add some error wrapping for debug * Add a NOTSPEC to common/events.go * Perform state resolution at send_join * Set default room version to v2 again * Tweak GetCapabilities * Add comments to ResolveConflictsAdhoc * Update sytest-blacklist * go mod tidy * Update sytest-whitelist, sytest-blacklist * Update versions * Updates from review comments * Update sytest-blacklist, sytest-whitelist * Check room versions compatible at make_join, add some comments, update gomatrixserverlib, other tweaks * Set default room version back to v2 * Update gomatrixserverlib, sytest-whitelist
2020-04-03use custom http client instead of the http DefaultClient (#823)Ben B
This commit replaces the default client from the http lib with a custom one. The previously used default client doesn't come with a timeout. This could cause unwanted locks. That solution chosen here creates a http client in the base component dendrite with a constant timeout of 30 seconds. If it should be necessary to overwrite this, we could include the timeout in the dendrite configuration. Here it would be a good idea to extend the type "Address" by a timeout and create an http client for each service. Closes #820 Signed-off-by: Benedikt Bongartz <benne@klimlive.de> Co-authored-by: Kegsay <kegan@matrix.org>
2020-03-30Rename the typing server to EDU server (#948)Kegsay
* Blunt move and sed rename * Rename common/ refs to typing * Rename internal stuff in eduserver * Rename docs and scripts * Rename constants/filenames, goimports everything to re-order imports
2020-03-27Further room version wiring (#936)Neil Alexander
* Room version 2 by default, other wiring updates, update gomatrixserverlib * Fix nil pointer exception * Fix some more nil pointer exceptions hopefully * Update gomatrixserverlib * Send all room versions when joining, not just stable ones * Remove room version cquery * Get room version when getting events from the roomserver database * Reset default back to room version 2 * Don't generate event IDs unless needed * Revert "Remove room version cquery" This reverts commit a170d5873360dd059614460acc8b21ab2cda9767. * Query room version in federation API, client API as needed * Improvements to make_join send_join dance * Make room server producers use headered events * Lint tweaks * Update gomatrixserverlib * Versioned SendJoin * Query room version in syncapi backfill * Handle transaction marshalling/unmarshalling within Dendrite * Sorta fix federation (kinda) * whoops commit federation API too * Use NewEventFromTrustedJSON when getting events from the database * Update gomatrixserverlib * Strip headers on federationapi endpoints * Fix bug in clientapi profile room version query * Update gomatrixserverlib * Return more useful error if room version query doesn't find the room * Update gomatrixserverlib * Update gomatrixserverlib * Maybe fix federation * Fix formatting directive * Update sytest whitelist and blacklist * Temporarily disable room versions 3 and 4 until gmsl is fixed * Fix count of EDUs in logging * Update gomatrixserverlib * Update gomatrixserverlib * Update gomatrixserverlib * Rely on EventBuilder in gmsl to generate the event IDs for us * Some review comments fixed * Move function out of common and into gmsl * Comment in federationsender destinationqueue * Update gomatrixserverlib
2020-03-18maintenance: Fix matrix-org#896 use %w format verb to wrap errors (#916)Abhinav Krishna C K
* maintenance: Fix matrix-org#896 use %w format verb to wrap errors * In Go version 1.13 a new formatting verb introduced for fmt.Errorf %w https://blog.golang.org/go1.13-errors * update %s to %w to wrap errors. * Update all instances of error type to use %w Signed-off-by: Abhinav Krishna C K <me@abhy.me> Co-authored-by: Kegsay <kegan@matrix.org>
2020-03-18Log errors from rows.Close (#920)Prateek Sachan
* Log errors from rows.Close * fixed imports * Added contextual messages * fixed review changes
2020-03-16Implement gomatrixserverlib.HeaderedEvent in roomserver query API (#912)Neil Alexander
* Implement gomatrixserverlib.HeaderedEvent, which should allow us to store room version headers along with the event across API boundaries and consumers/producers, and intercept unmarshalling to get the event structure right * Add federationsender to previous
2020-03-09Improve logging when sending events (#883)Kegsay
We have some failing sytests on sqlite but it's very difficult to debug due to lack of useful logging. This adds a log line for when a new event is sent (incl. logging the event ID) as well as adding a user_id field for all contextual logs so we know who initiated certain actions.
2020-03-06Add peer-to-peer support into Dendrite via libp2p and fetch (#880)Kegsay
* Use a fork of pq which supports userCurrent on wasm * Use sqlite3_js driver when running in JS * Add cmd/dendritejs to pull in sqlite3_js driver for wasm only * Update to latest go-sqlite-js version * Replace prometheus with a stub. sigh * Hard-code a config and don't use opentracing * Latest go-sqlite3-js version * Generate a key for now * Listen for fetch traffic rather than HTTP * Latest hacks for js * libp2p support * More libp2p * Fork gjson to allow us to enforce auth checks as before Previously, all events would come down redacted because the hash checks would fail. They would fail because sjson.DeleteBytes didn't remove keys not used for hashing. This didn't work because of a build tag which included a file which no-oped the index returned. See https://github.com/tidwall/gjson/issues/157 When it's resolved, let's go back to mainline. * Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157 * Use latest gomatrixserverlib for sig checks * Fix a bug which could cause exclude_from_sync to not be set Caused when sending events over federation. * Use query variadic to make lookups actually work! * Latest gomatrixserverlib * Add notes on getting p2p up and running Partly so I don't forget myself! * refactor: Move p2p specific stuff to cmd/dendritejs This is important or else the normal build of dendrite will fail because the p2p libraries depend on syscall/js which doesn't work on normal builds. Also, clean up main.go to read a bit better. * Update ho-http-js-libp2p to return errors from RoundTrip * Add an LRU cache around the key DB We actually need this for P2P because otherwise we can *segfault* with things like: "runtime: unexpected return pc for runtime.handleEvent" where the event is a `syscall/js` event, caused by spamming sql.js caused by "Checking event signatures for 14 events of room state" which hammers the key DB repeatedly in quick succession. Using a cache fixes this, though the underlying cause is probably a bug in the version of Go I'm on (1.13.7) * breaking: Add Tracing.Enabled to toggle whether we do opentracing Defaults to false, which is why this is a breaking change. We need this flag because WASM builds cannot do opentracing. * Start adding conditional builds for wasm to handle lib/pq The general idea here is to have the wasm build have a `NewXXXDatabase` that doesn't import any postgres package and hence we never import `lib/pq`, which doesn't work under WASM (undefined `userCurrent`). * Remove lib/pq for wasm for syncapi * Add conditional building to remaining storage APIs * Update build script to set env vars correctly for dendritejs * sqlite bug fixes * Docs * Add a no-op main for dendritejs when not building under wasm * Use the real prometheus, even for WASM Instead, the dendrite-sw.js must mock out `process.pid` and `fs.stat` - which must invoke the callback with an error (e.g `EINVAL`) in order for it to work: ``` global.process = { pid: 1, }; global.fs.stat = function(path, cb) { cb({ code: "EINVAL", }); } ``` * Linting
2020-02-13Support sqlite in addition to postgres (#869)Kegsay
* Move current work into single branch * Initial massaging of clientapi etc (not working yet) * Interfaces for accounts/devices databases * Duplicate postgres package for sqlite3 (no changes made to it yet) * Some keydb, accountdb, devicedb, common partition fixes, some more syncapi tweaking * Fix accounts DB, device DB * Update naffka dependency for SQLite * Naffka SQLite * Update naffka to latest master * SQLite support for federationsender * Mostly not-bad support for SQLite in syncapi (although there are problems where lots of events get classed incorrectly as backward extremities, probably because of IN/ANY clauses that are badly supported) * Update Dockerfile -> Go 1.13.7, add build-base (as gcc and friends are needed for SQLite) * Implement GET endpoints for account_data in clientapi * Nuke filtering for now... * Revert "Implement GET endpoints for account_data in clientapi" This reverts commit 4d80dff4583d278620d9b3ed437e9fcd8d4674ee. * Implement GET endpoints for account_data in clientapi (#861) * Implement GET endpoints for account_data in clientapi * Fix accountDB parameter * Remove fmt.Println * Fix insertAccountData SQLite query * Fix accountDB storage interfaces * Add empty push rules into account data on account creation (#862) * Put SaveAccountData into the right function this time * Not sure if roomserver is better or worse now * sqlite work * Allow empty last sent ID for the first event * sqlite: room creation works * Support sending messages * Nuke fmt.println * Move QueryVariadic etc into common, other device fixes * Fix some linter issues * Fix bugs * Fix some linting errors * Fix errcheck lint errors * Make naffka use postgres as fallback, fix couple of compile errors * What on earth happened to the /rooms/{roomID}/send/{eventType} routing Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-02-11Always defer *sql.Rows.Close and consult with Err (#844)Kiril Vladimiroff
* Always defer *sql.Rows.Close and consult with Err database/sql.Rows.Next() makes sure to call Close only after exhausting result rows which would NOT happen when returning early from a bad Scan. Close being idempotent makes it a great candidate to get always deferred regardless of what happens later on the result set. This change also makes sure call Err() after exhausting Next() and propagate non-nil results from it as the documentation advises. Closes #764 Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org> * Override named result parameters in last returns Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org> * Do the same over new changes that got merged Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-02-11Fix AppService bind addrs in test (#805)aditsachde
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-02-11Pass cfg by reference around the codebase (#819)Behouba Manassé
* Pass cfg by reference around the codebase * Merge branch 'master' into pass-cfg-by-ref Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-01-25Store our own keys in the keydb (#853)Neil Alexander
* Store our own keys in the keydb The DirectKeyFetcher makes the assumption that you can always reach the key/v2/server endpoint of any server, including our own. We previously haven't bothered to store our own keys in the keydb so this would mean we end up making key requests to ourselves. In the libp2p world as an example, self-dialling is not possible, therefore this would render it impossible to get our own keys. This commit adds our own keys into the keydb so that we don't create unnecessarily (and maybe impossible) requests. * Use golang.org/x/crypto/ed25519 instead of crypto/ed25519 for pre-Go 1.13
2020-01-09Fix #842 - again...Neil Alexander
2020-01-09Fall back to postgres when database connection string parsing fails (#842)Neil Alexander
* Fall back to postgres when parsing the database connection string for a URI schema fails * Fix behaviour so that it really tries postgres when URL parsing fails and it complains about unknown schema if it succeeds
2020-01-03Implement storage interfaces (#841)Neil Alexander
* Implement interfaces for federationsender storage * Implement interfaces for mediaapi storage * Implement interfaces for publicroomsapi storage * Implement interfaces for roomserver storage * Implement interfaces for syncapi storage * Implement interfaces for keydb storage * common.PartitionStorer in publicroomsapi interface * Update copyright notices
2019-12-17Replace deprecated prometheus.InstrumentHandler and unsafe time.TickerAndrew Morgan
2019-10-04Add AppService to test config and bind addrs (#800)Andrew Morgan
Because our unit tests are often forgotten about :/
2019-10-02patch dendrite microservices with bind config (#795)aditsachde
This PR adds a block in the dendrite config for the services to bind to. The microservices should bind to the addresses in the bind block, and will be contacted at the address in the listen block. This fixes an issue with the microservices and kubernetes services.
2019-10-02Add missing servers field in /directory/room/:alias response (#732)Alex Chen
2019-08-16Error when recaptcha enabled with empty configs (#786)Parminder Singh
2019-08-16Replace event content types with ones in gomatrixserverlib (#785)Alex Chen
The types that are not in gomatrixserverlib are not replaced. This also updates the gomatrixserverlib dependency.
2019-08-15Add auth fallback endpoint (#405)Parminder Singh
Also adds support for the recaptcha auth type.
2019-08-07Implement profile retrieval over federation (#726)Alex Chen
2019-08-06Fix transaction IDs in transaction cache have global scope (#772)Alex Chen
2019-07-24Implement room creation content (#754)Alex Chen
Fixes #660. Signed-off-by: Alex Chen minecnly@gmail.com
2019-07-24Add filepath and function name to log output (#755)Andrew Morgan
Adds detailed logging, describing which file/line a log message came from, as well as the name of the function that it was contained within.
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-18 11:11:19 +0000