aboutsummaryrefslogtreecommitdiff
path: root/appservice
AgeCommit message (Collapse)Author
2023-05-02Use PDU not *Event in HeaderedEvent (#3073)kegsay
Requires https://github.com/matrix-org/gomatrixserverlib/pull/376 This has numerous upsides: - Less type casting to `*Event` is required. - Making Dendrite work with `PDU` interfaces means we can swap out Event impls more easily. - Tests which represent weird event shapes are easier to write. Part of a series of refactors on GMSL.
2023-04-28Use PDU in more places (#3072)kegsay
2023-04-27Remove gmsl.HeaderedEvent (#3068)kegsay
Replaced with types.HeaderedEvent _for now_. In reality we want to move them all to gmsl.Event and only use HeaderedEvent when we _need_ to bundle the version/event ID with the event (seriailsation boundaries, and even then only when we don't have the room version). Requires https://github.com/matrix-org/gomatrixserverlib/pull/373
2023-04-19refactor: update GMSL (#3058)kegsay
Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/364 Read this commit by commit to avoid going insane.
2023-04-04Move GMSL client types to Dendrite (#3045)kegsay
GMSL is intended for Federation only. Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/357
2023-04-04app service unix socket support (#3022)Boris Rybalkin
This is the last part of unix socket support to talk to app servers, go based app services already support unix sockets: https://github.com/mautrix/go/commit/5a68173fe39345b8473e04bfa67cae5a13f6ca7f ``` appservice: # The address that the homeserver can use to connect to this appservice. address: unix:///var/snap/matrix/current/whatsapp.socket # The hostname and port where this appservice should listen. hostname: /var/snap/matrix/current/whatsapp.socket port: 0 ``` ### Pull Request Checklist <!-- Please read https://matrix-org.github.io/dendrite/development/contributing before submitting your pull request --> * [x] I have added Go unit tests or [Complement integration tests](https://github.com/matrix-org/complement) for this PR _or_ I have justified why this PR doesn't need tests * [x] Pull request includes a [sign off below using a legally identifiable name](https://matrix-org.github.io/dendrite/development/contributing#sign-off) _or_ I have already signed off privately Signed-off-by: `Boris Rybalkin <ribalkin@gmail.com>`
2023-04-03Add user profile tests, refactor user API methods (#3030)Till
This adds tests for `/profile`. Also, as a first change in this regard, refactors the methods defined on the `UserInternalAPI` to not use structs as the request/response parameters.
2023-03-27Add AS invite test, fix issue with invitations being processed twice (#3020)Till
The AS roomserver consumer would receive the events twice, one time as type `OutputTypeNewInviteEvent` and the other time as `OutputTypeNewRoomEvent`. [skip ci]
2023-03-22Remove `BaseDendrite` (#3023)Till
Removes `BaseDendrite` to, hopefully, make testing and composing of components easier in the future.
2023-03-17Preparations for removing `BaseDendrite` (#3016)Till
Preparations to actually remove/replace `BaseDendrite`. Quite a few changes: - SyncAPI accepts an `fulltext.Indexer` interface (fulltext is removed from `BaseDendrite`) - Caches are removed from `BaseDendrite` - Introduces a `Router` struct (likely to change) - also fixes #2903 - Introduces a `sqlutil.ConnectionManager`, which should remove `base.DatabaseConnection` later on - probably more
2023-03-13Add basic runtime tracing (#2996)Till
This allows us in almost all places to use regions to further trace down long running tasks. Also removes an unused function.
2023-03-01Refactor `StoreEvent`, add `MaybeRedactEvent`, create an `EventDatabase` (#2989)Till
This PR changes the following: - `StoreEvent` now only stores an event (and possibly prev event), instead of also doing redactions - Adds a `MaybeRedactEvent` (pulled out from `StoreEvent`), which should be called after storing events - a few other things
2023-02-20Merge keyserver & userapi (#2972)Till
As discussed yesterday, a first draft of merging the keyserver and the userapi.
2023-02-14Remove polylith/API mode (#2967)Till
This removes most of the code used for polylith/API mode. This removes the `/api` internal endpoints entirely. Binary size change roughly 5%: ``` 51437560 Feb 13 10:15 dendrite-monolith-server # old 48759008 Feb 13 10:15 dendrite-monolith-server # new ```
2022-12-05Use test.WithAllDatabasesTill Faelligen
2022-12-05Fix CI and test.WithAllDatabasesTill Faelligen
2022-12-05Enable/Disable internal metrics (#2899)Till
Basically enables us to use `test.WithAllDatabases` when testing internal HTTP APIs, as this would otherwise result in Prometheus complaining about already registered metric names.
2022-12-02Add tests for the AS internal API (#2898)Till
2022-11-11Virtual hosting schema and logic changes (#2876)Neil Alexander
Note that virtual users cannot federate correctly yet.
2022-11-02Implement `/thirdparty` endpoints (#2831)Till
Implements the following endpoints ``` GET /_matrix/client/v3/thirdparty/protocols GET /_matrix/client/v3/thirdparty/protocols/{protocol} GET /_matrix/client/v3/thirdparty/location GET /_matrix/client/v3/thirdparty/location/{protocol} GET /_matrix/client/v3/thirdparty/user GET /_matrix/client/v3/thirdparty/user/{protocol} ```
2022-10-05Add `RoomEventType` nats.Header to avoid unneeded unmarshalling (#2765)Till
2022-09-19Bug fix #2718 appservice txnid should be different for each batch of events ↵Tak Wai Wong
(#2719) See issue: [#2718](https://github.com/matrix-org/dendrite/issues/2718) for more details. The fix assumes that if the number of transaction items are different, then the txnid should be different. txnid := OriginalServerTS()_len(transactions) The case that it doesn't address is if the txnid generated this way is the same for 2 different batches of events which have the same OriginalServerTS and the same array length. Another option: txnid := OriginalServerTS()_hash(transactions) Would love to hear other ideas and ways to fix this. ### Pull Request Checklist * [x ] I have added added tests for PR _or_ I have justified why this PR doesn't need tests. * [x ] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Tak Wai Wong <tak@hntlabs.com>` Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-09-02Only send (invite) events to the AS if it's interested in it (#2692)Till
This should hopefully fix #2691 (and maybe also #2690)
2022-09-01Refactor appservices component (#2687)Neil Alexander
This PR refactors the app services component. It makes the following changes: * Each appservice now gets its own NATS JetStream consumer * The appservice database is now removed entirely, since we just use JetStream as a data source instead * The entire component is now much simpler and we deleted lots of lines of code 💅 The result is that it should be much lighter and hopefully much more performant.
2022-08-31Allow batching in `JetStreamConsumer` (#2686)Neil Alexander
This allows us to receive more than one message from NATS at a time if we want.
2022-08-29Log roomserver errorTill Faelligen
2022-08-11Generic-based internal HTTP API (#2626)Neil Alexander
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API) * Add `PerformInvite` * More tweaks * Fix metric name * Fix LookupStateIDs * Lots of changes to clients * Some serverside stuff * Some error handling * Use paths as metric names * Revert "Use paths as metric names" This reverts commit a9323a6a343f5ce6461a2e5bd570fe06465f1b15. * Namespace metric names * Remove duplicate entry * Remove another duplicate entry * Tweak error handling * Some more tweaks * Update error behaviour * Some more error tweaking * Fix API path for `PerformDeleteKeys` * Fix another path * Tweak federation client proxying * Fix another path * Don't return typed nils * Some more tweaks, not that it makes any difference * Tweak federation client proxying * Maybe fix the key backup test
2022-05-24Handle invite in appservice roomserver consumer (#2476)Aaron Ogle
Signed-off-by: Aaron Ogle <aaron@geekgonecrazy.com>
2022-05-10Back out matrix-org/dendrite#2421 by restoring `http.Client`sNeil Alexander
This creates problems with non-HTTPS endpoints and should fix #2444.
2022-05-09One NATS instance per `BaseDendrite` (#2438)Neil Alexander
* One NATS instance per `BaseDendrite` * Fix roomserver
2022-05-06Clean up interface definitions (#2427)kegsay
* tidy up interfaces * remove unused GetCreatorIDForAlias * Add RoomserverUserAPI interface * Define more interfaces * Use AppServiceInternalAPI for consistent naming * clean up federationapi constructor a bit * Fix monolith in -http mode
2022-05-05Define component interfaces based on consumers (1/2) (#2423)kegsay
* Specify interfaces used by appservice, do half of clientapi * convert more deps of clientapi to finer-grained interfaces * Convert mediaapi and rest of clientapi * Somehow this got missed
2022-05-05Use `gomatrixserverlib.Client` instead of `http.Client` (#2421)Neil Alexander
* Update to matrix-org/gomatrixserverlib#303 * Use `gomatrixserverlib.Client` for phone-home stats * Use `gomatrixserverlib.Client` for push notifications * Use `gomatrixserverlib.Client` for appservices * Use `gomatrixserverlib.Client` for three-PID invites
2022-05-03Global database connection pool (for monolith mode) (#2411)Neil Alexander
* Allow monolith components to share a single database pool * Don't yell about missing connection strings * Rename field * Setup tweaks * Fix panic * Improve configuration checks * Update config * Fix lint errors * Update comments
2022-04-06Fix bug where state events could be requested twice in appservices (#2328)Neil Alexander
* Fix bug where state events could be requested twice (should fix #2305) * Avoid a pointless HTTP request if possible
2022-03-24Refactor appservice & client API to use userapi internal (#2290)S7evinK
* Refactor user api internal * Refactor clientapi to use internal userapi * Use internal userapi instead of user DB directly * Remove AccountDB dependency * Fix linter issues Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-23Per-room consumers (#2293)Neil Alexander
* Roomserver input refactoring — again! * Ensure the actor runs again * Preserve consumer after unsubscribe * Another sprinkling of magic * Rename `TopicFor` to `Prefixed` * Recreate the stream if the config is bad * Check streams too * Prefix subjects, preserve inboxes * Recreate if subjects wrong * Remove stream subject * Reconstruct properly * Fix mutex unlock * Comments * Fix tests * Don't drop events * Review comments * Separate `queueInputRoomEvents` function * Re-jig control flow a bit
2022-03-21Wait for safe shutdown of NATS Server (#2289)Neil Alexander
2022-03-21Remove unused partition_offset_table (#2288)S7evinK
2022-03-16Stream tweaks, use same codepath for sync vs async input room events, wait ↵Neil Alexander
for error response via NATS messages (#2283)
2022-03-07Don't send `adds_state_events` in roomserver output events anymore (#2258)Neil Alexander
* Don't send `adds_state_events` in roomserver output events anymore * Set `omitempty` on some output fields that aren't always set * Add `AddsState` helper function * No-op if no added state event IDs * Revert "No-op if no added state event IDs" This reverts commit 71a0ef3df10e0d94234d916246c30b0a4e82b26e. * Revert "Add `AddsState` helper function" This reverts commit c9fbe45475eb12ae44d2a8da7c0fc3a002ad9819.
2022-02-18Merge both user API databases into one (#2186)Neil Alexander
* Merge user API databases into one * Remove DeviceDatabase from config * Fix tests * Try that again * Clean up keyserver device keys when the devices no longer exist in the user API * Tweak ordering * Fix UserExists flag, device check * Allow including empty entries so we can clean them up * Remove logging
2022-02-16Add account type (#2171)S7evinK
* Add account_type for sqlite3 * Add account_type for postgres * Remove CreateGuestAccount from interface * Add new AccountTypes & update test * Use newly added AccountType for account creation * Add migrations * Reuse type * Add AccounnType to Device, so it can be verified on requests * Rename migration, add missing update for appservices * Rename sqlite3 migration * Add missing AccountType to return value * Update sqlite migration Change allowance check on /admin/whois * Fix migration, add IS NULL * Move accountType to completeRegistration * Fix migrations * Add passing test
2022-02-04Remove sarama/saramajetstream dependencies (#2138)S7evinK
* Remove dependency on saramajetstream & sarama Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Remove internal.ContinualConsumer from federationapi * Remove internal.ContinualConsumer from syncapi * Remove internal.ContinualConsumer from keyserver * Move to new Prepare function * Remove saramajetstream & sarama dependency * Delete unneeded file * Remove duplicate import * Log error instead of silently irgnoring it * Move `OffsetNewest` and `OffsetOldest` into keyserver types, change them to be more sane values * Fix comments Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-02Use pull consumers (#2140)Neil Alexander
* Pull consumers * Pull consumers * Only nuke consumers if they are push consumers * Clean up old consumers * Better error handling * Update comments
2022-01-07NATS JetStream tweaks (#2086)Neil Alexander
* Use named NATS durable consumers * Build fixes * Remove dupe call to SetFederationAPI * Use namespaced consumer name * Fix namespacing * Fix unit tests hopefully
2022-01-05Add NATS JetStream support (#1866)S7evinK
* Add NATS JetStream support Update shopify/sarama * Fix addresses * Don't change Addresses in Defaults * Update saramajetstream * Add missing error check Keep typing events for at least one minute * Use all configured NATS addresses * Update saramajetstream * Try setting up with NATS * Make sure NATS uses own persistent directory (TODO: make this configurable) * Update go.mod/go.sum * Jetstream package * Various other refactoring * Build fixes * Config tweaks, make random jetstream storage path for CI * Disable interest policies * Try to sane default on jetstream base path * Try to use in-memory for CI * Restore storage/retention * Update nats.go dependency * Adapt changes to config * Remove unneeded TopicFor * Dep update * Revert "Remove unneeded TopicFor" This reverts commit f5a4e4a339b6f94ec215778dca22204adaa893d1. * Revert changes made to streams * Fix build problems * Update nats-server * Update go.mod/go.sum * Roomserver input API queuing using NATS * Fix topic naming * Prometheus metrics * More refactoring to remove saramajetstream * Add missing topic * Don't try to populate map that doesn't exist * Roomserver output topic * Update go.mod/go.sum * Message acknowledgements * Ack tweaks * Try to resume transaction re-sends * Try to resume transaction re-sends * Update to matrix-org/gomatrixserverlib@91dadfb * Remove internal.PartitionStorer from components that don't consume keychanges * Try to reduce re-allocations a bit in resolveConflictsV2 * Tweak delivery options on RS input * Publish send-to-device messages into correct JetStream subject * Async and sync roomserver input * Update dendrite-config.yaml * Remove roomserver tests for now (they need rewriting) * Remove roomserver test again (was merged back in) * Update documentation * Docker updates * More Docker updates * Update Docker readme again * Fix lint issues * Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset) * Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that * Go 1.16 instead of Go 1.13 for upgrade tests and Complement * Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that" This reverts commit 368675283fc44501f227639811bdb16dd5deef8c. * Don't report any errors on `/send` to see what fun that creates * Fix panics on closed channel sends * Enforce state key matches sender * Do the same for leave * Various tweaks to make tests happier Squashed commit of the following: commit 13f9028e7a63662759ce7c55504a9d2423058668 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 15:47:14 2022 +0000 Do the same for leave commit e6be7f05c349fafbdddfe818337a17a60c867be1 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 15:33:42 2022 +0000 Enforce state key matches sender commit 85ede6d64bf10ce9b91cdd6d80f87350ee55242f Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 14:07:04 2022 +0000 Fix panics on closed channel sends commit 9755494a98bed62450f8001d8128e40481d27e15 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 13:38:22 2022 +0000 Don't report any errors on `/send` to see what fun that creates commit 3bb4f87b5dd56882febb4db5621db484c8789b7c Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 13:00:26 2022 +0000 Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that" This reverts commit 368675283fc44501f227639811bdb16dd5deef8c. commit fe2673ed7be9559eaca134424e403a4faca100b0 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 12:09:34 2022 +0000 Go 1.16 instead of Go 1.13 for upgrade tests and Complement commit 368675283fc44501f227639811bdb16dd5deef8c Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 11:51:45 2022 +0000 Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that commit b028dfc08577bcf52e6cb498026e15fa5d46d07c Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 10:29:08 2022 +0000 Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset) * Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork * Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers * Fix consumer component name in federation API * Add comment explaining where streams are defined * Tweaks to roomserver input with comments * Finish that sentence that I apparently forgot to finish in INSTALL.md * Bump version number of config to 2 * Add comments around asynchronous sends to roomserver in processEventWithMissingState * More useful error message when the config version does not match * Set version in generate-config * Fix version in config.Defaults Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-12-03Add missing HTTP mode for userapi (#1982)S7evinK
* Add missing internal api endpoint Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Add missing performKeyBackup endpoint * Add missing http mode for userapi * Fix failing tests * Add error checks * Fix sytest * Update startup logic for HTTP mode * Use userImpl for AS (annoying) * Don't send device list updates for appservice devices * Fix build Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-11-24Merge `federationapi`, `federationsender`, `signingkeyserver` components (#2055)Neil Alexander
* Initial federation sender -> federation API refactoring * Move base into own package, avoids import cycle * Fix build errors * Fix tests * Add signing key server tables * Try to fold signing key server into federation API * Fix dendritejs builds * Update embedded interfaces * Fix panic, fix lint error * Update configs, docker * Rename some things * Reuse same keyring on the implementing side * Fix federation tests, `NewBaseDendrite` can accept freeform options * Fix build * Update create_db, configs * Name tables back * Don't rename federationsender consumer for now
2021-11-02Run gofmt on dendrite - apply go 1.17 preferred build tags (#2021)PiotrKozimor
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-19 09:08:21 +0000