aboutsummaryrefslogtreecommitdiff
path: root/cmd/dendrite-demo-pinecone/main.go
AgeCommit message (Collapse)Author
2023-01-23Initial Store & Forward Implementation (#2917)devonh
This adds store & forward relays into dendrite for p2p. A few things have changed: - new relay api serves new http endpoints for s&f federation - updated outbound federation queueing which will attempt to forward using s&f if appropriate - database entries to track s&f relays for other nodes
2022-12-12Cleanup stale device lists for users we don't share a room with anymore (#2857)Till
The stale device lists table might contain entries for users we don't share a room with anymore. This now asks the roomserver about left users and removes those entries from the table. Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-12-01Fix `dendrite-demo-pinecone`, `/_dendrite` namespace setupNeil Alexander
2022-12-01Expose `/_dendrite` and `/_synapse` on the P2P demo HTTP muxesNeil Alexander
2022-11-18Add p2p wakeup broadcast handling to pinecone demos (#2841)devonh
Adds wakeup broadcast handling to the pinecone demos. This will reset their blacklist status and interrupt any ongoing federation queue backoffs currently in progress for this peer. The end result is that any queued events will quickly be sent to the peer if they had disconnected while attempting to send events to them.
2022-11-02Fix `moderncsqlite` errors and rebase onto `main` (#2832)0x1a8510f2
This is #2819 but rebased on latest `main`. This PR is against main too as opposed to the `moderncsqlite` branch. The main change here is simply: ```go // add query parameters to the dsn if strings.Contains(dsn, "?") { dsn += "&" } else { dsn += "?" } // wait some time before erroring if the db is locked // https://gitlab.com/cznic/sqlite/-/issues/106#note_1058094993 dsn += "_pragma=busy_timeout%3d10000" ``` ### Pull Request Checklist <!-- Please read https://matrix-org.github.io/dendrite/development/contributing before submitting your pull request --> * [x] I have added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off below using a legally identifiable name](https://matrix-org.github.io/dendrite/development/contributing#sign-off) _or_ I have already signed off privately Signed off privately. Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-03Update P2P base directoriesNeil Alexander
2022-10-03Hopefully fix P2P `--config` error (re. #2756)Neil Alexander
2022-09-23Add `-dir` option to `dendrite-demo-pinecone` and `dendrite-demo-yggdrasil`Neil Alexander
2022-09-14Fix Pinecone demo build errors after Pinecone updateNeil Alexander
2022-09-12P2P demo tweaksNeil Alexander
2022-09-01Configuration tweaks (#2567)Neil Alexander
This makes the following changes: * The various `Defaults` functions are now responsible for setting sane defaults if `generate` is specified, rather than hiding them in `generate-config` * Some configuration options have been marked as `omitempty` so that they don't appear in generated configs unnecessarily (monolith-specific vs. polylith-specific options) * A new option `-polylith` has been added to `generate-config` to create a config that makes sense for polylith deployments (i.e. including the internal/external API listeners and per-component database sections) * A new option `-normalise` has been added to `generate-config` to take an existing file and add any missing options and/or defaults
2022-09-01Refactor appservices component (#2687)Neil Alexander
This PR refactors the app services component. It makes the following changes: * Each appservice now gets its own NATS JetStream consumer * The appservice database is now removed entirely, since we just use JetStream as a data source instead * The entire component is now much simpler and we deleted lots of lines of code 💅 The result is that it should be much lighter and hopefully much more performant.
2022-08-31Pinecone standalone refactoring (#2685)Neil Alexander
This refactors the `dendrite-demo-pinecone` executable so that it: 1. Converts the old `.key` file into a standard `.pem` file 2. Allows passing in the `--config` option to supply a normal Dendrite configuration file, so that you can configure PostgreSQL instead of SQLite, appservices and all the other usual stuff
2022-08-05Do not use `ioutil` as it is deprecated (#2625)Neil Alexander
2022-07-08Update Pinecone to matrix-org/pinecone@1ce778fNeil Alexander
2022-05-03Tidy up `AddPublicRoutes` (#2412)Neil Alexander
* Simplify federation API `AddPublicRoutes` * Simplify client API `AddPublicRoutes` * Simplify media API `AddPublicRoutes` * Simplify sync API `AddPublicRoutes` * Simplify `AddAllPublicRoutes`
2022-05-03Global database connection pool (for monolith mode) (#2411)Neil Alexander
* Allow monolith components to share a single database pool * Don't yell about missing connection strings * Rename field * Setup tweaks * Fix panic * Improve configuration checks * Update config * Fix lint errors * Update comments
2022-04-29Add restrictions for open registration (#2402)Till
* Add restrications for open registration * Make enable open registration a parameter * Enable registration for CI * Update error message * Shuffle things around a bit * Add a warning at every startup just to be extra annoying * Ignore shared secret when warning about open registration, since it's not strictly required when it is set if registration is otherwise enabled * Make CI happy? * Add missing parameter; try new parameter in upgrade-test Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-28Add `/_dendrite/admin/evacuateRoom/{roomID}` (#2401)Neil Alexander
* Add new endpoint to allow admins to evacuate the local server from the room * Guard endpoint * Use right prefix * Auth API * More useful return error rather than a panic * More useful return value again * Update the path * Try using inputer instead * oh provide the config * Try that again * Return affected user IDs * Don't create so many forward extremities * Add missing `Path` to name Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2022-04-08Use connection manager in Pinecone demosNeil Alexander
2022-04-04Enhanced calm at startup of Pinecone demosNeil Alexander
2022-04-04Update to matrix-org/pinecone@e526fa8Neil Alexander
2022-03-30Update Pinecone P2P demoNeil Alexander
2022-03-29Remove eduserver (#2306)S7evinK
* Move receipt sending to own JetStream producer * Move SendToDevice to producer * Remove most parts of the EDU server * Fix SendToDevice & copyrights * Move structs, cleanup EDU Server traces * Use HeadersOnly subscription * Missing file * Fix linter issues * Move consumers to own files * Rename durable consumer; Consumer cleanup * Docs/config cleanup
2022-03-28User directory for nearby Pinecone peers (P2P demo) (#2311)Neil Alexander
* User directory for nearby Pinecone peers * Fix mux routing * Use config to determine which server notices user to exclude
2022-03-03Implement Push Notifications (#1842)Dan
* Add Pushserver component with Pushers API Co-authored-by: Tommie Gannert <tommie@gannert.se> Co-authored-by: Dan Peleg <dan@globekeeper.com> * Wire Pushserver component Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> * Add PushGatewayClient. The full event format is required for Sytest. * Add a pushrules module. * Change user API account creation to use the new pushrules module's defaults. Introduces "scope" as required by client API, and some small field tweaks to make some 61push Sytests pass. * Add push rules query/put API in Pushserver. This manipulates account data over User API, and fires sync messages for changes. Those sync messages should, according to an existing TODO in clientapi, be moved to userapi. Forks clientapi/producers/syncapi.go to pushserver/ for later extension. * Add clientapi routes for push rules to Pushserver. A cleanup would be to move more of the name-splitting logic into pushrules.go, to depollute routing.go. * Output rooms.join.unread_notifications in /sync. This is the read-side. Pushserver will be the write-side. * Implement pushserver/storage for notifications. * Use PushGatewayClient and the pushrules module in Pushserver's room consumer. * Use one goroutine per user to avoid locking up the entire server for one bad push gateway. * Split pushing by format. * Send one device per push. Sytest does not support coalescing multiple devices into one push. Matches Synapse. Either we change Sytest, or remove the group-by-url-and-format logic. * Write OutputNotificationData from push server. Sync API is already the consumer. * Implement read receipt consumers in Pushserver. Supports m.read and m.fully_read receipts. * Add clientapi route for /unstable/notifications. * Rename to UpsertPusher for clarity and handle pusher update * Fix linter errors * Ignore body.Close() error check * Fix push server internal http wiring * Add 40 newly passing 61push tests to whitelist * Add next 12 newly passing 61push tests to whitelist * Send notification data before notifying users in EDU server consumer * NATS JetStream * Goodbye sarama * Fix `NewStreamTokenFromString` * Consume on the correct topic for the roomserver * Don't panic, NAK instead * Move push notifications into the User API * Don't set null values since that apparently causes Element upsetti * Also set omitempty on conditions * Fix bug so that we don't override the push rules unnecessarily * Tweak defaults * Update defaults * More tweaks * Move `/notifications` onto `r0`/`v3` mux * User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly Co-authored-by: Piotr Kozimor <p1996k@gmail.com> Co-authored-by: Tommie Gannert <tommie@gannert.se> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-18Merge both user API databases into one (#2186)Neil Alexander
* Merge user API databases into one * Remove DeviceDatabase from config * Fix tests * Try that again * Clean up keyserver device keys when the devices no longer exist in the user API * Tweak ordering * Fix UserExists flag, device check * Allow including empty entries so we can clean them up * Remove logging
2022-01-05Add NATS JetStream support (#1866)S7evinK
* Add NATS JetStream support Update shopify/sarama * Fix addresses * Don't change Addresses in Defaults * Update saramajetstream * Add missing error check Keep typing events for at least one minute * Use all configured NATS addresses * Update saramajetstream * Try setting up with NATS * Make sure NATS uses own persistent directory (TODO: make this configurable) * Update go.mod/go.sum * Jetstream package * Various other refactoring * Build fixes * Config tweaks, make random jetstream storage path for CI * Disable interest policies * Try to sane default on jetstream base path * Try to use in-memory for CI * Restore storage/retention * Update nats.go dependency * Adapt changes to config * Remove unneeded TopicFor * Dep update * Revert "Remove unneeded TopicFor" This reverts commit f5a4e4a339b6f94ec215778dca22204adaa893d1. * Revert changes made to streams * Fix build problems * Update nats-server * Update go.mod/go.sum * Roomserver input API queuing using NATS * Fix topic naming * Prometheus metrics * More refactoring to remove saramajetstream * Add missing topic * Don't try to populate map that doesn't exist * Roomserver output topic * Update go.mod/go.sum * Message acknowledgements * Ack tweaks * Try to resume transaction re-sends * Try to resume transaction re-sends * Update to matrix-org/gomatrixserverlib@91dadfb * Remove internal.PartitionStorer from components that don't consume keychanges * Try to reduce re-allocations a bit in resolveConflictsV2 * Tweak delivery options on RS input * Publish send-to-device messages into correct JetStream subject * Async and sync roomserver input * Update dendrite-config.yaml * Remove roomserver tests for now (they need rewriting) * Remove roomserver test again (was merged back in) * Update documentation * Docker updates * More Docker updates * Update Docker readme again * Fix lint issues * Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset) * Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that * Go 1.16 instead of Go 1.13 for upgrade tests and Complement * Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that" This reverts commit 368675283fc44501f227639811bdb16dd5deef8c. * Don't report any errors on `/send` to see what fun that creates * Fix panics on closed channel sends * Enforce state key matches sender * Do the same for leave * Various tweaks to make tests happier Squashed commit of the following: commit 13f9028e7a63662759ce7c55504a9d2423058668 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 15:47:14 2022 +0000 Do the same for leave commit e6be7f05c349fafbdddfe818337a17a60c867be1 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 15:33:42 2022 +0000 Enforce state key matches sender commit 85ede6d64bf10ce9b91cdd6d80f87350ee55242f Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 14:07:04 2022 +0000 Fix panics on closed channel sends commit 9755494a98bed62450f8001d8128e40481d27e15 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 13:38:22 2022 +0000 Don't report any errors on `/send` to see what fun that creates commit 3bb4f87b5dd56882febb4db5621db484c8789b7c Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 13:00:26 2022 +0000 Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that" This reverts commit 368675283fc44501f227639811bdb16dd5deef8c. commit fe2673ed7be9559eaca134424e403a4faca100b0 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 12:09:34 2022 +0000 Go 1.16 instead of Go 1.13 for upgrade tests and Complement commit 368675283fc44501f227639811bdb16dd5deef8c Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 11:51:45 2022 +0000 Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that commit b028dfc08577bcf52e6cb498026e15fa5d46d07c Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 10:29:08 2022 +0000 Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset) * Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork * Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers * Fix consumer component name in federation API * Add comment explaining where streams are defined * Tweaks to roomserver input with comments * Finish that sentence that I apparently forgot to finish in INSTALL.md * Bump version number of config to 2 * Add comments around asynchronous sends to roomserver in processEventWithMissingState * More useful error message when the config version does not match * Set version in generate-config * Fix version in config.Defaults Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-12-13Fix keyring regressions in previous P2P demoNeil Alexander
2021-11-25Fixes for multiple Pinecone peersNeil Alexander
2021-11-25Support connecting to multiple Pinecone static peers in the P2P demos ↵Neil Alexander
(supply a comma-separated list)
2021-11-24Don't populate config defaults where it doesn't make sense (#2058)Neil Alexander
* Don't populate config defaults where it doesn't make sense * Fix dendritejs builds
2021-11-24Merge `federationapi`, `federationsender`, `signingkeyserver` components (#2055)Neil Alexander
* Initial federation sender -> federation API refactoring * Move base into own package, avoids import cycle * Fix build errors * Fix tests * Add signing key server tables * Try to fold signing key server into federation API * Fix dendritejs builds * Update embedded interfaces * Fix panic, fix lint error * Update configs, docker * Rename some things * Reuse same keyring on the implementing side * Fix federation tests, `NewBaseDendrite` can accept freeform options * Fix build * Update create_db, configs * Name tables back * Don't rename federationsender consumer for now
2021-10-22Update Pinecone demoNeil Alexander
2021-10-08Update Pinecone demoNeil Alexander
2021-09-10Added .well-known/matrix/server endpoint (#1988)Ryan W
* Added .well-known/matrix/server endpoint Signed-off-by: Ryan Whittington <twentybitdev@gmail.com> * Replaced tabs with spaces Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>
2021-09-08- Removed double imports (#1989)Ryan W
- Lower cased error messages Signed-off-by: Ryan Whittington <twentybitdev@gmail.com> Co-authored-by: kegsay <kegan@matrix.org>
2021-08-04Cross-signing groundwork (#1953)Neil Alexander
* Cross-signing groundwork * Update to matrix-org/gomatrixserverlib#274 * Fix gobind builds, which stops unit tests in CI from yelling * Some changes from review comments * Fix build by passing in UIA * Update to matrix-org/gomatrixserverlib@bec8d22 * Process master/self-signing keys from devices call * nolint * Enum-ify the key type in the database * Process self-signing key too * Fix sanity check in device list updater * Fix check * Fix sytest, hopefully * Fix build
2021-07-20Only include go-sqlite3 on the relevant binaries (#1900)Neil Alexander
* Only include go-sqlite3 on the relevant binaries * The driver name is always sqlite3 now * Update to matrix-org/go-sqlite3-js@e537baa
2021-07-09Implement /_synapse/admin/v1/register (#1911)kegsay
* Implement /_synapse/admin/v1/register This is implemented identically to Synapse, so scripts which work with Synapse should work with Dendrite. ``` Test 27 POST /_synapse/admin/v1/register with shared secret... OK Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK ``` Sytest however has `implementation_specific => "synapse"` which stops these tests from running. * Add missing muxes to gobind * Linting
2021-06-22dendrite-demo-pinecone: Accept any originNeil Alexander
2021-06-14Pinecone demo updatesNeil Alexander
2021-05-24Allow clearing federation blacklist at startup for P2P demosNeil Alexander
2021-05-18Update dendrite-demo-pineconeNeil Alexander
2021-05-06Pinecone P2P demo (#1856)Neil Alexander
* Pinecone demo * Enable multicast, fix HTTP routing * Fix multicast import * Fix build * Update Pinecone demo * Fix the keys * Tweaks * Pinecone room directory support (early) * Fix gobind-pinecone * Add pinecone listener * Fix public key value * Use AuthenticatedConnect for dial * Fix gobind-pinecone * Stop panics * Give fsAPI to keyserver * Pinecone demo fixes * Update gobind build scripts * Account creation * Tweaks * Setup tweaks * API tweaks * API tweaks * API tweaks * Port mutex * Re-enable multicast * Add ReadCopy * Update quic-go, fixes * Shutdowns fixed for iOS * Update build script * Add WebSocket support * Bug fixes * Netconn context * Fix WebSocket connectivity * Fixes to gobind API * Strip frameworks * Configurability updates * Update go.mod * Update go.mod/go.sum * Update go.mod/go.sum * Update go.mod/go.sum * Try to stay connected tto static peer * Update gobind-pinecone * Update go.mod/go.sum * Test uTP+TLS * Use HTTP/2 * Don't use HTTP/2 * Update go.mod/go.sum * Attempt to reconnect to the static peer if it drops * Stay connected to static peers more stickily * Retry room directory lookups if they fail * NewQUIC -> NewSessions * Storage updates * Don't return immediately when there's nothing to sync * Updates * Try to reconnect to static peer more * Update go.mod/go.sum * Require Go 1.14 * Update go.mod/go.sum * Update go.mod/go.sum
generated by cgit v1.2.3 (git 2.26.2) at 2024-11-28 18:58:20 +0000