aboutsummaryrefslogtreecommitdiff
path: root/roomserver/internal/input
AgeCommit message (Collapse)Author
2022-01-31Tweak some logging (#2130)Neil Alexander
* Modify some log levels * Update gomatrixserverlib to matrix-org/gomatrixserverlib@336334f * Update gomatrixserverlib to matrix-org/gomatrixserverlib@cde7ac8 * Demote warning about key change producer * Add more useful roomserver logging * Further tweaking
2022-01-31Revert consumer changeNeil Alexander
2022-01-31Only limit context for fetching missing auth/prev events (#2131)Neil Alexander
2022-01-28Call hooks for outliers (#2119)Neil Alexander
* Move hook call when processing room events * Fix build * Call hooks for outliers too
2022-01-28Move hook call when processing room events (#2118)Neil Alexander
* Move hook call when processing room events * Fix build
2022-01-28Don't flood Sentry with context cancelled/deadline exceeded errors (#2115)Neil Alexander
2022-01-27Roomserver/federation input refactor (#2104)Neil Alexander
* Put federation client functions into their own file * Look for missing auth events in RS input * Remove retrieveMissingAuthEvents from federation API * Logging * Sorta transplanted the code over * Use event origin failing all else * Don't get stuck on mutexes: * Add verifier * Don't mark state events with zero snapshot NID as not existing * Check missing state if not an outlier before storing the event * Reject instead of soft-fail, don't copy roominfo so much * Use synchronous contexts, limit time to fetch missing events * Clean up some commented out bits * Simplify `/send` endpoint significantly * Submit async * Report errors on sending to RS input * Set max payload in NATS to 16MB * Tweak metrics * Add `workerForRoom` for tidiness * Try skipping unmarshalling errors for RespMissingEvents * Track missing prev events separately to avoid calculating state when not possible * Tweak logic around checking missing state * Care about state when checking missing prev events * Don't check missing state for create events * Try that again * Handle create events better * Send create room events as new * Use given event kind when sending auth/state events * Revert "Use given event kind when sending auth/state events" This reverts commit 089d64d271b5fca8c104e1554711187420dbebca. * Only search for missing prev events or state for new events * Tweaks * We only have missing prev if we don't supply state * Room version tweaks * Allow async inputs again * Apply backpressure to consumers/synchronous requests to hopefully stop things being overwhelmed * Set timeouts on roomserver input tasks (need to decide what timeout makes sense) * Use work queue policy, deliver all on restart * Reduce chance of duplicates being sent by NATS * Limit the number of servers we attempt to reduce backpressure * Some review comment fixes * Tidy up a couple things * Don't limit servers, randomise order using map * Some context refactoring * Update gmsl * Don't resend create events * Set stateIDs length correctly or else the roomserver thinks there are missing events when there aren't * Exclude our own servername * Try backing off servers * Make excluding self behaviour optional * Exclude self from g_m_e * Update sytest-whitelist * Update consumers for the roomserver output stream * Remember to send outliers for state returned from /gme * Make full HTTP tests less upsetti * Remove 'If a device list update goes missing, the server resyncs on the next one' from the sytest blacklist * Remove debugging test * Fix blacklist again, remove unnecessary duplicate context * Clearer contexts, don't use background in case there's something happening there * Don't queue up events more than once in memory * Correctly identify create events when checking for state * Fill in gaps again in /gme code * Remove `AuthEventIDs` from `InputRoomEvent` * Remove stray field Co-authored-by: Kegan Dougal <kegan@matrix.org>
2022-01-07NATS JetStream tweaks (#2086)Neil Alexander
* Use named NATS durable consumers * Build fixes * Remove dupe call to SetFederationAPI * Use namespaced consumer name * Fix namespacing * Fix unit tests hopefully
2022-01-07Fix panic at startup if roomserver was not given federation API reference by ↵Neil Alexander
the time NATS consumes an event, tweak backpressure metrics
2022-01-05Add NATS JetStream support (#1866)S7evinK
* Add NATS JetStream support Update shopify/sarama * Fix addresses * Don't change Addresses in Defaults * Update saramajetstream * Add missing error check Keep typing events for at least one minute * Use all configured NATS addresses * Update saramajetstream * Try setting up with NATS * Make sure NATS uses own persistent directory (TODO: make this configurable) * Update go.mod/go.sum * Jetstream package * Various other refactoring * Build fixes * Config tweaks, make random jetstream storage path for CI * Disable interest policies * Try to sane default on jetstream base path * Try to use in-memory for CI * Restore storage/retention * Update nats.go dependency * Adapt changes to config * Remove unneeded TopicFor * Dep update * Revert "Remove unneeded TopicFor" This reverts commit f5a4e4a339b6f94ec215778dca22204adaa893d1. * Revert changes made to streams * Fix build problems * Update nats-server * Update go.mod/go.sum * Roomserver input API queuing using NATS * Fix topic naming * Prometheus metrics * More refactoring to remove saramajetstream * Add missing topic * Don't try to populate map that doesn't exist * Roomserver output topic * Update go.mod/go.sum * Message acknowledgements * Ack tweaks * Try to resume transaction re-sends * Try to resume transaction re-sends * Update to matrix-org/gomatrixserverlib@91dadfb * Remove internal.PartitionStorer from components that don't consume keychanges * Try to reduce re-allocations a bit in resolveConflictsV2 * Tweak delivery options on RS input * Publish send-to-device messages into correct JetStream subject * Async and sync roomserver input * Update dendrite-config.yaml * Remove roomserver tests for now (they need rewriting) * Remove roomserver test again (was merged back in) * Update documentation * Docker updates * More Docker updates * Update Docker readme again * Fix lint issues * Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset) * Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that * Go 1.16 instead of Go 1.13 for upgrade tests and Complement * Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that" This reverts commit 368675283fc44501f227639811bdb16dd5deef8c. * Don't report any errors on `/send` to see what fun that creates * Fix panics on closed channel sends * Enforce state key matches sender * Do the same for leave * Various tweaks to make tests happier Squashed commit of the following: commit 13f9028e7a63662759ce7c55504a9d2423058668 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 15:47:14 2022 +0000 Do the same for leave commit e6be7f05c349fafbdddfe818337a17a60c867be1 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 15:33:42 2022 +0000 Enforce state key matches sender commit 85ede6d64bf10ce9b91cdd6d80f87350ee55242f Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 14:07:04 2022 +0000 Fix panics on closed channel sends commit 9755494a98bed62450f8001d8128e40481d27e15 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 13:38:22 2022 +0000 Don't report any errors on `/send` to see what fun that creates commit 3bb4f87b5dd56882febb4db5621db484c8789b7c Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 13:00:26 2022 +0000 Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that" This reverts commit 368675283fc44501f227639811bdb16dd5deef8c. commit fe2673ed7be9559eaca134424e403a4faca100b0 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 12:09:34 2022 +0000 Go 1.16 instead of Go 1.13 for upgrade tests and Complement commit 368675283fc44501f227639811bdb16dd5deef8c Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 11:51:45 2022 +0000 Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that commit b028dfc08577bcf52e6cb498026e15fa5d46d07c Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 10:29:08 2022 +0000 Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset) * Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork * Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers * Fix consumer component name in federation API * Add comment explaining where streams are defined * Tweaks to roomserver input with comments * Finish that sentence that I apparently forgot to finish in INSTALL.md * Bump version number of config to 2 * Add comments around asynchronous sends to roomserver in processEventWithMissingState * More useful error message when the config version does not match * Set version in generate-config * Fix version in config.Defaults Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-12-09Return event NID from `StoreEvent`, match PSQL vs SQLite behaviour, tweak ↵Neil Alexander
backfill persistence (#2071)
2021-11-22Don't persist transaction IDs in the roomserver (#2048)Neil Alexander
2021-07-22Track knocking in membership updater (#1935)Neil Alexander
* Topologically sort outliers in SendEventWithState * Knock in membership updater * Update gomatrixserverlib * Update gomatrixserverlib * Get the NID of the knock event properly for the membership updater
2021-07-13Expose more data when outputting output room events (#1916)kegsay
* Add more logging for content fields * Fix fields
2021-07-02Roomserver input backpressure metricNeil Alexander
Squashed commit of the following: commit 56e934ac0aeedcfb2c072010959ba49734d4e0cb Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Fri Jul 2 09:39:30 2021 +0100 Fix metric commit 3911f3a0c17b164b012e881c085ceca30f5de408 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Fri Jul 2 09:36:29 2021 +0100 Register correct metric commit a9ddbfaed421538a701151801e9451198a8be4f3 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Fri Jul 2 09:33:33 2021 +0100 Try to capture RS input backpressure metric
2021-06-28Use a custom FIFO queue for the RS input API (#1888)Neil Alexander
* Use a FIFO queue instead of a channel to reduce backpressure * Make sure someone wakes up * Tweaks * Add comments
2021-03-24Add Sentry support (#1803)Kegsay
* Add Sentry support * Use HTTP Sentry properly maybe * Capture panics * Log fed Sentry stuff correctly * British english linter
2021-02-06Fix crash in membership updater (#1753)Neil Alexander
* Fix nil pointer exception in membership updater * goimports
2021-02-04Remove redundant check (#1748)Neil Alexander
2021-01-18Don't create so many state snapshots when updating forward extremities (#1718)Neil Alexander
* Light-weight checking of state changes when updating forward extremities * Only do this for non-state events, since state events will always result in state change at extremities
2021-01-13Add dendrite_roomserver_processroomevent_duration_millis to prometheusNeil Alexander
Squashed commit of the following: commit e5e2d793119733ecbcf9b85f966e018ab0318741 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Wed Jan 13 17:28:12 2021 +0000 Add dendrite_roomserver_processroomevent_duration_millis to prometheus
2021-01-11Deep-checking forward extremities (#1698)Neil Alexander
2020-12-14Don't use request context for input room event queued tasks (#1640)Neil Alexander
2020-12-14Don't start more goroutines than needed on RS input, increase input worker ↵Neil Alexander
buffer size (#1638)
2020-12-09Adjust latest events updater (#1623)Neil Alexander
* Adjust forward elatest events updater * Populate newLatest in all cases * Re-add existingPrevs loop
2020-12-04MSC2836 threading: part 2 (#1596)Kegsay
* Update GMSL * Add MSC2836EventRelationships to fedsender * Call MSC2836EventRelationships in reqCtx * auth remote servers * Extract room ID and servers from previous events; refactor a bit * initial cut of federated threading * Use the right client/fed struct in the response * Add QueryAuthChain for use with MSC2836 * Add auth chain to federated response * Fix pointers * under CI: more logging and enable mscs, nil fix * Handle direction: up * Actually send message events to the roomserver.. * Add children and children_hash to unsigned, with tests * Add logic for exploring threads and tracking children; missing storage functions * Implement storage functions for children * Add fetchUnknownEvent * Do federated hits for include_children if we have unexplored children * Use /ev_rel rather than /event as the former includes child metadata * Remove cross-room threading impl * Enable MSC2836 in the p2p demo * Namespace mscs db * Enable msc2836 for ygg Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-19MSC2836: Threading - part one (#1589)Kegsay
* Add mscs/hooks package, begin work for msc2836 * Flesh out hooks and add SQL schema * Begin implementing core msc2836 logic * Add test harness * Linting * Implement visibility checks; stub out APIs for tests * Flesh out testing * Flesh out walkThread a bit * Persist the origin_server_ts as well * Edges table instead of relationships * Add nodes table for event metadata * LEFT JOIN to extract origin_server_ts for children * Add graph walking structs * Implement walking algorithm * Add more graph walking tests * Add auto_join for local rooms * Fix create table syntax on postgres * Add relationship_room_id|servers to the unsigned section of events * Persist the parent room_id/servers in edge metadata Other events cannot assert the true room_id/servers for the parent event, only make claims to them, hence why this is edge metadata. * guts to pass through room_id/servers * Refactor msc2836 to allow handling from federation * Add JoinedVia to PerformJoin responses * Fix tests; review comments
2020-11-16Pass pointers to events — reloaded (#1583)Neil Alexander
* Pass events as pointers * Fix lint errors * Update gomatrixserverlib * Update gomatrixserverlib * Update to matrix-org/gomatrixserverlib#240
2020-10-27Add possibilty to configure MaxMessageBytes for sarama (#1563)S7evinK
* Add configuration for max_message_bytes for sarama * Log all errors when sending multiple messages Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Add missing config * - Better comments on what MaxMessageBytes is used for - Also sets the size the consumer may use
2020-10-22Fix RewritesState bug (#1557)Neil Alexander
* Set RewritesState once * Check if any new state provided * Obey rewritesState * Don't nuke everything the sync API knows when purging state * Fix panic from duplicate insert * Consistency * Use HasState * Remove nolint * Clean up joined rooms on state rewrite
2020-10-21Don't set empty state snapshotsNeil Alexander
2020-10-21Refactor forward extremities (#1556)Neil Alexander
* Add resolve-state helper * Tweaks * Refactor forward extremities, again * Tweaks * Minor optimisation * Make path a bit clearer * Only process state/membership if forward extremities have changed * Usage comments in resolve-state
2020-10-20Remove invalid state delta check (#1550)Neil Alexander
2020-10-20LintingKegan Dougal
2020-10-20Fix sqlite locking bugs present on sytest (#1543)Kegsay
* Fix sqite locking bugs present on sytest Comments do the explaining. * Fix deadlock in sqlite mode Caused by starting a writer whilst within a writer * Only complain about invalid state deltas for non-overwrite events * Do not re-process outlier unnecessarily
2020-10-19KindOld (#1531)Neil Alexander
* Add KindOld * Don't process latest events/memberships for old events * Allow federationsender to ignore duplicate key entries when LatestEventIDs is duplicated by RS output events * Signal to downstream components if an event has become a forward extremity * Don't exclude from sync * Soft-fail checks on KindNew * Don't run the latest events updater at all for KindOld * Don't make federation sender change after all * Kind in federation sender join * Don't send isForwardExtremity * Fix syncapi * Update comments * Fix SendEventWithState * Update sytest-whitelist * Generate old output events * Sync API consumes old room events * Update comments
2020-10-14Improved state handling in /send (#1521)Neil Alexander
* Capture errors * Don't request only state key tuples needed for auth (we end up discarding room state this way) * QueryStateAfterEvent returns all state when no tuples supplied * Resolve state * Comments
2020-10-08Don't get into situations where we have no forward extremitiesNeil Alexander
2020-10-08Don't set new state NID if state regressionNeil Alexander
2020-10-08Return a non-fatal error to the federation API on a state regression (#1498)Neil Alexander
* Return a non-fatal error to the federation API on a state regression * Return no error but don't morph state
2020-10-07Deep checking of forward extremities (#1491)Neil Alexander
* Deep forward extremity calculation * Use updater txn * Update error * Update error * Create previous event references in StoreEvent * Use latest events updater to row-lock prev events * Fix unexpected fallthrough * Fix deadlock * Don't roll back * Update comments in calculateLatest * Don't include events that we can't find references for in the forward extremities * Add another passing test
2020-10-06Remove KindRewrite (#1481)Neil Alexander
* Don't send rewrite events * Remove final traces of rewrite events * Remove test that is no longer needed * Revert "Remove test that is no longer needed" This reverts commit 9a45babff690480acd656a52f2c2950a5f7e9ada. * Update test to use KindOutlier
2020-10-05Resolve state after event against current room state when determining latest ↵Neil Alexander
state changes (#1479) * Resolve state after event against current room state when determining latest state changes * Update sytest-whitelist * Update sytest-whitelist, blacklist
2020-09-29Fetch missing auth events, implement QueryMissingAuthPrevEvents, try other ↵Neil Alexander
servers in room for /event and /get_missing_events (#1450) * Try to ask other servers in the room for missing events if the origin won't provide them * Logging * More logging * Implement QueryMissingAuthPrevEvents * Try to get missing auth events badly * Use processEvent * Logging * Update QueryMissingAuthPrevEvents * Try to find missing auth events * Patchy fix for test * Logging tweaks * Send auth events as outliers * Update check in QueryMissingAuthPrevEvents * Error responses * More return codes * Don't return error on reject/soft-fail since it was ultimately handled * More tweaks * More error tweaks
2020-09-21Soft-fail (#1364)Neil Alexander
* Initial work on soft-fail * Fix state block retrieval * Copy-pasta QueryLatestEventsAndState code * Fix state lookup * Clean up * Fix up failing sytest * Linting * Update previous events SQLite insert query * Update SQLite InsertPreviousEvent properly * Hopefully fix the event references updates Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-16Implement rejected events (#1426)Kegsay
* WIP Event rejection * Still send back errors for rejected events Instead, discard them at the federationapi /send layer rather than re-implementing checks at the clientapi/PerformJoin layer. * Implement rejected events Critically, rejected events CAN cause state resolution to happen as it can merge forks in the DAG. This is fine, _provided_ we do not add the rejected event when performing state resolution, which is what this PR does. It also fixes the error handling when NotAllowed happens, as we were checking too early and needlessly handling NotAllowed in more than one place. * Update test to match reality * Modify InputRoomEvents to no longer return an error Errors do not serialise across HTTP boundaries in polylith mode, so instead set fields on the InputRoomEventsResponse. Add `Err()` function to make the API shape basically the same. * Remove redundant returns; linting * Update blacklist
2020-09-15Handle state with input event as new events (#1415)Neil Alexander
* SendEventWithState events as new * Use cumulative state IDs for final event * Error wrapping in calculateAndSetState * Handle overwriting same event type and state key * Hacky way to spot historical events * Don't exclude from sync * Don't generate output events when rewriting forward extremities * Update output event check * Historical output events * Define output room event type * Notify key changes on state * Don't send our membership event twice * Deduplicate state entries * Tweaks * Remove unnecessary nolint * Fix current state upsert in sync API * Send auth events as outliers, state events as rewrite * Sync API don't consume state events * Process events actually * Improve outlier check * Fix local room check * Remove extra room check, it seems to break the whole damn world * Fix federated join check * Fix nil pointer exception * Better comments on DeduplicateStateEntries * Reflow forced federated joins * Don't force federated join for possibly even local invites * Comment SendEventWithState better * Rewrite room state in sync API storage * Add TODO * Clean up all room data when receiving create event * Don't generate output events for rewrites, but instead notify that state is rewritten on the final new event * Rename to PurgeRoom * Exclude backfilled messages from /sync * Split out rewriting state from updating state from state res Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-04Remove ServerACLs from the current state server (#1390)Kegsay
* Remove ServerACLs from the current state server Functionality moved to roomserver * Nothing to see here, move along
2020-09-03FIFO ordering of input events (#1386)Neil Alexander
* Initial FIFOing of roomserver inputs * Remove EventID response from api.InputRoomEventsResponse * Don't send back event ID unnecessarily * Fix ordering hopefully * Reduce copies, use buffered task channel to reduce contention on other rooms * Fix error handling
2020-09-02Add Queryer and Inputer and factor out more RSAPI stuff (#1382)Kegsay
* Add Queryer and use embedded structs * Add Inputer and factor out more RS API stuff This neatly splits up the RS API based on the functionality it provides, whilst providing a useful place for code sharing via the `helpers` package.
generated by cgit v1.2.3 (git 2.26.2) at 2024-11-25 23:25:07 +0000