aboutsummaryrefslogtreecommitdiff
path: root/roomserver
AgeCommit message (Collapse)Author
2021-02-16Don't query for servers so often in /send (#1766)Neil Alexander
* Look up servers less often, don't hit API for missing auth events unless there are actually missing auth events * Remove ResolveConflictsAdhoc (since it is already in GMSL), other tweaks * Update gomatrixserverlib to matrix-org/gomatrixserverlib#254 * Fix resolve-state * Initialise t.servers on first use
2021-02-06Fix crash in membership updater (#1753)Neil Alexander
* Fix nil pointer exception in membership updater * goimports
2021-02-04Remove redundant check (#1748)Neil Alexander
2021-01-29Add m.room.create to invite stripped state (#1740)Kegsay
MSC1772 needs this because the create event contains info on if the room is a space or not. The create event itself isn't sensitive so other people may find this useful too.
2021-01-26Don't retrieve same state events over and over again (#1737)Neil Alexander
2021-01-22Peeking over federation via MSC2444 (#1391)Matthew Hodgson
* a very very WIP first cut of peeking via MSC2753. doesn't yet compile or work. needs to actually add the peeking block into the sync response. checking in now before it gets any bigger, and to gather any initial feedback on the vague shape of it. * make PeekingDeviceSet private * add server_name param * blind stab at adding a `peek` section to /sync * make it build * make it launch * add peeking to getResponseWithPDUsForCompleteSync * cancel any peeks when we join a room * spell out how to runoutside of docker if you want speed * fix SQL * remove unnecessary txn for SelectPeeks * fix s/join/peek/ cargocult fail * HACK: Track goroutine IDs to determine when we write by the wrong thread To use: set `DENDRITE_TRACE_SQL=1` then grep for `unsafe` * Track partition offsets and only log unsafe for non-selects * Put redactions in the writer goroutine * Update filters on writer goroutine * wrap peek storage in goid hack * use exclusive writer, and MarkPeeksAsOld more efficiently * don't log ascii in binary at sql trace... * strip out empty roomd deltas * re-add txn to SelectPeeks * re-add accidentally deleted field * reject peeks for non-worldreadable rooms * move perform_peek * fix package * correctly refactor perform_peek * WIP of implementing MSC2444 * typo * Revert "Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking" This reverts commit 3cebd8dbfbccdf82b7930b7b6eda92095ca6ef41, reversing changes made to ed4b3a58a7855acc43530693cc855b439edf9c7c. * (almost) make it build * clean up bad merge * support SendEventWithState with optional event * fix build & lint * fix build & lint * reinstate federated peeks in the roomserver (doh) * fix sql thinko * todo for authenticating state returned by /peek * support returning current state from QueryStateAndAuthChain * handle SS /peek * reimplement SS /peek to prod the RS to tell the FS about the peek * rename RemotePeeks as OutboundPeeks * rename remote_peeks_table as outbound_peeks_table * add perform_handle_remote_peek.go * flesh out federation doc * add inbound peeks table and hook it up * rename ambiguous RemotePeek as InboundPeek * rename FSAPI's PerformPeek as PerformOutboundPeek * setup inbound peeks db correctly * fix api.SendEventWithState with no event * track latestevent on /peek * go fmt * document the peek send stream race better * fix SendEventWithRewrite not to bail if handed a non-state event * add fixme * switch SS /peek to use SendEventWithRewrite * fix comment * use reverse topo ordering to find latest extrem * support postgres for federated peeking * go fmt * back out bogus go.mod change * Fix performOutboundPeekUsingServer * Fix getAuthChain -> GetAuthChain * Fix build issues * Fix build again * Fix getAuthChain -> GetAuthChain * Don't repeat outbound peeks for the same room ID to the same servers * Fix lint * Don't omitempty to appease sytest Co-authored-by: Kegan Dougal <kegan@matrix.org> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-18Don't create so many state snapshots when updating forward extremities (#1718)Neil Alexander
* Light-weight checking of state changes when updating forward extremities * Only do this for non-state events, since state events will always result in state change at extremities
2021-01-13Add dendrite_roomserver_processroomevent_duration_millis to prometheusNeil Alexander
Squashed commit of the following: commit e5e2d793119733ecbcf9b85f966e018ab0318741 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Wed Jan 13 17:28:12 2021 +0000 Add dendrite_roomserver_processroomevent_duration_millis to prometheus
2021-01-11Deep-checking forward extremities (#1698)Neil Alexander
2020-12-18Fix #1655 by re-adding the appservice alias query (#1660)Neil Alexander
2020-12-16Add RoomInfo cache, remove RoomServerRoomNIDsCache (#1646)Neil Alexander
* Add RoomInfo cache, remove RoomServerRoomNID cache, ensure caches are thread-safe * Don't panic if the roomInfo isn't known yet * LRU package is already threadsafe * Use RoomInfo cache to find room version if possible in Events() * Adding comments about RoomInfoCache safety
2020-12-16Hit the database far less in Events to find room NIDs and room versions (#1643)Neil Alexander
* Hit the database far less to find room NIDs for event NIDs * Close the rows * Fix SQLite selectRoomNIDsForEventNIDsSQL * Give same treatment to room version lookups
2020-12-14Don't use request context for input room event queued tasks (#1640)Neil Alexander
2020-12-14Don't start more goroutines than needed on RS input, increase input worker ↵Neil Alexander
buffer size (#1638)
2020-12-09Adjust latest events updater (#1623)Neil Alexander
* Adjust forward elatest events updater * Populate newLatest in all cases * Re-add existingPrevs loop
2020-12-04MSC2836 threading: part 2 (#1596)Kegsay
* Update GMSL * Add MSC2836EventRelationships to fedsender * Call MSC2836EventRelationships in reqCtx * auth remote servers * Extract room ID and servers from previous events; refactor a bit * initial cut of federated threading * Use the right client/fed struct in the response * Add QueryAuthChain for use with MSC2836 * Add auth chain to federated response * Fix pointers * under CI: more logging and enable mscs, nil fix * Handle direction: up * Actually send message events to the roomserver.. * Add children and children_hash to unsigned, with tests * Add logic for exploring threads and tracking children; missing storage functions * Implement storage functions for children * Add fetchUnknownEvent * Do federated hits for include_children if we have unexplored children * Use /ev_rel rather than /event as the former includes child metadata * Remove cross-room threading impl * Enable MSC2836 in the p2p demo * Namespace mscs db * Enable msc2836 for ygg Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-12-04Don't recalculate event IDs so often (#1610)Neil Alexander
* Don't recalculate event IDs so often * Revert invite change * Make sure we're using the right NIDs * Update gomatrixserverlib * Update to NewEventFromTrustedJSONWithEventID * Fix go.mod * Update gomatrixserverlib to matrix-org/gomatrixserverlib#243 * Use BulkSelectEventID
2020-12-03Peeking updates (#1607)Neil Alexander
* Add unpeek * Don't allow peeks into encrypted rooms * Fix send tests * Update consumers
2020-12-02Top-level setup package (#1605)Neil Alexander
* Move config, setup, mscs into "setup" top-level folder * oops, forgot the EDU server * Add setup * goimports
2020-12-02Optimise CheckServerAllowedToSeeEvent (#1602)Neil Alexander
* Try to limit how many state events we have to unmarshal * Comments
2020-11-19MSC2836: Threading - part one (#1589)Kegsay
* Add mscs/hooks package, begin work for msc2836 * Flesh out hooks and add SQL schema * Begin implementing core msc2836 logic * Add test harness * Linting * Implement visibility checks; stub out APIs for tests * Flesh out testing * Flesh out walkThread a bit * Persist the origin_server_ts as well * Edges table instead of relationships * Add nodes table for event metadata * LEFT JOIN to extract origin_server_ts for children * Add graph walking structs * Implement walking algorithm * Add more graph walking tests * Add auto_join for local rooms * Fix create table syntax on postgres * Add relationship_room_id|servers to the unsigned section of events * Persist the parent room_id/servers in edge metadata Other events cannot assert the true room_id/servers for the parent event, only make claims to them, hence why this is edge metadata. * guts to pass through room_id/servers * Refactor msc2836 to allow handling from federation * Add JoinedVia to PerformJoin responses * Fix tests; review comments
2020-11-16Pass pointers to events — reloaded (#1583)Neil Alexander
* Pass events as pointers * Fix lint errors * Update gomatrixserverlib * Update gomatrixserverlib * Update to matrix-org/gomatrixserverlib#240
2020-11-16Fix Dendrite not backfilling on world_readable rooms (#1575)Mayeul Cantan
The previous implementation was only checking if room history was "shared", which it wasn't for rooms where a user was invited, or world readable rooms. This implementation leverages the IsServerAllowed method, which already implements the complete verification algorithm. Signed-off-by: `Mayeul Cantan <oss+matrix@mayeul.net>` Co-authored-by: Kegsay <kegan@matrix.org>
2020-11-05Implement forgetting about rooms (#1572)S7evinK
* Add basic storage methods * Add internal api handler * Add check for forgotten room * Add /rooms/{roomID}/forget endpoint * Add missing rsAPI method * Remove unused parameters * Add passing tests Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Add missing file * Add postgres migration * Add sqlite migration * Use Forgetter to forget room * Remove empty line * Update HTTP status codes It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-10-27Add possibilty to configure MaxMessageBytes for sarama (#1563)S7evinK
* Add configuration for max_message_bytes for sarama * Log all errors when sending multiple messages Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Add missing config * - Better comments on what MaxMessageBytes is used for - Also sets the size the consumer may use
2020-10-22Fix RewritesState bug (#1557)Neil Alexander
* Set RewritesState once * Check if any new state provided * Obey rewritesState * Don't nuke everything the sync API knows when purging state * Fix panic from duplicate insert * Consistency * Use HasState * Remove nolint * Clean up joined rooms on state rewrite
2020-10-21Don't set empty state snapshotsNeil Alexander
2020-10-21Refactor forward extremities (#1556)Neil Alexander
* Add resolve-state helper * Tweaks * Refactor forward extremities, again * Tweaks * Minor optimisation * Make path a bit clearer * Only process state/membership if forward extremities have changed * Usage comments in resolve-state
2020-10-20Always call overridden form of GetLatestEventsForUpdate (#1554)Kegsay
This ensures we don't make txns on sqlite still, which can cause 'database is locked' errors.
2020-10-20Remove invalid state delta check (#1550)Neil Alexander
2020-10-20LintingKegan Dougal
2020-10-20Fix sqlite locking bugs present on sytest (#1543)Kegsay
* Fix sqite locking bugs present on sytest Comments do the explaining. * Fix deadlock in sqlite mode Caused by starting a writer whilst within a writer * Only complain about invalid state deltas for non-overwrite events * Do not re-process outlier unnecessarily
2020-10-19KindOld (#1531)Neil Alexander
* Add KindOld * Don't process latest events/memberships for old events * Allow federationsender to ignore duplicate key entries when LatestEventIDs is duplicated by RS output events * Signal to downstream components if an event has become a forward extremity * Don't exclude from sync * Soft-fail checks on KindNew * Don't run the latest events updater at all for KindOld * Don't make federation sender change after all * Kind in federation sender join * Don't send isForwardExtremity * Fix syncapi * Update comments * Fix SendEventWithState * Update sytest-whitelist * Generate old output events * Sync API consumes old room events * Update comments
2020-10-15Better logging around db.StoreEventNeil Alexander
2020-10-15Start Kafka connections for each component that needs them (#1527)Neil Alexander
* Start Kafka connection for each component that needs one * Fix roomserver unit tests * Rename to naffkaInstance (@Kegsay review comment) * Fix import cycle
2020-10-14txn nil guard when rolling back LatestEventsUpdater (#1524)Kegsay
* txn nil guard when rolling back LatestEventsUpdater * Spell lint correctly
2020-10-14Ignore duplicate redaction entries (#1522)Neil Alexander
2020-10-14Improved state handling in /send (#1521)Neil Alexander
* Capture errors * Don't request only state key tuples needed for auth (we end up discarding room state this way) * QueryStateAfterEvent returns all state when no tuples supplied * Resolve state * Comments
2020-10-13Wrap NewMembershipUpdater in a db writer (#1515)Kegsay
2020-10-12Get missing event tweaks (#1514)Neil Alexander
* Adjust backfill to send backward extremity with state before other backfilled events, include prev_events with no state amongst missing events * Not finished refactor * Fix test * Remove isInboundTxn * Remove debug logging
2020-10-08Don't get into situations where we have no forward extremitiesNeil Alexander
2020-10-08Don't set new state NID if state regressionNeil Alexander
2020-10-08Return what we have when we encounter missing events when servicing ↵Kegsay
backfill/gme (#1499) We expect to have missing events as we walk back in the DAG over federation as we didn't always create the room. When checking if the server is allowed to see those events, just give up and stop rather than fail the request.
2020-10-08Return a non-fatal error to the federation API on a state regression (#1498)Neil Alexander
* Return a non-fatal error to the federation API on a state regression * Return no error but don't morph state
2020-10-07Rename serverkeyapi to signingkeyserver (#1492)Kegsay
* Rename serverkeyapi to signingkeyserver We use "api" for public facing stuff and "server" for internal stuff. As the server key API is internal only, we call it 'signing key server', which also clarifies the type of key (as opposed to TLS keys, E2E keys, etc) * Convert docker/scripts to use signing-key-server * Rename missed bits
2020-10-07Deep checking of forward extremities (#1491)Neil Alexander
* Deep forward extremity calculation * Use updater txn * Update error * Update error * Create previous event references in StoreEvent * Use latest events updater to row-lock prev events * Fix unexpected fallthrough * Fix deadlock * Don't roll back * Update comments in calculateLatest * Don't include events that we can't find references for in the forward extremities * Add another passing test
2020-10-07Don't return 500s on checking to see if a remote server is allowed to see an ↵Neil Alexander
event we don't know about (#1490)
2020-10-06Optimise servers to backfill from (#1485)Kegsay
- Prefer perspective servers if they are in the room. - Limit the number of backfill servers to 5 to avoid taking too long.
2020-10-06Don't give up if we fail to fetch a key (#1483)Neil Alexander
* Don't give up if we fail to fetch a key * Fix logging line * furl nolint
2020-10-06Remove KindRewrite (#1481)Neil Alexander
* Don't send rewrite events * Remove final traces of rewrite events * Remove test that is no longer needed * Revert "Remove test that is no longer needed" This reverts commit 9a45babff690480acd656a52f2c2950a5f7e9ada. * Update test to use KindOutlier
generated by cgit v1.2.3 (git 2.26.2) at 2025-02-28 07:28:19 +0000