aboutsummaryrefslogtreecommitdiff
path: root/roomserver/internal/input/input_missing.go
AgeCommit message (Collapse)Author
2023-07-07Add event size checks similar to Synapse (#3140)Till
Companion to https://github.com/matrix-org/gomatrixserverlib/pull/400 This tries to mimic the logic found in Synapse, as dropping events can break rooms (and we may end up in endless loops..)
2023-06-14Merge SenderID & Per Room User Key work (#3109)devonh
2023-06-07Use SenderID Type (#3105)devonh
2023-06-06PDU Sender split (#3100)devonh
Initial cut of splitting PDU Sender into SenderID & looking up UserID where required.
2023-05-24Drop `reference_sha` column (#3083)Till
Companion PR to https://github.com/matrix-org/gomatrixserverlib/pull/383
2023-05-08Only require room version instead of room info for db.Events() (#3079)devonh
This reduces the API requirements for the Events database to align with what is actually required.
2023-05-03Use PDU in even more places (#3074)kegsay
- No longer rely on *Event returning from NewEventFrom... functions Requires https://github.com/matrix-org/gomatrixserverlib/pull/377
2023-05-02Use PDU not *Event in HeaderedEvent (#3073)kegsay
Requires https://github.com/matrix-org/gomatrixserverlib/pull/376 This has numerous upsides: - Less type casting to `*Event` is required. - Making Dendrite work with `PDU` interfaces means we can swap out Event impls more easily. - Tests which represent weird event shapes are easier to write. Part of a series of refactors on GMSL.
2023-04-27Use PDU interface (#3070)kegsay
We only use it in a few places currently, enough to get things to compile and run. We should be using it in much more places. Similarly, in some places we cast []PDU back to []*Event, we need to not do that. Likewise, in some places we cast PDU to *Event, we need to not do that. For now though, hopefully this is a start.
2023-04-27Remove gmsl.HeaderedEvent (#3068)kegsay
Replaced with types.HeaderedEvent _for now_. In reality we want to move them all to gmsl.Event and only use HeaderedEvent when we _need_ to bundle the version/event ID with the event (seriailsation boundaries, and even then only when we don't have the room version). Requires https://github.com/matrix-org/gomatrixserverlib/pull/373
2023-04-21Implement new RoomVersionImpl API (#3062)kegsay
As outlined in https://github.com/matrix-org/gomatrixserverlib/pull/368 The main change Dendrite side is that `RoomVersion` no longer has any methods on it. Instead, you need to bounce via `gmsl.GetRoomVersion`. It's very interesting to see where exactly Dendrite cares about this. For some places it's creating events (fine) but others are way more specific. Those areas will need to migrate to GMSL at some point.
2023-04-20refactor: funnel event creation through room versions (#3060)kegsay
In preparation of interfacing up the room version value.
2023-04-19refactor: update GMSL (#3058)kegsay
Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/364 Read this commit by commit to avoid going insane.
2023-04-14Update GMSL: use static Check functions (#3052)kegsay
Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/359 A nice side effect is that we don't need to re-parse the events in some cases.
2023-04-06refactor: use latest GMSL which splits fed client from matrix room logic (#3051)kegsay
Part of a series of refactors on GMSL.
2023-03-13Add basic runtime tracing (#2996)Till
This allows us in almost all places to use regions to further trace down long running tasks. Also removes an unused function.
2023-03-01Refactor `StoreEvent`, add `MaybeRedactEvent`, create an `EventDatabase` (#2989)Till
This PR changes the following: - `StoreEvent` now only stores an event (and possibly prev event), instead of also doing redactions - Adds a `MaybeRedactEvent` (pulled out from `StoreEvent`), which should be called after storing events - a few other things
2023-02-24Refactor `StoreEvent` and create a new `RoomDatabase` interface (#2985)Till
This PR changes a few things: - It pulls out the creation of several NIDs from the `StoreEvent` function to make the functions more reusable - Uses more caching when using those NIDs to avoid DB round trips
2022-11-15Federation fixes for virtual hostingNeil Alexander
2022-11-02Improve logging for `processEventWithMissingState`Neil Alexander
2022-09-12Fix missing signature check on the `/get_missing_events` responseNeil Alexander
2022-08-03Try more servers when calling `/state_ids` (#2610)Neil Alexander
* Try more servers when calling `/state_ids` * More logging * Maybe fix concurrent map write * Revert "Maybe fix concurrent map write" This reverts commit da0dbb836207a911afe77e6f6d63c4809669693c. * Enforce a limit of 20s per server, 5 mins total
2022-08-02Fix data race in `lookupMissingStateViaStateIDs`Neil Alexander
2022-08-02Various roominfo tweaks (#2607)Neil Alexander
2022-06-07Optimise `loadAuthEvents`, add roomserver tracingNeil Alexander
2022-05-05Define component interfaces based on consumers (2/2) (#2425)kegsay
* convert remaining interfaces * Tidy up the userapi interfaces
2022-04-05Update to matrix-org/gomatrixserverlib#300Neil Alexander
2022-02-28Update `Events` call-sites which now don't return an error, update ↵Neil Alexander
`parsedRespState` to sort (#2227) * Topologically sort with `SendEventWithState`, so that earlier events should satisfy auth for later ones * Revert "Topologically sort with `SendEventWithState`, so that earlier events should satisfy auth for later ones" This reverts commit b0cd706012b4c9b6724b11e16f19c4cb732ab286. * Update to matrix-org/gomatrixserverlib#293 * `Events` no longer returns an error, other tweaks * Make sure `Events` is sorted for `parsedRespState` too
2022-02-23Relax roomserver input transactional isolation (#2224)Neil Alexander
* Don't force full transactional isolation on roomserver input * Set succeeded * Tweak `MissingAuthPrevEvents`
2022-02-11Ensure only one transaction is used for RS input per room (#2178)Neil Alexander
* Ensure the input API only uses a single transaction * Remove more of the dead query API call * Tidy up * Fix tests hopefully * Don't do unnecessary work for rooms that don't exist * Improve error, fix another case where transaction wasn't used properly * Add a unit test for checking single transaction on RS input API * Fix logic oops when deciding whether to use a transaction in storeEvent
2022-02-10Fix fetching missing state (#2163)Neil Alexander
* Check that we have a populated state snapshot when determining if we closed the gap * Do the same in the query API * Use HasState more opportunistically * Try to avoid falling down the hole of using a trustworthy but empty state snapshot for non-create events * Refactor missing state and make sure that we really solve the problem for the new event * Comments * Review comments * Tweak that check again * Tidy up that create check further * Fix build hopefully * Update sendOutliers to use OrderAuthAndStateEvents * Don't go out of bounds on missingEvents
2022-02-09Unmarshal events at the Dendrite level not GMSL level (#2164)kegsay
* Use new event json types in gmsl * Fix EventJSON to actually unmarshal events * Update GMSL * Bump GMSL and improve error messages * Send back the correct RespState * Update GMSL
2022-02-07Fix bugs that could wedge rooms (#2154)Neil Alexander
* Don't flake so badly for rejected events * Moar * Fix panic * Don't count rejected events as missing * Don't treat rejected events without state as missing * Revert "Don't count rejected events as missing" This reverts commit 4b6139b62eb91ba059b47415b0275964b37d9b43. * Missing events should be KindOld * If we have state, use it, regardless of memberships which could be stale now * Fetch missing state for KindOld too * Tweak the condition again * Clean up a bit * Use room updater to get latest events in a race-free way * Return the correct error * Improve errors
2022-02-04Remove roomserver input deadlines (#2144)Neil Alexander
It isn't really clear that the deadlines actually help in any way. Currently we can use up our 2 minutes doing something, run out of context time and then return an error which causes the transaction to rollback and forgetting everything we've done. If the message came to us from NATS then we probably will end up retrying just to be in the same situation. We'd be really a lot better if we just spent the time reconciling the problem in the first place, and then we're much less likely to need to fetch those missing auth or prev events in the future. Also includes matrix-org/gomatrixserverlib#287 so we don't wait so long for servers that are obviously dead.
2022-02-04Full roomserver input transactional isolation (#2141)Neil Alexander
* Add transaction to all database tables in roomserver, rename latest events updater to room updater, use room updater for all RS input * Better transaction management * Tweak order * Handle cases where the room does not exist * Other fixes * More tweaks * Fill some gaps * Fill in the gaps * good lord it gets worse * Don't roll back transactions when events rejected * Pass through errors properly * Fix bugs * Fix incorrect error check * Don't panic on nil txns * Tweaks * Hopefully fix panics for good in SQLite this time * Fix rollback * Minor bug fixes with latest event updater * Some review comments * Revert "Some review comments" This reverts commit 0caf8cf53e62c33f7b83c52e9df1d963871f751e. * Fix a couple of bugs * Clearer commit and rollback results * Remove unnecessary prepares
2022-01-31More logging tweaksNeil Alexander
2022-01-31Roomserver fixes (#2133)Neil Alexander
* Improve server selection somewhat * Remove things from the map when we're done * Be less panicky about auth event signatures in case they are not fatal after all * Accept HasState in all cases * Send join asynchronously * Revert "Send join asynchronously" This reverts commit 5b685bfcd0b1150a66c7b1e70fb3a3eda509efd1. * Joins and leaves use background context
2022-01-31Tweak some logging (#2130)Neil Alexander
* Modify some log levels * Update gomatrixserverlib to matrix-org/gomatrixserverlib@336334f * Update gomatrixserverlib to matrix-org/gomatrixserverlib@cde7ac8 * Demote warning about key change producer * Add more useful roomserver logging * Further tweaking
2022-01-31Only limit context for fetching missing auth/prev events (#2131)Neil Alexander
2022-01-27Roomserver/federation input refactor (#2104)Neil Alexander
* Put federation client functions into their own file * Look for missing auth events in RS input * Remove retrieveMissingAuthEvents from federation API * Logging * Sorta transplanted the code over * Use event origin failing all else * Don't get stuck on mutexes: * Add verifier * Don't mark state events with zero snapshot NID as not existing * Check missing state if not an outlier before storing the event * Reject instead of soft-fail, don't copy roominfo so much * Use synchronous contexts, limit time to fetch missing events * Clean up some commented out bits * Simplify `/send` endpoint significantly * Submit async * Report errors on sending to RS input * Set max payload in NATS to 16MB * Tweak metrics * Add `workerForRoom` for tidiness * Try skipping unmarshalling errors for RespMissingEvents * Track missing prev events separately to avoid calculating state when not possible * Tweak logic around checking missing state * Care about state when checking missing prev events * Don't check missing state for create events * Try that again * Handle create events better * Send create room events as new * Use given event kind when sending auth/state events * Revert "Use given event kind when sending auth/state events" This reverts commit 089d64d271b5fca8c104e1554711187420dbebca. * Only search for missing prev events or state for new events * Tweaks * We only have missing prev if we don't supply state * Room version tweaks * Allow async inputs again * Apply backpressure to consumers/synchronous requests to hopefully stop things being overwhelmed * Set timeouts on roomserver input tasks (need to decide what timeout makes sense) * Use work queue policy, deliver all on restart * Reduce chance of duplicates being sent by NATS * Limit the number of servers we attempt to reduce backpressure * Some review comment fixes * Tidy up a couple things * Don't limit servers, randomise order using map * Some context refactoring * Update gmsl * Don't resend create events * Set stateIDs length correctly or else the roomserver thinks there are missing events when there aren't * Exclude our own servername * Try backing off servers * Make excluding self behaviour optional * Exclude self from g_m_e * Update sytest-whitelist * Update consumers for the roomserver output stream * Remember to send outliers for state returned from /gme * Make full HTTP tests less upsetti * Remove 'If a device list update goes missing, the server resyncs on the next one' from the sytest blacklist * Remove debugging test * Fix blacklist again, remove unnecessary duplicate context * Clearer contexts, don't use background in case there's something happening there * Don't queue up events more than once in memory * Correctly identify create events when checking for state * Fill in gaps again in /gme code * Remove `AuthEventIDs` from `InputRoomEvent` * Remove stray field Co-authored-by: Kegan Dougal <kegan@matrix.org>
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-19 15:59:48 +0000