aboutsummaryrefslogtreecommitdiff
path: root/syncapi/storage
AgeCommit message (Collapse)Author
2021-07-20Only include go-sqlite3 on the relevant binaries (#1900)Neil Alexander
* Only include go-sqlite3 on the relevant binaries * The driver name is always sqlite3 now * Update to matrix-org/go-sqlite3-js@e537baa
2021-03-03Increase gocyclo complexity to 25 (and remove all but 2 golint directives ↵Neil Alexander
related to it) (#1783)
2021-02-17Don't exclude an event from sync if it was previously not excluded (#1767)Neil Alexander
2021-02-04Don't re-request state events that are already in the timeline (#1739)Neil Alexander
* Don't request state events if we already have the timeline events (Postgres only) * Rename variable * nocyclo * Add SQLite * Tweaks * Revert query change * Don't dedupe if asking for full state * Update query
2021-02-04Fix ON CONFLICT on sync API account data (#1745) (#1750)Neil Alexander
2021-01-20Add sync API memberships table (#1726)Neil Alexander
2021-01-19Basic sync filtering (#1721)Neil Alexander
* Add some filtering (postgres only for now) * Fix build error * Try to use request filter * Use default filter as a template when retrieving from the database * Remove unused strut * Update sytest-whitelist * Add filtering to SelectEarlyEvents * Fix Postgres selectEarlyEvents query * Attempt filtering on SQLite * Test limit, set field for limit/order in prepareWithFilters * Remove debug logging, add comments * Tweaks, debug logging * Separate SQLite stream IDs * Fix filtering in current state table * Fix lock issues * More tweaks * Current state requires room ID * Review comments
2021-01-13Simplify send-to-device messaging (#1702)Neil Alexander
* Simplify send-to-device messaging * Don't return error if there's no work to do * Remove SQLite migrations for now * Tweak Postgres migrations * Tweaks * Fixes * Cleanup separately * Fix SQLite migration
2021-01-08Sync refactor — Part 1 (#1688)Neil Alexander
* It's half-alive * Wakeups largely working * Other tweaks, typing works * Fix bugs, add receipt stream * Delete notifier, other tweaks * Dedupe a bit, add a template for the invite stream * Clean up, add templates for other streams * Don't leak channels * Bring forward some more PDU logic, clean up other places * Add some more wakeups * Use addRoomDeltaToResponse * Log tweaks, typing fixed? * Fix timed out syncs * Don't reset next batch position on timeout * Add account data stream/position * End of day * Fix complete sync for receipt, typing * Streams package * Clean up a bit * Complete sync send-to-device * Don't drop errors * More lightweight notifications * Fix typing positions * Don't advance position on remove again unless needed * Device list updates * Advance account data position * Use limit for incremental sync * Limit fixes, amongst other things * Remove some fmt.Println * Tweaks * Re-add notifier * Fix invite position * Fixes * Notify account data without advancing PDU position in notifier * Apply account data position * Get initial position for account data * Fix position update * Fix complete sync positions * Review comments @Kegsay * Room consumer parameters
2020-12-21fix imports (#1665)6543
* fix imports Signed-off-by: 6543 <6543@obermui.de> * add sqlite driver import back Signed-off-by: 6543 <6543@obermui.de> * rm import of userapi/storage/accounts/sqlite3/storage.go
2020-12-18More sane next batch handling, typing notification tweaks, give invites ↵Neil Alexander
their own stream position, device list fix (#1641) * Update sync responses * Fix positions, add ApplyUpdates * Fix MarshalText as non-pointer, PrevBatch is optional * Increment by number of read receipts * Merge branch 'master' into neilalexander/devicelist * Tweak typing * Include keyserver position tweak * Fix typing next position in all cases * Tweaks * Fix typo * Tweaks, restore StreamingToken.MarshalText which somehow went missing? * Rely on positions from notifier rather than manually advancing them * Revert "Rely on positions from notifier rather than manually advancing them" This reverts commit 53112a62cc3bfd9989acab518e69eeb27938117a. * Give invites their own position, fix other things * Fix test * Fix invites maybe * Un-whitelist tests that look to be genuinely wrong * Use real receipt positions * Ensure send-to-device uses real positions too
2020-12-16Add event ID index on current state table (helps performance) (#1649)Neil Alexander
2020-12-11Give receipts their own stream ID in the database (#1631)Neil Alexander
* Give read recipts their own database sequence * Give receipts their own stream ID * Change migration names * Reset sequences * Add max receipt queries, missing stream_id table entry for SQLite
2020-12-10Refactor sync tokens (#1628)Neil Alexander
* Refactor sync tokens * Comment out broken notifier test * Update types, sytest-whitelist * More robust token checking * Remove New functions for streaming tokens * Export Logs in StreamingToken * Fix tests
2020-12-09Don't recalculate event ID so often in sync (#1624)Neil Alexander
* Don't bail so quickly in fetchMissingStateEvents * Don't recalculate event IDs so often in sync API * Add comments * Fix comments * Update to matrix-org/gomatrixserverlib@eb6a890
2020-12-03Peeking updates (#1607)Neil Alexander
* Add unpeek * Don't allow peeks into encrypted rooms * Fix send tests * Update consumers
2020-12-02Top-level setup package (#1605)Neil Alexander
* Move config, setup, mscs into "setup" top-level folder * oops, forgot the EDU server * Add setup * goimports
2020-11-16Pass pointers to events — reloaded (#1583)Neil Alexander
* Pass events as pointers * Fix lint errors * Update gomatrixserverlib * Update gomatrixserverlib * Update to matrix-org/gomatrixserverlib#240
2020-11-09Implement read receipts (#1528)S7evinK
* fix conversion from int to string yields a string of one rune, not a string of digits * Add receipts table to syncapi * Use StreamingToken as the since value * Add required method to testEDUProducer * Make receipt json creation "easier" to read * Add receipts api to the eduserver * Add receipts endpoint * Add eduserver kafka consumer * Add missing kafka config * Add passing tests to whitelist Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Fix copy & paste error * Fix column count error * Make outbound federation receipts pass * Make "Inbound federation rejects receipts from wrong remote" pass * Don't use errors package * - Add TODO for batching requests - Rename variable * Return a better error message * - Use OutputReceiptEvent instead of InputReceiptEvent as result - Don't use the errors package for errors - Defer CloseAndLogIfError to close rows - Fix Copyright * Better creation/usage of JoinResponse * Query all joined rooms instead of just one * Update gomatrixserverlib * Add sqlite3 migration * Add postgres migration * Ensure required sequence exists before running migrations * Clarification on comment * - Fix a bug when creating client receipts - Use concrete types instead of interface{} * Remove dead code Use key for timestamp * Fix postgres query... * Remove single purpose struct * Use key/value directly * Only apply receipts on initial sync or if edu positions differ, otherwise we'll be sending the same receipts over and over again. * Actually update the id, so it is correctly send in syncs * Set receipt on request to /read_markers * Fix issue with receipts getting overwritten * Use fmt.Errorf instead of pkg/errors * Revert "Add postgres migration" This reverts commit 722fe5a04628882b787d096942459961db159b06. * Revert "Add sqlite3 migration" This reverts commit d113b03f6495a4b8f8bcf158a3d00b510b4240cc. * Fix selectRoomReceipts query * Make golangci-lint happy Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-10-22Fix RewritesState bug (#1557)Neil Alexander
* Set RewritesState once * Check if any new state provided * Obey rewritesState * Don't nuke everything the sync API knows when purging state * Fix panic from duplicate insert * Consistency * Use HasState * Remove nolint * Clean up joined rooms on state rewrite
2020-10-02Add history visibility guards (#1470)Kegsay
* Add history visibility guards Default to 'joined' visibility to avoid leaking events, until we get around to implementing history visibility completely. Related #617 * Don't apply his vis checks on shared rooms * Fix order of checks * Linting and remove another misleading check * Update whitelist
2020-10-02Fix initial sync (#1465)Neil Alexander
* Fix complete sync check * Remove unnecessary 'since' copy * Fix failing test * Un-whitelist a couple of tests Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-10-01Fix golangci-lint issues (#1464)S7evinK
* Fix S1039: unnecessary use of fmt.Sprintf * Fix S1036: unnecessary guard around map access Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-09-24Update all usages of tx.Stmt to sqlutil.TxStmt (#1423)Sam
* Replace all usages of txn.Stmt with sqlutil.TxStmt Signed-off-by: Sam Day <me@samcday.com> * Fix sign off link in PR template. Signed-off-by: Sam Day <me@samcday.com> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-09-15Handle state with input event as new events (#1415)Neil Alexander
* SendEventWithState events as new * Use cumulative state IDs for final event * Error wrapping in calculateAndSetState * Handle overwriting same event type and state key * Hacky way to spot historical events * Don't exclude from sync * Don't generate output events when rewriting forward extremities * Update output event check * Historical output events * Define output room event type * Notify key changes on state * Don't send our membership event twice * Deduplicate state entries * Tweaks * Remove unnecessary nolint * Fix current state upsert in sync API * Send auth events as outliers, state events as rewrite * Sync API don't consume state events * Process events actually * Improve outlier check * Fix local room check * Remove extra room check, it seems to break the whole damn world * Fix federated join check * Fix nil pointer exception * Better comments on DeduplicateStateEntries * Reflow forced federated joins * Don't force federated join for possibly even local invites * Comment SendEventWithState better * Rewrite room state in sync API storage * Add TODO * Clean up all room data when receiving create event * Don't generate output events for rewrites, but instead notify that state is rewritten on the final new event * Rename to PurgeRoom * Exclude backfilled messages from /sync * Split out rewriting state from updating state from state res Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-10Peeking via MSC2753 (#1370)Matthew Hodgson
Initial implementation of MSC2753, as tested by https://github.com/matrix-org/sytest/pull/944. Doesn't yet handle unpeeks, peeked EDUs, or history viz changing during a peek - these will follow. https://github.com/matrix-org/dendrite/pull/1370 has full details.
2020-09-04Sync bug fixes (#1394)Neil Alexander
* Sync bug fixes * Remove logging
2020-09-01Put redactions/filters in the writer goroutine (#1378)Kegsay
* Put redactions in the writer goroutine * Update filters on writer goroutine
2020-09-01Fix duplicate writers (#1376)Neil Alexander
* Fix writers * Don't use writers in both shared and sqlite3
2020-09-01Storage tweaks (#1373)Neil Alexander
* Sync API tweaks * User API tweaks
2020-09-01Remove unused SyncStreamPositionNeil Alexander
2020-08-28Add some error wrapping to sync API, use background context for sync (#1363)Neil Alexander
* Add some error wrapping to sync API * Don't use request context for BeginTx until mattn/go-sqlite3#764 is fixed
2020-08-25[readability] use event.StateKeyEquals where relevant and minor for-loop ↵oliverpool
refactoring (#1339) * use event.StateKeyEquals where relevant Signed-off-by: Olivier Charvin <git@olivier.pfad.fr> * use userID
2020-08-25Fix 'Invited user can reject invite over federation several times' (#1341)Kegsay
2020-08-25Don't overwrite global err before return (#1293)oliverpool
Signed-off-by: Olivier Charvin <git@olivier.pfad.fr>
2020-08-21Component-wide TransactionWriters (#1290)Neil Alexander
* Offset updates take place using TransactionWriter * Refactor TransactionWriter in current state server * Refactor TransactionWriter in federation sender * Refactor TransactionWriter in key server * Refactor TransactionWriter in media API * Refactor TransactionWriter in server key API * Refactor TransactionWriter in sync API * Refactor TransactionWriter in user API * Fix deadlocking Sync API tests * Un-deadlock device database * Fix appservice API * Rename TransactionWriters to Writers * Move writers up a layer in sync API * Document sqlutil.Writer interface * Add note to Writer documentation
2020-08-21Select distinct on room memberships in sync API (#1292)Neil Alexander
2020-08-19Transaction writer changes, move roomserver writers (#1285)Neil Alexander
* Updated TransactionWriters, moved locks in roomserver, various other tweaks * Fix redaction deadlocks * Fix lint issue * Rename SQLiteTransactionWriter to ExclusiveTransactionWriter * Fix us not sending transactions through in latest events updater
2020-08-10Configuration format v1 (#1230)Neil Alexander
* Initial pass at refactoring config (not finished) * Don't forget current state and EDU servers * More shifting around * Update server key API tests * Fix roomserver test * Fix more tests * Further tweaks * Fix current state server test (sort of) * Maybe fix appservices * Fix client API test * Include database connection string in database options * Fix sync API build * Update config test * Fix unit tests * Fix federation sender build * Fix gobind build * Set Listen address for all services in HTTP monolith mode * Validate config, reinstate appservice derived in directory, tweaks * Tweak federation API test * Set MaxOpenConnections/MaxIdleConnections to previous values * Update generate-config
2020-07-29Add support for logs in StreamingToken (#1229)Kegsay
* Add support for logs in StreamingToken Tokens now end up looking like `s11_22|dl-0-123|ab-0-12224` where `dl` and `ab` are log names, `0` is the partition and `123` and `12224` are the offsets. * Also test reserialisation * s/|/./g so tokens url escape nicely
2020-07-27Don't use more than 999 variables in SQLite querys. (#1224)Henrik Sölver
Closes #1223 Signed-off-by: Henrik Sölver <henrik.solver@gmail.com> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-07-21Use TransactionWriter in other component SQLite (#1209)Neil Alexander
* Use TransactionWriter on other component SQLites * Fix sync API tests * Fix panic in media API * Fix a couple of transactions * Fix wrong query, add some logging output * Add debug logging into StoreEvent * Adjust InsertRoomNID * Update logging
2020-07-20Deduplicate FS database, EDU persistence table (#1207)Neil Alexander
* Deduplicate FS database, add some EDU persistence groundwork * Extend TransactionWriter to use optional existing transaction, use that for FS SQLite database writes * Fix build due to bad keyserver import * Working EDU persistence * gocyclo, unsurprisingly * Remove unused * Update copyright notices
2020-07-08Finish implementing redactions (#1189)Kegsay
* Add a bit more logging to the fedsender * bugfix: continue sending PDUs if ones are added whilst sending another PDU Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't fire because `pendingPDUs` is already > 0. This should fix a flakey sytest. * Break if no txn is sent * WIP syncapi work * More debugging * Bump GMSL version to pull in working Event.Redact * Remove logging * Make redactions work on v3+ * Fix more tests
2020-06-26Implement /sync `limited` and read timeline limit from stored filters (#1168)Kegsay
* Move filter table to syncapi where it is used * Implement /sync `limited` and read timeline limit from stored filters We now fully handle `room.timeline.limit` filters (in-line + stored) and return the right value for `limited` syncs. * Update whitelist * Default to the default timeline limit if it's unset, also strip the extra event correctly * Update whitelist
2020-06-26Finish implementing retiring invites (#1166)Kegsay
* Pass retired invites to the syncapi with the event ID of the invite * Implement retire invite streaming * Update whitelist
2020-06-16Make userapi responsible for checking access tokens (#1133)Kegsay
* Make userapi responsible for checking access tokens There's still plenty of dependencies on account/device DBs, but this is a start. This is a breaking change as it adds a required config value `listen.user_api`. * Cleanup * Review comments and test fix
2020-06-12Rehuffle where things are in the internal package (#1122)Kegsay
renamed: internal/eventcontent.go -> internal/eventutil/eventcontent.go renamed: internal/events.go -> internal/eventutil/events.go renamed: internal/types.go -> internal/eventutil/types.go renamed: internal/http/http.go -> internal/httputil/http.go renamed: internal/httpapi.go -> internal/httputil/httpapi.go renamed: internal/httpapi_test.go -> internal/httputil/httpapi_test.go renamed: internal/httpapis/paths.go -> internal/httputil/paths.go renamed: internal/routing.go -> internal/httputil/routing.go renamed: internal/basecomponent/base.go -> internal/setup/base.go renamed: internal/basecomponent/flags.go -> internal/setup/flags.go renamed: internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go renamed: internal/postgres.go -> internal/sqlutil/postgres.go renamed: internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go renamed: internal/sql.go -> internal/sqlutil/sql.go
2020-06-04Add ParseFileURI and use it when dealing with file URIs (#1088)Kegsay
* Add ParseFileURI and use it when dealing with file URIs Fixes #1059 * Missing file * Linting
2020-06-01Send-to-device support (#1072)Neil Alexander
* Groundwork for send-to-device messaging * Update sample config * Add unstable routing for now * Send to device consumer in sync API * Start the send-to-device consumer * fix indentation in dendrite-config.yaml * Create send-to-device database tables, other tweaks * Add some logic for send-to-device messages, add them into sync stream * Handle incoming send-to-device messages, count them with EDU stream pos * Undo changes to test * pq.Array * Fix sync * Logging * Fix a couple of transaction things, fix client API * Add send-to-device test, hopefully fix bugs * Comments * Refactor a bit * Fix schema * Fix queries * Debug logging * Fix storing and retrieving of send-to-device messages * Try to avoid database locks * Update sync position * Use latest sync position * Jiggle about sync a bit * Fix tests * Break out the retrieval from the update/delete behaviour * Comments * nolint on getResponseWithPDUsForCompleteSync * Try to line up sync tokens again * Implement wildcard * Add all send-to-device tests to whitelist, what could possibly go wrong? * Only care about wildcard when targeted locally * Deduplicate transactions * Handle tokens properly, return immediately if waiting send-to-device messages * Fix sync * Update sytest-whitelist * Fix copyright notice (need to do more of this) * Comments, copyrights * Return errors from Do, fix dendritejs * Review comments * Comments * Constructor for TransactionWriter * defletions * Update gomatrixserverlib, sytest-blacklist
generated by cgit v1.2.3 (git 2.26.2) at 2025-01-11 13:55:03 +0000