aboutsummaryrefslogtreecommitdiff
path: root/keyserver/storage/postgres
AgeCommit message (Collapse)Author
2023-02-20Merge keyserver & userapi (#2972)Till
As discussed yesterday, a first draft of merging the keyserver and the userapi.
2022-12-12Cleanup stale device lists for users we don't share a room with anymore (#2857)Till
The stale device lists table might contain entries for users we don't share a room with anymore. This now asks the roomserver about left users and removes those entries from the table. Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-11-07Add custom build flag to satisfy SytestTill Faelligen
2022-10-24Fix slow querying of cross-signing signaturesNeil Alexander
2022-10-07Add test for `QueryDeviceMessages` (#2773)Till
Adds tests for `QueryDeviceMessages` and also includes some optimizations to reduce allocations in the DB layer.
2022-09-13Check unique constraint errors when manually inserting migrations (#2712)Till
This should avoid unnecessary logging on startup if the migration (were we need `InsertMigration`) was already executed. This now checks for "unique constraint errors" for SQLite and Postgres and fails the startup process if the migration couldn't be manually inserted for some other reason.
2022-09-09Change detection of already executed migrations (#2665)Till
This changes the detection of already executed migrations for the roomserver state block and keychange refactor. It now uses schema tables provided by the database engine to check if the column was already removed. We now also store the migration in the migrations table. This should stop e.g. Postgres from logging errors like `ERROR: column "event_nid" does not exist at character 8`.
2022-08-08Fix issues with migrations not getting executed (#2628)Till
* Fix issues with migrations not getting executed * Check actual postgres error * Return error if it's not "column does not exist"
2022-07-25Update database migrations, remove goose (#2264)Till
* Add new db migration * Update migrations Remove goose * Add possibility to test direct upgrades * Try to fix WASM test * Add checks for specific migrations * Remove AddMigration Use WithTransaction Add Dendrite version to table * Fix linter issues * Update tests * Update comments, outdent if * Namespace migrations * Add direct upgrade tests, skipping over one version * Split migrations * Update go version in CI * Fix copy&paste mistake * Use contexts in migrations Co-authored-by: kegsay <kegan@matrix.org> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-05-11Fix OTK upload spam (#2448)Till
* Fix OTK spam * Update comment * Optimize selectKeysCountSQL to only return max 100 keys * Return CurrentPosition if the request timed out * Revert "Return CurrentPosition if the request timed out" This reverts commit 7dbdda964189f5542048c06ce5ffc6d4da1814e6. Co-authored-by: kegsay <kegan@matrix.org>
2022-05-09Add `(user_id, device_id)` index on OTK table (#2435)Neil Alexander
2022-05-03Global database connection pool (for monolith mode) (#2411)Neil Alexander
* Allow monolith components to share a single database pool * Don't yell about missing connection strings * Rename field * Setup tweaks * Fix panic * Improve configuration checks * Update config * Fix lint errors * Update comments
2022-04-26Fix bug when uploading device signatures (#2377)Neil Alexander
* Find the complete key ID when uploading signatures * Try that again * Try splitting the right thing * Don't do it for device keys * Refactor `QuerySignatures` * Revert "Refactor `QuerySignatures`" This reverts commit c02832a3e92569f64f180dec1555056dc8f8c3e3. * Both requested key IDs and master/self/user keys * Fix uniqueness * Try tweaking GMSL * Update GMSL again * Revert "Update GMSL again" This reverts commit bd6916cc379dd8d9e3f38d979c6550bd658938aa. * Revert "Try tweaking GMSL" This reverts commit 2a054524da9d64c6a2a5228262fbba5fde28798c. * Database migrations
2022-04-22Fix retrieving cross-signing signatures in `/user/devices/{userId}` (#2368)Neil Alexander
* Fix retrieving cross-signing signatures in `/user/devices/{userId}` We need to know the target device IDs in order to get the signatures and we weren't populating those. * Fix up signature retrieval * Fix SQLite * Always include the target's own signatures as well as the requesting user
2022-03-21Remove unused partition_offset_table (#2288)S7evinK
2022-03-1064-bit stream IDs for device list updates (#2267)Neil Alexander
2022-02-21Delete one-time keys when deleting a device (#2208)Neil Alexander
2022-02-18Merge both user API databases into one (#2186)Neil Alexander
* Merge user API databases into one * Remove DeviceDatabase from config * Fix tests * Try that again * Clean up keyserver device keys when the devices no longer exist in the user API * Tweak ordering * Fix UserExists flag, device check * Allow including empty entries so we can clean them up * Remove logging
2022-02-04Remove sarama/saramajetstream dependencies (#2138)S7evinK
* Remove dependency on saramajetstream & sarama Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Remove internal.ContinualConsumer from federationapi * Remove internal.ContinualConsumer from syncapi * Remove internal.ContinualConsumer from keyserver * Move to new Prepare function * Remove saramajetstream & sarama dependency * Delete unneeded file * Remove duplicate import * Log error instead of silently irgnoring it * Move `OffsetNewest` and `OffsetOldest` into keyserver types, change them to be more sane values * Fix comments Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-01-21Remodel how device list change IDs are created (#2098)kegsay
* Remodel how device list change IDs are created Previously we made them using the offset Kafka supplied. We don't run Kafka anymore, so now we make the SQL table assign the change ID via an AUTOINCREMENTing ID. Redesign the `keyserver_key_changes` table to have `UNIQUE(user_id)` so we don't accumulate key changes forevermore, we now have at most 1 row per user which contains the highest change ID. This needs a SQL migration. * Ensure we bump the change ID on sqlite * Actually read the DeviceChangeID not the Offset in synapi * Add SQL migrations * Prepare after migration; fixup dendrite-upgrade-test logging * Use higher version numbers; fix sqlite query to increment better * Default 0 on postgres * fixup postgres migration on fresh dendrite instances
2021-08-18Delete device keys/signatures from key server when deleting devices (#1979)Neil Alexander
* Delete device keys/signatures from key server when deleting device from user API * Move loop to within database transaction * Don't fall over deleting no rows
2021-08-11Add type field to DeviceMessage, allow fields to be nullable (#1969)Neil Alexander
2021-08-04Cross-signing storage code (#1959)Neil Alexander
2021-06-07Try to optimize SelectOneTimeKeys (#1851)S7evinK
* Try to optimize SelectOneTimeKeys Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Use pg.Array when using ANY... Co-authored-by: Kegsay <kegan@matrix.org>
2020-12-18More sane next batch handling, typing notification tweaks, give invites ↵Neil Alexander
their own stream position, device list fix (#1641) * Update sync responses * Fix positions, add ApplyUpdates * Fix MarshalText as non-pointer, PrevBatch is optional * Increment by number of read receipts * Merge branch 'master' into neilalexander/devicelist * Tweak typing * Include keyserver position tweak * Fix typing next position in all cases * Tweaks * Fix typo * Tweaks, restore StreamingToken.MarshalText which somehow went missing? * Rely on positions from notifier rather than manually advancing them * Revert "Rely on positions from notifier rather than manually advancing them" This reverts commit 53112a62cc3bfd9989acab518e69eeb27938117a. * Give invites their own position, fix other things * Fix test * Fix invites maybe * Un-whitelist tests that look to be genuinely wrong * Use real receipt positions * Ensure send-to-device uses real positions too
2020-12-02Top-level setup package (#1605)Neil Alexander
* Move config, setup, mscs into "setup" top-level folder * oops, forgot the EDU server * Add setup * goimports
2020-09-24Update all usages of tx.Stmt to sqlutil.TxStmt (#1423)Sam
* Replace all usages of txn.Stmt with sqlutil.TxStmt Signed-off-by: Sam Day <me@samcday.com> * Fix sign off link in PR template. Signed-off-by: Sam Day <me@samcday.com> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-09-08Exclude deleted keys from selectBatchDeviceKeysSQL (#1412)Neil Alexander
2020-08-25Use Writer in shared package (#1296)Neil Alexander
2020-08-12Fix more E2E sytests (#1265)Kegsay
* WIP: Eagerly sync device lists on /user/keys/query requests Also notify servers when a user's device display name changes. Few caveats: - sytest `Device deletion propagates over federation` fails - `populateResponseWithDeviceKeysFromDatabase` is called from multiple goroutines and hence is unsafe. * Handle deleted devices correctly over federation
2020-08-10Configuration format v1 (#1230)Neil Alexander
* Initial pass at refactoring config (not finished) * Don't forget current state and EDU servers * More shifting around * Update server key API tests * Fix roomserver test * Fix more tests * Further tweaks * Fix current state server test (sort of) * Maybe fix appservices * Fix client API test * Include database connection string in database options * Fix sync API build * Update config test * Fix unit tests * Fix federation sender build * Fix gobind build * Set Listen address for all services in HTTP monolith mode * Validate config, reinstate appservice derived in directory, tweaks * Tweak federation API test * Set MaxOpenConnections/MaxIdleConnections to previous values * Update generate-config
2020-08-07Finish inbound E2E device lists (#1243)Kegsay
* Add tests for device list updates * Add stale_device_lists table and use db before asking remote for device keys * Fetch remote keys if all devices are requested * Add display_name col to store remote device names Few other tweaks to make `Server correctly handles incoming m.device_list_update` pass. * Fix sqlite otk bug * Unbuffered channel to block /send causing sytest to not race anymore * Linting and fix bug whereby we didn't send updated dl tokens to the client causing a tightloop on /sync sometimes * No longer assert staleness as Update blocks on workers now * Back out tweaks * Bugfixes
2020-08-05Process inbound device list updates from federation (#1240)Kegsay
* Add InputDeviceListUpdate * Unbreak unit tests * Process inbound device list updates from federation - Persist the keys in the keyserver and produce key changes - Does not currently fetch keys from the remote server if the prev IDs are missing * Linting
2020-08-03Generate stream IDs for locally uploaded device keys (#1236)Kegsay
* Breaking: add stream_id to keyserver_device_keys table * Add tests for stream ID generation * Fix whitelist
2020-08-03Produce OTK counts in /sync response (#1235)Kegsay
* Add QueryOneTimeKeys for /sync extensions * Unbreak tests * Produce OTK counts in /sync response * Linting
2020-07-30Implement /keys/changes (#1232)Kegsay
* Implement /keys/changes And refactor QueryKeyChanges to accept a `to` offset. * Unbreak tests * Sort keys when serialising log tokens
2020-07-28Persist partition|offset|user_id in the keyserver (#1226)Kegsay
* Persist partition|offset|user_id in the keyserver Required for a query API which will be used by the syncapi which will be called when a `/sync` request comes in which will return a list of user IDs of people who have changed their device keys between two tokens. * Add tests and fix maxOffset bug * s/offset/log_offset/g because 'offset' is a reserved word in postgres
2020-07-21Implement claiming one-time keys locally (#1210)Kegsay
* Add API shape for claiming keys * Implement claiming one-time keys locally Fairly boring, nothing too special going on.
2020-07-15Implement /keys/query locally (#1204)Kegsay
* Implement /keys/query locally * Fix sqlite tests and close rows
2020-07-15Implement key uploads (#1202)Kegsay
* Add storage layer for postgres/sqlite * Return OTK counts when inserting new keys * Hook up the key DB and make a test pass * Convert postgres queries to be sqlite queries * Blacklist test due to requiring rejected events * Unbreak tests * Update blacklist
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-20 12:22:08 +0000