aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2022-10-03Allow multiple static peers in Pinecone iOS/Android demosNeil Alexander
2022-10-03P2P demo tweaksNeil Alexander
2022-10-03Update P2P base directoriesNeil Alexander
2022-10-03Stop CPU burn in `PerformMarkAsStaleIfNeeded`Neil Alexander
2022-10-03Fix fragility of selectEventsWithEventIDsSQL queries (#2757)Ashley Nelson
This fixes a temporary workaround with the `selectEventsWithEventIDsSQL` queries where fields need to be artificially added to the queries so the row results match the format of the `syncapi_output_room_events` table. I made similar functions that accept row results from the `syncapi_current_room_state` table and convert them into StreamEvents without the fields that are specific to output room events. There is also a unit test in the first commit to ensure the resulting behavior doesn't change from the modified queries and functions. Fixes #601. ### Pull Request Checklist <!-- Please read docs/CONTRIBUTING.md before submitting your pull request --> * [x] I have added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Ashley Nelson <fant@shley.email>` Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-03Hopefully fix P2P `--config` error (re. #2756)Neil Alexander
2022-10-03Modify sync transaction behaviour (#2758)Neil Alexander
This now uses a transaction per stream, so that errors in one stream don't propagate to another, and we therefore no longer need to do hacks to reopen a new transaction after aborting a failed one.
2022-10-02Use non-HTTPS as default URL, as most people will be running behind aTill Faelligen
reverse proxy
2022-09-30Version 0.10.1v0.10.1Neil Alexander
2022-09-30Reset transaction after a failureNeil Alexander
2022-09-30Handle case when applying history visibility failedNeil Alexander
2022-09-30Another `/sync` fixNeil Alexander
2022-09-30Remove SendAccountData, since InputAccountData is already doing thatTill Faelligen
2022-09-30Update NATS Server to v2.9.2Neil Alexander
2022-09-30Fix broken `/sync` due to transaction errorNeil Alexander
2022-09-30Update 2_domainname.mdNeil Alexander
2022-09-30Version 0.10.0 (#2753)v0.10.0Neil Alexander
Changelog and version bump.
2022-09-30Transactional isolation for `/sync` (#2745)Neil Alexander
This should transactional snapshot isolation for `/sync` etc requests. For now we don't use repeatable read due to some odd test failures with invites.
2022-09-30Allow more time for device list updates (#2749)Neil Alexander
This updates the device list updater so that it has a context per-request, rather than a global 30 seconds for the entire server. This could mean that talking to a slow remote server or requesting a lot of user IDs was pretty much guaranteed to fail. It also uses the process context to allow correct cancellation when Dendrite wants to shut down cleanly.
2022-09-30Add `/_dendrite/admin/refreshDevices/{userID}` (#2746)Till
Allows to immediately query `/devices/{userID}` over federation to (hopefully) resolve E2EE issues.
2022-09-29Enable knocking on room versions 8 and 9 (update to ↵Neil Alexander
matrix-org/gomatrixserverlib@91d455c)
2022-09-29Add network interface registration to pinecone demo (#2750)devonh
### Pull Request Checklist <!-- Please read docs/CONTRIBUTING.md before submitting your pull request --> No additional tests have been added as this simply extends the API for the embedded pinecone demo. Signed-off-by: `Devon Hudson <devonhudson@librem.one>`
2022-09-29Update to matrix-org/gomatrixserverlib@377b320Neil Alexander
2022-09-29Remove `Suppressing send-to-device` log lineNeil Alexander
The behaviour is correct and I have no idea why we are logging it.
2022-09-29Improve device list update parsing (update to matrix-org/gomatrixserverlib#342)Neil Alexander
2022-09-29Fix Go 1.18Neil Alexander
2022-09-29P2P demo fixesNeil Alexander
2022-09-28Consistent `*sql.Tx` usage across sync API (#2744)Neil Alexander
This tidies up the `storage` package so that everything takes a transaction parameter instead of something things that do and some that don't.
2022-09-28Fix for `sql: converting argument $1 type: unsupported type []interfa… (#2743)texuf
…ce {}, a slice of interface` in new notifications select The sqlite3 version was just not working, original pr here: https://github.com/matrix-org/dendrite/pull/2688 signed off by: austin ellis <austin@hntlabs.com> This doesn't fix the notification counts, they still only work about 1 out of every 5 times in my tests. I will stick with my other fix locally for reliable notification delivery: https://github.com/matrix-org/dendrite/pull/2701
2022-09-27Promote reindexing log levelNeil Alexander
2022-09-27Update search docsNeil Alexander
2022-09-27Fulltext implementation using Bleve (#2675)Till
Based on #2480 This actually indexes events based on their event type. They are removed from the index if we receive a `m.room.redaction` event on the `OutputRoomEvent` stream. An admin endpoint is added to reindex all existing events. Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27Return `M_UNRECOGNIZED` for unknown CS API endpoints/actions (#2740)Neil Alexander
Fixes #2739.
2022-09-27Refactor notifications (#2688)Till
This PR changes the handling of notifications - removes the `StreamEvent` and `ReadUpdate` stream - listens on the `OutputRoomEvent` stream in the UserAPI to inform the SyncAPI about unread notifications - listens on the `OutputReceiptEvent` stream in the UserAPI to set receipts/update notifications - sets the `read_markers` directly from within the internal UserAPI Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27Pinecone hybrid routing (update to matrix-org/pinecone#67)Neil Alexander
2022-09-27Build Docker images using Go 1.19 (related to #2714)Neil Alexander
2022-09-27Use /usr/bin/env bash in shebangs to make them universal (#2735)Dov Alperin
Some systems (like nixos) don't have bash living at `/bin/bash` so using `/usr/bin/env bash` we can make these scripts universal. ### Pull Request Checklist <!-- Please read docs/CONTRIBUTING.md before submitting your pull request --> * [X] I have added added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Dov Alperin <git@dov.dev>` Signed-off-by: `Dov Alperin <git@dov.dev>`
2022-09-27Improve selectRoomIDsWithAnyMembershipSQL performance (#2738)PiotrKozimor
Recently I have observed that dendrite spends a lot of time (~390s) in `selectRoomIDsWithAnyMembershipSQL` query ``` dendrite_syncapi=# select total_exec_time, left(query,100) from pg_stat_statements order by total_exec_time desc limit 5 ; total_exec_time | left --------------------+------------------------------------------------------------------------------------------------------ 747826.5800519128 | SELECT event_id, id, headered_event_json, session_id, exclude_from_sync, transaction_id, history_vis 389130.5490339942 | SELECT DISTINCT room_id, membership FROM syncapi_current_room_state WHERE type = $2 AND state_key = 376104.17514700035 | SELECT psd.datname, xact_commit, xact_rollback, blks_read, blks_hit, tup_returned, tup_fetched, tup_ 363644.164092031 | SELECT event_type_nid, event_state_key_nid, event_nid FROM roomserver_events WHERE event_nid = ANY($ 58570.48104699995 | SELECT event_id, headered_event_json FROM syncapi_current_room_state WHERE room_id = $1 AND ( $2::te (5 rows) ``` Explain analyze showed correct usage of `syncapi_room_state_unique` index: ``` dendrite_syncapi=# explain analyze SELECT distinct room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com'; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Unique (cost=2749.38..2749.56 rows=24 width=52) (actual time=2.933..2.956 rows=65 loops=1) -> Sort (cost=2749.38..2749.44 rows=24 width=52) (actual time=2.932..2.937 rows=65 loops=1) Sort Key: room_id, membership Sort Method: quicksort Memory: 34kB -> Index Scan using syncapi_room_state_unique on syncapi_current_room_state (cost=0.41..2748.83 rows=24 width=52) (actual time=0.030..2.890 rows=65 loops=1) Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Planning Time: 0.140 ms Execution Time: 2.990 ms (8 rows) ``` Multi-column indexes in Postgres shall perform well for leftmost columns, but I gave it a try and created `syncapi_current_room_state_type_state_key_idx` index. I could observe significant performance improvement. Execution time dropped from 2.9 ms to 0.24 ms: ``` explain analyze SELECT distinct room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com'; QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unique (cost=96.46..96.64 rows=24 width=52) (actual time=0.199..0.218 rows=65 loops=1) -> Sort (cost=96.46..96.52 rows=24 width=52) (actual time=0.199..0.202 rows=65 loops=1) Sort Key: room_id, membership Sort Method: quicksort Memory: 34kB -> Bitmap Heap Scan on syncapi_current_room_state (cost=4.53..95.91 rows=24 width=52) (actual time=0.048..0.139 rows=65 loops=1) Recheck Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Heap Blocks: exact=59 -> Bitmap Index Scan on syncapi_current_room_state_type_state_key_idx (cost=0.00..4.53 rows=24 width=0) (actual time=0.037..0.037 rows=65 loops=1) Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Planning Time: 0.236 ms Execution Time: 0.242 ms (11 rows) ``` Next improvement is skipping DISTINCT and rely on map assignment in `SelectRoomIDsWithAnyMembership`. Execution time drops by almost half: ``` explain analyze SELECT room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com'; QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------- Bitmap Heap Scan on syncapi_current_room_state (cost=4.53..95.91 rows=24 width=52) (actual time=0.032..0.113 rows=65 loops=1) Recheck Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Heap Blocks: exact=59 -> Bitmap Index Scan on syncapi_current_room_state_type_state_key_idx (cost=0.00..4.53 rows=24 width=0) (actual time=0.021..0.021 rows=65 loops=1) Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text)) Planning Time: 0.087 ms Execution Time: 0.136 ms (7 rows) ``` In our env we spend only 1s on inserting to table, so the write penalty of creating an index should be small. ``` dendrite_syncapi=# select total_exec_time, left(query,100) from pg_stat_statements where query like '%INSERT%syncapi_current_room_state%' order by total_exec_time desc; total_exec_time | left --------------------+------------------------------------------------------------------------------------------------------ 1139.9057619999971 | INSERT INTO syncapi_current_room_state (room_id, event_id, type, sender, contains_url, state_key, he (1 row) ``` This PR does not require test modifications. ### Pull Request Checklist <!-- Please read docs/CONTRIBUTING.md before submitting your pull request --> * [x] I have added added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: `Piotr Kozimor <p1996k@gmail.com>`
2022-09-27Add pinecone demo container image (#2710)networkException
This pull request adds the configuration and CI steps to build and publish a container wrapping the `dendrite-demo-pinecone` command as well as fixes a sentence structure issue in the pull request template. As this does not touch any go source code no tests have been added ### Pull Request Checklist <!-- Please read docs/CONTRIBUTING.md before submitting your pull request --> * [x] I have added tests for PR _or_ I have justified why this PR doesn't need tests. * [x] Pull request includes a [sign off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off) Signed-off-by: networkException <git@nwex.de> (by private sign-off) Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-26Remove `origin` field from PDUs (#2737)Neil Alexander
This nukes the `origin` field from PDUs as per matrix-org/matrix-spec#998, matrix-org/gomatrixserverlib#341.
2022-09-26Use `TxStmt` in SQLite pusher tableNeil Alexander
2022-09-26Fix possible "Database is locked" issueTill Faelligen
2022-09-26Update documentation to state that Dendrite requires PostgreSQL UTF-8 encodingNeil Alexander
2022-09-23Update to matrix-org/pinecone@0900fceecb89ad0c14ee0ce825be638bf2a18474Neil Alexander
2022-09-23Add `-dir` option to `dendrite-demo-pinecone` and `dendrite-demo-yggdrasil`Neil Alexander
2022-09-23Tweak mainline ordering (update to matrix-org/gomatrixserverlib@2217f6c)Neil Alexander
2022-09-22Get the `DeviceListPosition` before anything else in complete syncs (#2733)Till
This should hopefully unflake `Can query remote device keys using POST` in Complement.
2022-09-22Version 0.9.9 (#2732)v0.9.9Neil Alexander
Changelog and version bump.
2022-09-22Tweak `InsertMigration` to avoid logging (#2720)Till
We'd still produce logs in Postgres when trying to insert a migration we already ran. This should stop us from creating those log entries.
2022-09-22Update embedded NATS Server to v2.9.1-beta1 (as this includes a fix that ↵Neil Alexander
prevents high CPU usage after Dendrite startup)
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-18 03:40:35 +0000