aboutsummaryrefslogtreecommitdiff
path: root/mediaapi
AgeCommit message (Collapse)Author
2022-05-02Return `null` if MaxFileSizeBytes is 0 (#2409)Till
* Return "null" if MaxFileSizeBytes is 0 * Add comment and nil check (better save than sorry) * Simplify config
2022-04-29Fix `TestThumbnailsStorage` failing when media results come back in ↵Brian Meek
non-deterministic order; silence expected error when tests are run multiple times against the same postgres database (cherry-picked from #2395) Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-04-14Refactor media storage layer, add tests (#2352)Till
* Refactor mediaapi storage layer * Verify filetype before trying to create thumbnails * Add media api storage tests * Fix returned values
2022-04-04Fix unlimitedSize uploads (#2317)S7evinK
2022-03-18Fix panic that can occur from nil pointer exception in media API ↵Neil Alexander
`fetchRemoteFile`
2022-03-16Return error from remote download requests (#2266)S7evinK
* Return error from remote download requests * Remove redundant err check, return better message if file does not exist
2022-03-04Un-ratelimit calls to /thumbnail (#2251)S7evinK
2022-03-02Media endpoints on `/v3` (#2242)Neil Alexander
* Media endpoints on `/v3` * Keep `/v1` too?
2022-01-31Tweak some logging (#2130)Neil Alexander
* Modify some log levels * Update gomatrixserverlib to matrix-org/gomatrixserverlib@336334f * Update gomatrixserverlib to matrix-org/gomatrixserverlib@cde7ac8 * Demote warning about key change producer * Add more useful roomserver logging * Further tweaking
2021-11-24Ratelimit requests to /media/r0/download|upload (#2020)S7evinK
* Add /media/r0/config handler Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Add rate limiting to media api * Rename variable * Add passing tests * Don't send multiple headers Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-11-02Run gofmt on dendrite - apply go 1.17 preferred build tags (#2021)PiotrKozimor
2021-11-020️⃣ Media API: Fix maxFileSizeBytes=0 handling (#1993)database64128
2021-11-02Support WebP based thumbnails (#2033)hacktivista
Co-authored-by: Felix Freeman <libsys@hacktivista.org>
2021-09-08- Removed double imports (#1989)Ryan W
- Lower cased error messages Signed-off-by: Ryan Whittington <twentybitdev@gmail.com> Co-authored-by: kegsay <kegan@matrix.org>
2021-07-20Only include go-sqlite3 on the relevant binaries (#1900)Neil Alexander
* Only include go-sqlite3 on the relevant binaries * The driver name is always sqlite3 now * Update to matrix-org/go-sqlite3-js@e537baa
2021-07-19Set MaxFileSizeBytes <= 0 to unlimited (#1876)S7evinK
* Revert "Set MaxFileSizeBytes <= 0 to "unlimited" (#1875)" This reverts commit 9ed044042156a9bdfafb4ae9830b3ed41225444e. * Actually allow unlimited upload Signed-off-by: Till Faelligen <tfaelligen@gmail.com> Co-authored-by: kegsay <kegan@matrix.org>
2021-06-22✂️ Media API: Handle unlimited file size (#1881)database64128
2021-06-14Set MaxFileSizeBytes <= 0 to "unlimited" (#1875)S7evinK
* Set MaxFileSizeBytes < 0 to "unlimited" Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * int64 overflows later in mediaapi/routing/upload.go[doUpload] * Prevent int overflow when uploading
2021-06-07Use LimitReader to prevent DoS risk (#1843)S7evinK
* Use LimitReader to prevent DoS risk Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Check if bytesWritten is equal to the maxFileSize Add tests Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Use oldschool defer to cleanup after the tests * Let LimitReader read MaxFileSizeBytes + 1 Co-authored-by: Kegsay <kegan@matrix.org>
2021-04-19Remove the 'Content-Type' request header requirement (#1834)Fero
2021-04-14Remove the Content-Length requirement on upload request (#1831)Fero
* Remove the Content-Length requirement * Make sure that the file size does not exceed max content size * Address review comment - universally check if temp file size exceeds max file size
2021-02-17Retrieve remote file size without the Content-Size header (#1537)darkgallium
* Get downloaded file size regardless of the Content-Size header Signed-off-by: Florian Le Minoux <darkgallium+git@deuxfleurs.fr> * fixing lint issues * Don't exhaust memory for large files, don't limit more than necessary * Don't use errors.Wrap in download.go Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-12-02Top-level setup package (#1605)Neil Alexander
* Move config, setup, mscs into "setup" top-level folder * oops, forgot the EDU server * Add setup * goimports
2020-08-28Fix #1361 (#1362)Neil Alexander
2020-08-26Port of #945 (#1343)Kegsay
- Make sure we always cleanup the temp directory on error. - Complain about it having an error prone API shape.
2020-08-25Randomly generated media IDs (#1336)Neil Alexander
* Filenames based on hashes but media IDs randomly generated and per-upload metadata * Lint * Comment generateMediaID
2020-08-21Component-wide TransactionWriters (#1290)Neil Alexander
* Offset updates take place using TransactionWriter * Refactor TransactionWriter in current state server * Refactor TransactionWriter in federation sender * Refactor TransactionWriter in key server * Refactor TransactionWriter in media API * Refactor TransactionWriter in server key API * Refactor TransactionWriter in sync API * Refactor TransactionWriter in user API * Fix deadlocking Sync API tests * Un-deadlock device database * Fix appservice API * Rename TransactionWriters to Writers * Move writers up a layer in sync API * Document sqlutil.Writer interface * Add note to Writer documentation
2020-08-19Transaction writer changes, move roomserver writers (#1285)Neil Alexander
* Updated TransactionWriters, moved locks in roomserver, various other tweaks * Fix redaction deadlocks * Fix lint issue * Rename SQLiteTransactionWriter to ExclusiveTransactionWriter * Fix us not sending transactions through in latest events updater
2020-08-13API setup refactoring (#1266)Neil Alexander
* Start HTTP endpoint refactoring * Update SetupAndServeHTTP * Fix builds * Don't set up external listener if no address configured * TLS HTTP setup * Break apart client/federation/key/media muxes * Tweaks * Fix P2P demos * Fix media API routing * Review comments @Kegsay * Update sample config * Fix gobind build * Fix External -> Public in federation API test
2020-08-10Configuration format v1 (#1230)Neil Alexander
* Initial pass at refactoring config (not finished) * Don't forget current state and EDU servers * More shifting around * Update server key API tests * Fix roomserver test * Fix more tests * Further tweaks * Fix current state server test (sort of) * Maybe fix appservices * Fix client API test * Include database connection string in database options * Fix sync API build * Update config test * Fix unit tests * Fix federation sender build * Fix gobind build * Set Listen address for all services in HTTP monolith mode * Validate config, reinstate appservice derived in directory, tweaks * Tweak federation API test * Set MaxOpenConnections/MaxIdleConnections to previous values * Update generate-config
2020-07-21Use TransactionWriter in other component SQLite (#1209)Neil Alexander
* Use TransactionWriter on other component SQLites * Fix sync API tests * Fix panic in media API * Fix a couple of transactions * Fix wrong query, add some logging output * Add debug logging into StoreEvent * Adjust InsertRoomNID * Update logging
2020-07-08Yet another v1 vs r0 media endpoint (#1190)Neil Alexander
2020-06-17Fallback parsing for Content-Disposition minefield (#1144)Neil Alexander
2020-06-17Media filename handling improvements (#1140)Neil Alexander
* Derive content ID from hash+filename but preserve dedupe, improve Content-Disposition handling and ASCII handling * Linter fix * Some more comments * Update sytest-whitelist
2020-06-16Update whitelist for sytest media fix (#1137)Neil Alexander
* Update sytest-whitelist, are-we-synapse-yet.list * Update gomatrixserverlib * Update gomatrixserverlib * Loop avoidance * Return UTF-8 filenames * Replace quotes only, instead of using strconv.Quote * Update sytest-whitelist * Update sytest-whitelist
2020-06-16Fix media API for demos and possibly Synapse (#1134)Neil Alexander
* Fix media API for demos and possibly Synapse * User API * goimports
2020-06-16Make userapi responsible for checking access tokens (#1133)Kegsay
* Make userapi responsible for checking access tokens There's still plenty of dependencies on account/device DBs, but this is a start. This is a breaking change as it adds a required config value `listen.user_api`. * Cleanup * Review comments and test fix
2020-06-12Rehuffle where things are in the internal package (#1122)Kegsay
renamed: internal/eventcontent.go -> internal/eventutil/eventcontent.go renamed: internal/events.go -> internal/eventutil/events.go renamed: internal/types.go -> internal/eventutil/types.go renamed: internal/http/http.go -> internal/httputil/http.go renamed: internal/httpapi.go -> internal/httputil/httpapi.go renamed: internal/httpapi_test.go -> internal/httputil/httpapi_test.go renamed: internal/httpapis/paths.go -> internal/httputil/paths.go renamed: internal/routing.go -> internal/httputil/routing.go renamed: internal/basecomponent/base.go -> internal/setup/base.go renamed: internal/basecomponent/flags.go -> internal/setup/flags.go renamed: internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go renamed: internal/postgres.go -> internal/sqlutil/postgres.go renamed: internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go renamed: internal/sql.go -> internal/sqlutil/sql.go
2020-06-08Split out SetupFooComponent (#1106)Kegsay
* Split out adding HTTP routes from making internal APIs for clarity * Split out more components * Split out more things * Finish converting * internal mux for internal routes
2020-06-04Add ParseFileURI and use it when dealing with file URIs (#1088)Kegsay
* Add ParseFileURI and use it when dealing with file URIs Fixes #1059 * Missing file * Linting
2020-05-22dendritejs tweaks for persisting sqlite DBs (#1058)Kegsay
* Use uri.path so we don't have file: in the filename * New go-sqlite-js version
2020-05-22Separate muxes for public and internal APIs (#1056)Neil Alexander
* Separate muxes for public and internal APIs * Update client-api-proxy and federation-api-proxy so they don't add /api to the path * Tidy up * Consistent HTTP setup * Set up prefixes properly
2020-05-21Fix #897 and shuffle directory around (#1054)Kegsay
* Fix #897 and shuffle directory around * Update find-lint * goimports Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-13Enforce `mediaIDRegex` to be only valid `mediaIDCharacters` (#1020)Sid Karunaratne
Error messages indicate that: > mediaId must be a non-empty string using only characters in `mediaIDCharacters` However the regex used only required that some characters in the filename match the restriction, not that the entire filename does. This commit ensures that the filename must entirely fullfill the `mediaIDCharacters` restriction Signed-off-by: Sid Karunaratne <sid@karunaratne.net> Co-authored-by: Kegsay <kegan@matrix.org>
2020-05-11Don't return 500s from media API download requestsNeil Alexander
2020-05-01Limit database connections (#980, #564) (#998)Neil Alexander
* Limit database connections (#564) - Add new options to the config file database: max_open_conns: 100 max_idle_conns: 2 conn_max_lifetime: -1 - Implement connection parameter setup on the *DB (database/sql) in internal/sqlutil/trace.go:Open() - Propagate the values in the form of DbProperties interface via all the Open() and NewDatabase() functions Signed-off-by: Tomas Jirka <tomas.jirka@email.cz> * Fix wasm builds * Remove file accidentally added from working tree Co-authored-by: Tomas Jirka <tomas.jirka@email.cz>
2020-04-16Add SQL tracing via DENDRITE_TRACE_SQL (#968)Kegsay
* Add SQL tracing via DENDRITE_TRACE_SQL Add this to `internal/sqlutil` in preparation for #897 * Not entirely
2020-03-18maintenance: Fix matrix-org#896 use %w format verb to wrap errors (#916)Abhinav Krishna C K
* maintenance: Fix matrix-org#896 use %w format verb to wrap errors * In Go version 1.13 a new formatting verb introduced for fmt.Errorf %w https://blog.golang.org/go1.13-errors * update %s to %w to wrap errors. * Update all instances of error type to use %w Signed-off-by: Abhinav Krishna C K <me@abhy.me> Co-authored-by: Kegsay <kegan@matrix.org>
2020-03-18Log errors from rows.Close (#920)Prateek Sachan
* Log errors from rows.Close * fixed imports * Added contextual messages * fixed review changes
2020-03-06Add peer-to-peer support into Dendrite via libp2p and fetch (#880)Kegsay
* Use a fork of pq which supports userCurrent on wasm * Use sqlite3_js driver when running in JS * Add cmd/dendritejs to pull in sqlite3_js driver for wasm only * Update to latest go-sqlite-js version * Replace prometheus with a stub. sigh * Hard-code a config and don't use opentracing * Latest go-sqlite3-js version * Generate a key for now * Listen for fetch traffic rather than HTTP * Latest hacks for js * libp2p support * More libp2p * Fork gjson to allow us to enforce auth checks as before Previously, all events would come down redacted because the hash checks would fail. They would fail because sjson.DeleteBytes didn't remove keys not used for hashing. This didn't work because of a build tag which included a file which no-oped the index returned. See https://github.com/tidwall/gjson/issues/157 When it's resolved, let's go back to mainline. * Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157 * Use latest gomatrixserverlib for sig checks * Fix a bug which could cause exclude_from_sync to not be set Caused when sending events over federation. * Use query variadic to make lookups actually work! * Latest gomatrixserverlib * Add notes on getting p2p up and running Partly so I don't forget myself! * refactor: Move p2p specific stuff to cmd/dendritejs This is important or else the normal build of dendrite will fail because the p2p libraries depend on syscall/js which doesn't work on normal builds. Also, clean up main.go to read a bit better. * Update ho-http-js-libp2p to return errors from RoundTrip * Add an LRU cache around the key DB We actually need this for P2P because otherwise we can *segfault* with things like: "runtime: unexpected return pc for runtime.handleEvent" where the event is a `syscall/js` event, caused by spamming sql.js caused by "Checking event signatures for 14 events of room state" which hammers the key DB repeatedly in quick succession. Using a cache fixes this, though the underlying cause is probably a bug in the version of Go I'm on (1.13.7) * breaking: Add Tracing.Enabled to toggle whether we do opentracing Defaults to false, which is why this is a breaking change. We need this flag because WASM builds cannot do opentracing. * Start adding conditional builds for wasm to handle lib/pq The general idea here is to have the wasm build have a `NewXXXDatabase` that doesn't import any postgres package and hence we never import `lib/pq`, which doesn't work under WASM (undefined `userCurrent`). * Remove lib/pq for wasm for syncapi * Add conditional building to remaining storage APIs * Update build script to set env vars correctly for dendritejs * sqlite bug fixes * Docs * Add a no-op main for dendritejs when not building under wasm * Use the real prometheus, even for WASM Instead, the dendrite-sw.js must mock out `process.pid` and `fs.stat` - which must invoke the callback with an error (e.g `EINVAL`) in order for it to work: ``` global.process = { pid: 1, }; global.fs.stat = function(path, cb) { cb({ code: "EINVAL", }); } ``` * Linting
generated by cgit v1.2.3 (git 2.26.2) at 2024-12-19 02:37:57 +0000