c5b589f2e8
Log when we respond with 404
2018-01-16 12:01:40 +00:00
a4c5e4a645
Fix thumbnailing remote files
2018-01-16 11:37:50 +00:00
1159abbdd2
Merge pull request #2767 from matrix-org/erikj/media_storage_refactor
...
Refactor MediaRepository to separate out storage
2018-01-16 10:23:50 +00:00
21bf87a146
Reinstate media download on thumbnail request
...
We need to actually download the remote media when we get a request for a
thumbnail.
2018-01-12 15:38:06 +00:00
694f1c1b18
Fix up comments
2018-01-12 15:02:46 +00:00
e21370ba54
Correctly reraise exception
2018-01-12 14:44:02 +00:00
85a4d78213
Make Responder a context manager
2018-01-12 13:32:03 +00:00
dcc8eded41
Add missing class var
2018-01-12 13:16:27 +00:00
1e4edd1717
Remove unnecessary condition
2018-01-12 11:28:32 +00:00
c6c009603c
Remove unused variables
2018-01-12 11:24:05 +00:00
4d88958cf6
Make class var local
2018-01-12 11:23:54 +00:00
227c491510
Comments
2018-01-12 11:22:41 +00:00
8f03aa9f61
Add StorageProvider concept
2018-01-09 16:16:12 +00:00
2442e9876c
Make PreviewUrlResource use MediaStorage
2018-01-09 16:15:07 +00:00
9d30a7691c
Make ThumbnailResource use MediaStorage
2018-01-09 16:15:07 +00:00
9e20840e02
Use MediaStorage for remote media
2018-01-09 16:15:07 +00:00
dd3092c3a3
Use MediaStorage for local files
2018-01-09 16:15:07 +00:00
ada470bccb
Add MediaStorage class
2018-01-09 16:15:07 +00:00
1ee787912b
Add some helper classes
2018-01-09 16:15:07 +00:00
47ca5eb882
Split out add_file_headers
2018-01-09 16:15:07 +00:00
b6c9deffda
Remove dead TODO
2018-01-09 15:53:23 +00:00
b30cd5b107
Remove dead code related to default thumbnails
2018-01-09 14:38:33 +00:00
5a4da5bf78
Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error
...
Fix error on sqlite 3.7
2017-11-27 12:25:48 +00:00
8132a6b7ac
Fix OPTIONS on preview_url
...
Fixes #2706
2017-11-23 17:52:31 +00:00
2908f955d1
Check database in has_completed_background_updates
...
so that the right thing happens on workers.
2017-11-22 18:02:15 +00:00
7098b65cb8
Fix error on sqlite 3.7
...
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.
To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.
Fixes https://github.com/matrix-org/synapse/issues/2572 .
2017-11-21 11:14:17 +00:00
5d15abb120
Bit more logging
2017-11-10 16:58:04 +00:00
46790f50cf
Cache failures in url_preview handler
...
Reshuffle the caching logic in the url_preview handler so that failures are
cached (and to generally simplify things and fix the logcontext leaks).
2017-11-10 16:50:50 +00:00
5287e57c86
Ignore noscript tags when generating URL previews
2017-10-25 20:44:34 -04:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
d03cfc4258
Fix a logcontext leak in the media repo
2017-10-23 14:34:27 +01:00
bd5718d0ad
Fix typo in thumbnail generation
2017-10-19 10:27:18 +01:00
a6245478c8
fix thumbnailing ( #2548 )
...
in commit 0e28281a
2017-10-17 12:45:33 +02:00
1b6b0b1e66
Add try/finally block to close t_byte_source
2017-10-13 15:34:08 +01:00
6b725cf56a
Remove old comment
2017-10-13 15:23:41 +01:00
2b24416e90
Don't reuse source but instead copy from primary media store to backup
2017-10-13 14:11:34 +01:00
b92a8e6e4a
PEP8
2017-10-13 13:58:57 +01:00
31aa7bd8d1
Move type into key
2017-10-13 13:47:38 +01:00
ad1911bbf4
Comment
2017-10-13 13:47:05 +01:00
c021c39cbd
Remove spurious addition
2017-10-13 13:46:53 +01:00
1f43d22397
Don't needlessly rename variable
2017-10-13 11:42:07 +01:00
a675bd08bd
Add paths back in...
2017-10-13 11:41:06 +01:00
4d7e1dde70
Remove unnecessary diff
2017-10-13 11:36:32 +01:00
ae5d18617a
Make things be absolute paths again
2017-10-13 11:35:44 +01:00
9732ec6797
s/write_to_file/write_to_file_and_backup/
2017-10-13 11:34:41 +01:00
0e28281a02
Fix up
2017-10-13 11:33:49 +01:00
505371414f
Fix up thumbnailing function
2017-10-13 11:23:53 +01:00
e3428d26ca
Fix typo
2017-10-13 10:39:59 +01:00
35332298ef
Fix up comments
2017-10-13 10:39:32 +01:00
64db043a71
Move makedirs to thread
2017-10-13 10:25:01 +01:00
b60859d6cc
Use make_deferred_yieldable
2017-10-13 10:24:19 +01:00
d76621a47b
Fix comments
2017-10-12 18:16:25 +01:00
4ae85ae121
Don't close prematurely..
2017-10-12 17:57:31 +01:00
cc505b4b5e
getvalue closes buffer
2017-10-12 17:52:30 +01:00
1259a76047
Get len before close
2017-10-12 17:39:23 +01:00
802ca12d05
Don't close file prematurely
2017-10-12 17:37:21 +01:00
e283b555b1
Copy everything to backup
2017-10-12 17:31:24 +01:00
b77a13812c
Typo
2017-10-12 15:32:32 +01:00
6dfde6d485
Remove dead code
2017-10-12 15:30:26 +01:00
c8eeef6947
Fix typos
2017-10-12 15:28:24 +01:00
67cb89fbdf
Fix typo
2017-10-12 15:23:41 +01:00
bf4fb1fb40
Basic implementation of backup media store
2017-10-12 15:20:59 +01:00
d5694ac5fa
Only log if we've removed media
2017-09-28 16:08:08 +01:00
7cc483aa0e
Clear up expired url cache every 10s
2017-09-28 13:56:53 +01:00
e1e7d76cf1
Actually assign result to variable
2017-09-28 13:55:29 +01:00
5f501ec7e2
Fix typo in url cache expiry timer
2017-09-28 12:59:01 +01:00
ace8079086
Support new and old style media id formats
2017-09-28 12:52:51 +01:00
ae79764fe5
Change expires column to expires_ts
2017-09-28 12:37:53 +01:00
9ccb4226ba
Delete expired url cache data
2017-09-28 12:18:06 +01:00
7fe8ed1787
Store URL cache preview downloads seperately
...
This makes it easier to clear old media out at a later date
2017-06-23 11:14:11 +01:00
b8b936a6ea
Add API to quarantine media
2017-06-19 17:39:21 +01:00
48d2949416
Throw exception when not retrying when downloading media
2017-06-13 10:23:14 +01:00
836d5c44b6
actually trim oversize og:description meta
2017-05-22 21:14:20 +01:00
d12ae7fd1c
Don't log exceptions for NotRetryingDestination
2017-05-15 15:42:18 +01:00
1d09586599
Address review comments
...
- don't blindly proxy all HTTPRequestExceptions
- log unexpected exceptions at error
- avoid `isinstance`
- improve docs on `from_http_response_exception`
2017-03-14 14:15:37 +00:00
170ccc9de5
Fix routing loop when fetching remote media
...
When we proxy a media request to a remote server, add a query-param, which will
tell the remote server to 404 if it doesn't recognise the server_name.
This should fix a routing loop where the server keeps forwarding back to
itself.
Also improves the error handling on remote media fetches, so that we don't
always return a rather obscure 502.
2017-03-13 16:30:36 +00:00
aea5461488
Fix dynamic thumbnails aspect
2017-02-24 22:43:27 +01:00
32019c9897
Log which files we saved attachments to in the media_repository
2017-01-10 14:19:50 +00:00
f7085ac84f
Name linearizer's for better logs
2017-01-09 17:17:10 +00:00
24c16fc349
Fix crash in url preview when html tag has no text
...
Signed-off-by: Marcin Bachry <hegel666@gmail.com>
2016-12-14 22:38:18 +01:00
32c8b5507c
preview_url_resource: Ellipsis must be in unicode string
...
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-01 13:12:13 +01:00
b1c27975d0
Set CORs headers on responses from the media repo
2016-11-02 11:29:25 +00:00
d51b8a1674
Add quotes and be explicity about script-src
2016-09-05 17:35:01 +01:00
662b031a30
Allow PDF to be rendered from media repo
2016-09-05 17:25:26 +01:00
0af9e1a637
Set `Content-Security-Policy` on media repo
...
This is to inform browsers that they should sandbox the returned
media. This is particularly cruical for javascript/HTML files.
2016-08-17 16:27:39 +01:00
f90b3d83a3
Add None check to _iterate_over_text
2016-08-17 15:17:17 +01:00
109a560905
Flake8
2016-08-16 14:57:21 +01:00
48b5829aea
Fix up preview URL API. Add tests.
...
This includes:
- Splitting out methods of a class into stand alone functions, to make
them easier to test.
- Adding unit tests to split out functions, testing HTML -> preview.
- Handle the fact that elements in lxml may have tail text.
2016-08-16 14:53:24 +01:00
5bcccfde6c
Don't include html comments in description
2016-08-05 14:45:11 +01:00
b5525c76d1
Typo
2016-08-04 16:10:08 +01:00
e97648c4e2
Test summarization
2016-08-04 16:09:09 +01:00
58c9653c6b
Don't infer paragrahs from newlines
2016-08-02 18:50:24 +01:00
6b58ade2f0
Comment on why we clone
2016-08-02 18:41:22 +01:00
9e66c58ceb
Spelling.
2016-08-02 18:37:31 +01:00
f83f5fbce8
Make it actually compile
2016-08-02 18:32:42 +01:00
aecaec3e10
Change the way we summarize URLs
...
Using XPath is slow on some machines (for unknown reasons), so use a
different approach to get a list of text nodes.
Try to generate a summary that respect paragraph and then word
boundaries, adding ellipses when appropriate.
2016-08-02 18:25:53 +01:00
f52cb4cd78
Remove race
2016-06-29 15:24:50 +01:00
a70688445d
Implement purge_media_cache admin API
2016-06-29 14:57:59 +01:00
314b146b2e
Track approximate last access time for remote media
2016-06-29 11:41:20 +01:00
13e334506c
Remove the legacy v0 content upload API.
...
The existing content can still be downloaded. The last upload to the
matrix.org server was in January 2015, so it is probably safe to remove
the upload API.
2016-06-21 11:47:39 +01:00
Erik Johnston