Commit Graph

216 Commits

Author SHA1 Message Date
Erik Johnston c1dbe84c3d
Add release helper script (#9713)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-04-13 11:51:10 +01:00
Dan Callahan 3efde8b69a
Add option to skip unit tests when building debs (#9793)
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-12 15:27:05 +01:00
Patrick Cloke abc814dcbf
Enable complement tests for MSC2946. (#9771)
By providing the additional build tag for `msc2946`.
2021-04-09 08:11:51 -04:00
Richard van der Hoff 9e167d9c53 Merge remote-tracking branch 'origin/develop' into rav/drop_py35 2021-04-08 18:30:38 +01:00
Richard van der Hoff 9278eb701e drop support for stretch and xenial 2021-04-08 16:42:32 +01:00
Patrick Cloke 452991527a
MSC3083: Check for space membership during a local join of restricted rooms. (#9735)
When joining a room with join rules set to 'restricted', check if the
user is a member of the spaces defined in the 'allow' key of the join rules.

This only applies to an experimental room version, as defined in MSC3083.
2021-04-08 08:28:32 -04:00
Patrick Cloke e32294f54b Merge branch 'release-v1.31.0' into develop 2021-03-31 14:19:14 -04:00
Patrick Cloke 5fe38e07e7
Revert "Use 'dmypy run' in lint.sh instead of 'mypy' (#9701)" (#9720) 2021-03-31 14:17:52 -04:00
Andrew Morgan ac99774dac
Rewrite complement.sh (#9685)
This PR rewrites the original complement.sh script with a number of improvements:

* We can now use a local checkout of Complement (configurable with `COMPLEMENT_DIR`), though the default behaviour still downloads the master branch.
* You can now specify a regex of test names to run, or just run all tests.
* We now use the Synapse test blacklist tag (so all tests will pass).
2021-03-31 11:58:12 +01:00
Andrew Morgan f380bb77d1
Use 'dmypy run' in lint.sh instead of 'mypy' (#9701)
For it's obvious performance benefits. `dmypy` support landed in #9692.
2021-03-30 10:30:43 +01:00
Quentin Gliech d4c4798a25
Use interpreter from $PATH instead of absolute paths in various scripts using /usr/bin/env (#9689)
On NixOS, `bash` isn't under `/bin/bash` but rather in some directory in `$PATH`. Locally, I've been patching those scripts to make them work.

`/usr/bin/env` seems to be the only [portable way](https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my) to use binaries from the PATH as interpreters.

Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
2021-03-25 16:53:54 +00:00
Richard van der Hoff 0e35584734
federation_client: handle inline signing_keys in hs.yaml (#9647) 2021-03-18 21:12:07 +00:00
Richard van der Hoff 201178db1a
federation_client: stop adding URL prefix (#9645) 2021-03-18 20:31:47 +00:00
Andrew Morgan 22db45bd4d
Prevent the config-lint script erroring out on any sample_config changes (#9562)
I noticed that I'd occasionally have `scripts-dev/lint.sh` fail when messing about with config options in my PR. The script calls `scripts-dev/config-lint.sh`, which attempts some validation on the sample config.

 It does this by using `sed` to edit the sample_config, and then seeing if the file changed using `git diff`.

The problem is: if you changed the sample_config as part of your commit, this script will error regardless.

This PR attempts to change the check so that existing, unstaged changes to the sample_config will not cause the script to report an invalid file.
2021-03-09 11:11:42 +00:00
Eric Eastwood 0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Eric Eastwood 80d6dc9783
Remove conflicting sqlite tables that are "reserved" (shadow fts4 tables) (#9003)
Remove conflicting sqlite tables that throw sqlite3.OperationalError: object name reserved for internal use: event_search_content when running the twisted unit tests.

Fix #8996
2021-02-10 20:12:57 +00:00
Andrew Morgan 31c5382d7a
Align the directories linted in CI with the defaults in scripts-dev/lint.sh (#9191)
The lists of source directories to lint between `tox.ini` and `lint.sh` became out of sync. This PR tightens them up and adds some comments reminding any future readers to keep the list in sync.
2021-01-21 18:26:52 +00:00
Patrick Cloke 5d34f40d49
Add type hints to the push module. (#8901) 2020-12-11 11:43:53 -05:00
Richard van der Hoff 97b35ee259
Add a script to sign arbitrary json objects. (#8772) 2020-11-24 12:53:00 +00:00
Erik Johnston 22eeb6bc54
Fix cache call signature to accept `on_invalidate`. (#8684)
Cached functions accept an `on_invalidate` function, which we failed to add to the type signature. It's rarely used in the files that we have typed, which is why we haven't noticed it before.
2020-10-29 15:18:17 +00:00
Patrick Cloke 00b24aa545
Support generating structured logs in addition to standard logs. (#8607)
This modifies the configuration of structured logging to be usable from
the standard Python logging configuration.

This also separates the formatting of logs from the transport allowing
JSON logs to files or standard logs to sockets.
2020-10-29 07:27:37 -04:00
kleph 29ce6d43b5
Run mypy as part of the lint.sh script. (#8633) 2020-10-28 08:49:08 -04:00
Andrew Morgan 654e239b25
Add option to scripts-dev/lint.sh to only lint files changed since the last git commit (#8472)
This PR makes several changes to the `./scripts-dev/lint.sh` script, which lints the codebase with a number of tools:

* Adds usage information, with `-h` flag to show it. Otherwise it will show when providing an unknown flag.
* Adds option `-d` which will check both staged and unstaged files that have changed since the last commit and add them to the list of files to lint.
  - Note that only files without an extension, or with a `.py` extension will be allowed. This prevents editing bash scripts causing the linters to break on non-python files.
* Improves the print-out of which files/directories are being linted.
2020-10-15 15:45:13 +01:00
Erik Johnston 1baa895310 Merge remote-tracking branch 'origin/release-v1.21.0' into develop 2020-10-07 17:20:24 +01:00
Patrick Cloke d9b55bd830
Add Ubuntu 20.10 (Groovy Gorilla) to build scripts. (#8475) 2020-10-07 08:48:54 -04:00
Andrew Morgan 01f82bfe32
Remove docs/sphinx and related references (#8480)
https://github.com/matrix-org/synapse/tree/develop/docs/sphinx doesn't seem to really be utilised or changed recently since the initial commit. I like the idea of exportable documentation of the codebase, but at the moment after running through the build instructions the generated website wasn't very useful...
2020-10-07 11:45:31 +01:00
Andrew Morgan 12f0d18611
Add support for running Complement against the local checkout (#8317)
This PR adds a script that:

* Builds the local Synapse checkout using our existing `docker/Dockerfile` image.
* Downloads [Complement](https://github.com/matrix-org/complement/)'s source code.
* Builds the [Synapse.Dockerfile](https://github.com/matrix-org/complement/blob/master/dockerfiles/Synapse.Dockerfile) using the above dockerfile as a base.
* Builds and runs Complement against it.

This set up differs slightly from [that of the dendrite repo](https://github.com/matrix-org/dendrite/blob/master/build/scripts/complement.sh) (`complement.sh`, `Complement.Dockerfile`), which instead stores a separate, but slightly modified, dockerfile in Dendrite's repo rather than running the one stored in Complement's repo. That synapse equivalent to that dockerfile (`Synapse.Dockerfile`) in Complement's repo is just based on top of `matrixdotorg/synapse:latest`, which we opt to build here locally.

Thus copying over the files from Complement's repo wouldn't change any functionality, and would result in two instances of the same files. So just using the dockerfile in Complement's repo was decided upon instead.
2020-09-29 13:47:47 +01:00
Patrick Cloke 8a4a4186de
Simplify super() calls to Python 3 syntax. (#8344)
This converts calls like super(Foo, self) -> super().

Generated with:

    sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
2020-09-18 09:56:44 -04:00
Jonathan de Jong 837293c314
Remove obsolete __future__ imports (#8337) 2020-09-17 08:37:01 -04:00
Patrick Cloke c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Erik Johnston 208e1d3eb3
Fix typing for `@cached` wrapped functions (#8240)
This requires adding a mypy plugin to fiddle with the type signatures a bit.
2020-09-03 15:38:32 +01:00
Richard van der Hoff 88b9807ba4
Update the test federation client to handle streaming responses (#8130)
Now that the server supports streaming back JSON responses, it would be nice to
show the response as it is streamed, in the test tool.
2020-08-26 14:11:38 +01:00
Erik Johnston a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Andrew Morgan 7cac9006d6
Spruce up the check-newsfragment CI output (#8024)
This PR:

* Reduces the amount of noise in the `check-newsfragment` CI output by hiding the dependency installation output by default.
* Prints a link to the changelog/debian changelog section of the contributing guide if an error is found.
2020-08-04 22:10:23 +01:00
Richard van der Hoff 8078dec3be
Fix exit code for `check_line_terminators.sh` (#7970)
If there are *no* files with CRLF line endings, then the xargs exits with a
non-zero exit code (as expected), but then, since that is the last thing to
happen in the script, the script as a whole exits non-zero, making the whole
thing fail.

using `if/then/fi` instead of `&& (...)` means that the script exits with a
zero exit code.
2020-07-28 08:52:25 -04:00
Andrew Morgan c4ce0da6fe
Add script for finding files with unix line terminators (#7965)
This PRs adds a script to check for unix-line terminators in the repo. It will be used to address https://github.com/matrix-org/synapse/issues/7943 by adding the check to CI.

I've changed the original script slightly as proposed in https://github.com/matrix-org/pipelines/pull/81#discussion_r460580664
2020-07-28 01:26:50 +01:00
Andrew Morgan b7ddece2a6
Lint the contrib/ directory in CI and linting scripts, add synctl to linting script (#7914)
Run `isort`, `flake8` and `black` over the `contrib/` directory and `synctl` script. The latter was already being done in CI, but now the linting script does it too.

Fixes https://github.com/matrix-org/synapse/issues/7910
2020-07-20 21:43:49 +01:00
Gary Kim f2af3e4fc5
Remove Ubuntu Eoan that is now EOL (#7888) 2020-07-17 15:38:41 -04:00
Will Hunt 62b1ce8539
isort 5 compatibility (#7786)
The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
2020-07-05 16:32:02 +01:00
Andrew Morgan 177b2d0c19
Move flake8 to end. Don't exit script on failure (#7738) 2020-06-25 17:58:55 +01:00
Dagfinn Ilmari Mannsåker a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Richard van der Hoff 86d814cdde
Check the changelog number in check-newsfragment (#7623) 2020-06-03 17:01:43 +01:00
Patrick Cloke b2b8699070
Remove Ubuntu Cosmic and Disco which are both EOL. (#7539) 2020-05-20 10:08:46 -04:00
Patrick Cloke 02919bf4d8 Merge branch 'master' into develop 2020-05-19 09:56:15 -04:00
Richard van der Hoff 1fc8914f76
update dh-virtualenv (#7526) 2020-05-19 13:48:41 +01:00
Richard van der Hoff e6027562e2 remove `builtins.buffer` code from storage code
this is no longer needed on python 3
2020-05-15 19:37:41 +01:00
Patrick Cloke 509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Patrick Cloke ba547ec3a9
Use BSD-compatible in-place editing for sed. (#6887) 2020-02-12 07:02:19 -05:00
Brendan Abolivier 07124d028d
Port synapse_port_db to async/await (#6718)
* Raise an exception if there are pending background updates

So we return with a non-0 code

* Changelog

* Port synapse_port_db to async/await

* Port update_database to async/await

* Add version string to mocked homeservers

* Remove unused imports

* Convert overseen bits to async/await

* Fixup logging contexts

* Fix imports

* Add a way to print an error without raising an exception

* Incorporate review
2020-01-21 19:04:58 +00:00
Richard van der Hoff 08815566bc
Automate generation of the sample and debian log configs (#6627) 2020-01-03 17:14:00 +00:00
Erik Johnston 2284eb3a53
Add database config class (#6513)
This encapsulates config for a given database and is the way to get new
connections.
2019-12-18 10:45:12 +00:00
Erik Johnston b8e4b39b69
Merge pull request #6511 from matrix-org/erikj/remove_db_config_from_apps
Move database config from apps into HomeServer object
2019-12-12 10:37:56 +00:00
Andrew Morgan 6676ee9c4a
Add dev script to generate full SQL schema files (#6394) 2019-12-11 13:16:01 +00:00
Erik Johnston 67c991b78f Fix upgrade db script 2019-12-10 14:34:17 +00:00
Erik Johnston 8b77fc6506 Fix DB scripts 2019-12-05 11:20:49 +00:00
Erik Johnston ee86abb2d6 Remove underscore from SQLBaseStore functions 2019-12-04 16:23:43 +00:00
Richard van der Hoff 915903eada Merge branch 'master' into develop 2019-11-06 13:51:11 +00:00
Richard van der Hoff 4257feb20f build debs for eoan and bullseye 2019-11-06 13:35:56 +00:00
Neil Pilgrim befd58f47b Document lint.sh & allow application to specified files only (#6312) 2019-11-01 10:52:20 +00:00
Andrew Morgan 2cab02f9d1
Update CI to run isort on scripts and scripts-dev (#6270) 2019-10-30 11:17:14 +00:00
Brendan Abolivier 14504ad573
Add CI for synapse_port_db (#6140)
This adds:

* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)

as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:

1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)

Step `2` is done via a new script located at `scripts-dev/update_database`.

The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
2019-10-28 17:45:32 +00:00
Andrew Morgan 409c62b27b
Add config linting script that checks for bool casing (#6203)
Add a linting script that enforces all boolean values in the default config be lowercase.

This has annoyed me for a while so I decided to fix it.
2019-10-23 13:22:54 +01:00
Richard van der Hoff cad0132fb5 Remove dead check_auth script
This doesn't work, and afaict hasn't been used since 2015.
2019-10-11 11:24:03 +01:00
Andrew Morgan 3dd61d12cd
Add a linting script (#5627)
Add a dev script to cover all the different linting steps.
2019-07-10 14:03:18 +01:00
Amber Brown 0e97284dfa
Remove & changelog (#5548) 2019-06-26 04:36:34 +10:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Richard van der Hoff b59a4eba64
Updates to the federation_client script (#5447)
* py3 fixes for federation_client
* .well-known support for federation_client
2019-06-13 14:49:25 +01:00
Amber Brown 2889b05554
Unify v1 and v2 REST client APIs (#5226) 2019-06-03 21:28:59 +10:00
Richard van der Hoff 0b5cf95607 include disco in deb build target list 2019-05-03 18:55:13 +01:00
Erik Johnston 5fee9d8067
Merge pull request #4869 from matrix-org/erikj/yaml_load
Fix yaml warnings by using safe_load
2019-03-22 11:58:13 +00:00
Erik Johnston 3677548a82 Use yaml safe_load 2019-03-22 10:20:17 +00:00
Richard van der Hoff 224783a73f
Allow newsfragments to end with exclamation marks! (#4912) 2019-03-21 15:28:19 +00:00
Richard van der Hoff 685704536f
Fix check-newsfragment for debian-only changes. (#4825) 2019-03-07 14:03:05 +00:00
Richard van der Hoff 8e28bc5eee
Include a default configuration file in the 'docs' directory. (#4791) 2019-03-04 17:14:58 +00:00
Richard van der Hoff 57426ec6a3 Fix check-newsfragment script (#4750)
* Fix check-newsfragment script

I previously broke this so that it always succeeded...

* more fixes

* fix newsfiles
2019-02-26 13:13:41 -08:00
Richard van der Hoff e1666af9be
Better checks on newsfragments (#4698)
* You need an entry in the debian changelog (and not a regular newsfragment)
  for debian packaging changes.
* Regular newsfragments must end in full stops.
2019-02-22 10:56:59 +00:00
Amber Brown 6129e52f43
Support ACME for certificate provisioning (#4384) 2019-01-23 19:39:06 +11:00
Richard van der Hoff b5b868d41e Rewrite build_debian_packages
Rewrite this in python so that it can be run in parallel.
2019-01-12 12:40:58 +00:00
Brendan Abolivier 0f3f0a64bf
Merge pull request #4168 from matrix-org/babolivier/federation-client-content-type
Add a Content-Type header on POST requests to the federation client script
2018-11-09 11:00:55 +00:00
Brendan Abolivier 91d96759c9
Add a Content-Type header on POST requests to the federation client 2018-11-09 10:41:34 +00:00
Amber Brown 4cd1c9f2ff
Delete the disused & unspecced identicon functionality (#4106) 2018-10-29 23:57:24 +11:00
Richard van der Hoff ef771cc4c2 Fix a number of flake8 errors
Broadly three things here:

* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
  them
* use `r""` for strings which include backslashes

Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
2018-10-24 10:39:03 +01:00
Amber Brown e1728dfcbe
Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068) 2018-10-20 11:16:55 +11:00
Amber Brown 058a4c665e
Remove Jenkins & other old dev junk (#3988) 2018-10-03 00:59:11 +10:00
Schnuffle dc5db01ff2 Replaced all occurences of e.message with str(e)
Signed-off-by: Schnuffle  <schnuffle@github.com>
2018-09-27 13:38:50 +02:00
Amber Brown d4e0861ff9
Reduce the load on our CI (#3957)
* changelog

* reduce circleci config

* plus a handy script

* fix regex
2018-09-27 00:23:21 +10:00
Richard van der Hoff e7b3b4d8c2 Remove nuke-room-from-db.sh script
The problem with this script is that it is largely untested, entirely
unmaintained, and running it is likely to make your synapse blow up in
exciting ways.

For example, it leaves a bunch of tables with dead values in it, like
event_to_state_groups.

Having it here sends a message that it is a supported part of
synapse, which is absolutely not the case.
2018-09-17 11:42:42 +01:00
Richard van der Hoff 96bad44f87 Fix federation_client to send the right Host
This appears to have stopped working since matrix.org moved to cloudflare. The
Host header should match the name of the server, not whatever is in the SRV
record.
2018-06-12 14:14:36 +01:00
Richard van der Hoff 60590211c1
Merge pull request #3194 from rubo77/fix-nuke
nuke-room-from-db.sh: fix deletion from search table
2018-05-09 11:58:07 +01:00
rubo77 d11b8b6b65 nuke-room-from-db.sh: nuke from table event_search too 2018-05-09 00:46:47 +02:00
rubo77 8ff8ab3bce Dont nuke non-existing table event_search_content 2018-05-09 00:21:00 +02:00
rubo77 6c957e26f0 nuke-room-from-db.sh: added postgresql option and help 2018-05-09 00:14:01 +02:00
Richard van der Hoff fd04968f32 federation_client script: Support for posting content 2017-11-28 11:59:24 +00:00
Richard van der Hoff a04c6bbf8f test federation client: Allow server-name and key-file as options
so that you don't necessarily need a config file.
2017-08-22 11:19:30 +01:00
Richard van der Hoff 046b659ce2 Improvements to the federation test client
Make it read the config file, primarily.
2017-08-17 16:59:11 +01:00
Christian W. Zuckschwerdt 20746d8150 bring nuke-room script to current schema
Signed-off-by: Christian W. Zuckschwerdt <christian@zuckschwerdt.org>
2017-02-19 05:27:45 +01:00
Erik Johnston f45be05305 Print newline after result in federation_client script 2016-08-05 16:46:14 +01:00
Erik Johnston 97f072db74 Print status code in federation_client.py 2016-08-03 13:46:56 +01:00
Erik Johnston fcde5b2a97 Print authorization header for federation_client.py 2016-08-02 15:06:17 +01:00
Mark Haines 4ebb688f4f Add option to definitions.py to search for functions a function refers to 2016-03-17 10:59:12 +00:00
Mark Haines 60a0f81c7a Add a /replication API for extracting the updates that happened on
synapse

This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.

This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.

Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.

Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.

The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Mark Haines 77c7ed0e93 Report the v1 and v2 patterns separately 2016-02-05 15:43:27 +00:00
Mark Haines b052621f67 List the URL patterns in synapse 2016-02-05 15:18:58 +00:00
Daniel Wagner-Hall 5f280837a6 Add macaroon inspection script 2016-02-03 11:27:39 +00:00
Matthew Hodgson 6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Mark Haines 5d321e4b9a Fix definitions script 2015-12-01 21:17:58 +00:00
Mark Haines f593a6e5f8 Add options to definitions.py to fetch referrers and to output dot 2015-12-01 14:29:42 +00:00
Mark Haines 314aabba82 Fix scripts-dev/definitions.py argparse options 2015-09-23 10:45:33 +01:00
Mark Haines 04abf53a56 Use argparse for definition finder 2015-09-23 10:17:50 +01:00
Mark Haines cc3ab0c214 Add dev script for finding where functions are called from, and finding functions that aren't called at all 2015-09-22 18:13:06 +01:00
Erik Johnston c0d1f37baf Don't require pdus in check_auth script 2015-09-09 13:47:14 +01:00
Mark Haines f093873d69 Replace syutil references in scripts 2015-08-24 16:30:35 +01:00
Erik Johnston df431b127b Add forgotten .items() 2015-05-19 13:14:21 +01:00
Erik Johnston 882ac83d8d Fix scripts-dev/convert_server_keys.py to have correct format 2015-05-19 13:12:55 +01:00
Mark Haines 3c224f4d0e SYN-376: Add script for converting server keys from v1 to v2 2015-05-11 11:00:17 +01:00
Erik Johnston 673375fe2d Acutally add scripts-dev/ 2015-05-06 11:46:02 +01:00