diff --git a/README.md b/README.md index e817db0..b14f396 100644 --- a/README.md +++ b/README.md @@ -6,12 +6,8 @@ In the vein of [feature parity with Gitter](https://github.com/vector-im/roadmap/issues/26), the goal is to make a public archive site for `world_readable` Matrix rooms like Gitter's archives which search engines can index and keep all of the content accessible/available. -There is already https://view.matrix.org/ -(https://github.com/matrix-org/matrix-static) but there is some desire to make -something with more Element-feeling polish and loading faster (avoid the slow -502's errors that are frequent on `view.matrix.org`). -![](https://user-images.githubusercontent.com/558581/179578263-e224ed59-dbba-464e-8b34-89a72ee0ae71.png) +![A reference for how the Matrix Public Archive looks. Showing off a day of messages in `#gitter:matrix.org` on 2021-08-06. There is a date picker calendar in the right sidebar and a traditional chat app layout on the left.](https://user-images.githubusercontent.com/558581/234765275-28c70c49-c27f-473a-88ba-f4392ddae871.png) ## Demo videos @@ -31,17 +27,21 @@ messages from a Matrix homeserver. Re-using Hydrogen gets us pretty and native(to Element) looking styles and keeps the maintenance burden of supporting more event types in Hydrogen. +## FAQ + +See the [FAQ page](docs/faq.md). + ## Setup ### Prerequisites -- Node.js v16 +- [Node.js](https://nodejs.org/) v16 - We only need v16 because it includes [`require('crypto').webcrypto.subtle`](https://nodejs.org/docs/latest-v16.x/api/webcrypto.html#cryptosubtle) for [Matrix encryption (olm) which can't be disabled in Hydrogen](https://github.com/vector-im/hydrogen-web/issues/579) yet. - A Matrix homeserver that supports [MSC3030's](https://github.com/matrix-org/matrix-spec-proposals/pull/3030) `/timestamp_to_event` endpoint - - Synapse 1.73.0+ + - [Synapse](https://matrix.org/docs/projects/server/synapse) 1.73.0+ ### Get the app running @@ -53,8 +53,6 @@ $ npm install $ cp config/config.default.json config/config.user-overrides.json $ npm run start -# To enable tracing, add the `--tracing` flag -$ npm run start -- --tracing ``` ## Development @@ -71,7 +69,8 @@ $ cp config/config.default.json config/config.user-overrides.json # This will watch for changes, rebuild bundles and restart the server $ npm run start-dev -# To enable tracing, add the `--tracing` flag +# To enable tracing, add the `--tracing` flag (you will also need setup a +# Jaeger instance and set the `jaegerTracesEndpoint` config option) $ npm run start-dev -- --tracing ``` @@ -96,4 +95,8 @@ $ npm link hydrogen-view-sdk ### Running tests -See the [testing readme](./test/README.md). +See the [testing documentation](./docs/testing.md). + +### Tracing + +See the [tracing documentation](./docs/tracing.md). diff --git a/docs/faq.md b/docs/faq.md new file mode 100644 index 0000000..2b93579 --- /dev/null +++ b/docs/faq.md @@ -0,0 +1,49 @@ +# FAQ + +## Can I run my own instance? + +Yes! We host a public canonical version of the Matrix Public Archive at +[archive.matrix.org](https://archive.matrix.org/) that everyone can use but feel free to +also run your own instance (setup instructions in the [readme](../README.md)). + +## How is this different from [`view.matrix.org`](https://view.matrix.org/)? + +https://view.matrix.org/ (https://github.com/matrix-org/matrix-static) already existed +before the Matrix Public Archive but there was some desire to make something with more +Element-feeling polish and loading faster (avoid the slow 502's errors that are frequent +on `view.matrix.org`). + +## How do I opt out and keep my room from being indexed by search engines? + +All public Matrix rooms are accessible to view in the Matrix Public Archive. But only +rooms with history visibility set to `world_readable` will be indexed by search engines. + +Also see https://github.com/matrix-org/matrix-public-archive/issues/47 + +## Technical details + +The main readme has an [technical overview](../README.md#technical-overview) of the +project. Here are a few more details. + +### How do I figure out what version of the Matrix Public Archive is running? + +Just visit the `/health-check` endpoint which will return information like the following: + +``` +{ + "ok": true, + "commit": "954b22995a44bf11bfcd5850b62e206e46ee2db9", + "version": "main", + "versionDate": "2023-04-05T09:26:12.524Z", + "packageVersion": "0.0.0" +} +``` + +### How does the archive room URL relate to what is displayed on the page? + +We start the end of the date/time specified in the URL looking backward up to the limit. + +### Why does the time selector only appear for some pages? + +The time selector only appears for pages that have a lot of messages on a given +day/hour/minute/second (more than the configured `archiveMessageLimit`). diff --git a/test/README.md b/docs/testing.md similarity index 51% rename from test/README.md rename to docs/testing.md index 0040cec..320abdf 100644 --- a/test/README.md +++ b/docs/testing.md @@ -4,12 +4,15 @@ If you haven't setup `matrix-public-archive` yet, see the [_Setup_ section in the root `README.md`](../README.md#setup) +Then we need to setup the federation cluster of homeservers that we will test with. Sorry, this isn't automated yet when you run the tests 🙇 ```sh +# Build the test homeserver image that are pre-configured to federate with each other $ docker pull matrixdotorg/synapse:latest $ docker build -t matrix-public-archive-test-homeserver -f test/dockerfiles/Synapse.Dockerfile test/dockerfiles/ +# Start the test homeservers $ docker-compose --project-name matrix_public_archive_test -f test/docker-compose.yml up -d --no-recreate ``` @@ -19,14 +22,23 @@ $ docker-compose --project-name matrix_public_archive_test -f test/docker-compos $ npm run test ``` -Or if you want to keep `matrix-public-archive` server running after the tests run and explore the output from the interactive URL's printed on the screen, use: +Or if you want to keep the Matrix Public Archive server running after the tests run and +explore the UI from the interactive URL's printed on the screen to better debug, use: ```sh $ npm run test-interactive ``` +Caveat: You might not see the same result that a test is seeing when visiting the +interactive URL. Some tests set config like the `archiveMessageLimit` which is reset +after each test case unless you are using `npm run test-interactive` and visiting the +interactive URL for a failed test. Otherwise, we reset config between each test case so +they don't leak and contaminate each other. + ### Developer utility +Some copy-pasta to help you manage the Docker containers for the test homeservers: + ```sh $ docker ps --all | grep test_hs $ docker logs -f --tail 10 matrix_public_archive_test_hs1_1 diff --git a/docs/tracing.md b/docs/tracing.md new file mode 100644 index 0000000..00d4065 --- /dev/null +++ b/docs/tracing.md @@ -0,0 +1,55 @@ +# Tracing + +Tracing allows you to see the flow of a request through the system and where time is +taken up in functions. This is useful for debugging and performance analysis. + + + +## Setup + +1. Get the all-in-one Jaeger Docker container running (via https://www.jaegertracing.io/docs/1.35/getting-started/) + ``` + docker run -d --name jaeger \ + -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ + -e COLLECTOR_OTLP_ENABLED=true \ + -p 6831:6831/udp \ + -p 6832:6832/udp \ + -p 5778:5778 \ + -p 5775:5775/udp \ + -p 16686:16686 \ + -p 4317:4317 \ + -p 4318:4318 \ + -p 14250:14250 \ + -p 14268:14268 \ + -p 14269:14269 \ + -p 9411:9411 \ + jaegertracing/all-in-one:1.35 + ``` +1. Add `jaegerTracesEndpoint` to your `config.json`: + ```json5 + { + // ... + jaegerTracesEndpoint: 'http://localhost:14268/api/traces', + } + ``` + +## Run the app with the OpenTelemetry tracing enabled + +``` +npm run start -- --tracing +# or +npm run start-dev -- --tracing +``` + +Manually: + +``` +node --require './server/tracing.js' server/server.js +``` + +## Viewing traces in Jaeger + +Once you have the all-in-one Jaeger Docker container running, just visit +http://localhost:16686 to see a dashboard of the collected traces and dive in. + +Traces are made up of many spans. Each span defines a `traceId` which it is associated with. diff --git a/server/README.md b/server/README.md deleted file mode 100644 index 68bd020..0000000 --- a/server/README.md +++ /dev/null @@ -1,58 +0,0 @@ -## Tracing - -Run the app with the OpenTelemetry tracing. - -``` -npm run start -- --tracing -# or -npm run start-dev -- --tracing -``` - -Manually: - -``` -node --require './server/tracing.js' server/server.js -``` - ---- - -Traces are made up of many spans. Each span defines a `traceId` which it is associated with. - -## Viewing traces in Jaeger - -- Dashboard where users can see visualisations -> http://localhost:16686 - -via https://www.jaegertracing.io/docs/1.35/getting-started/ - -``` -docker run -d --name jaeger \ - -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ - -e COLLECTOR_OTLP_ENABLED=true \ - -p 6831:6831/udp \ - -p 6832:6832/udp \ - -p 5778:5778 \ - -p 5775:5775/udp \ - -p 16686:16686 \ - -p 4317:4317 \ - -p 4318:4318 \ - -p 14250:14250 \ - -p 14268:14268 \ - -p 14269:14269 \ - -p 9411:9411 \ - jaegertracing/all-in-one:1.35 -``` - -| Port | Protocol | Component | Function | -| ----- | -------- | --------- | -------------------------------------------------------------------------------------------- | -| 6831 | UDP | agent | accept jaeger.thrift over Thrift-compact protocol (used by most SDKs) | -| 6832 | UDP | agent | accept jaeger.thrift over Thrift-binary protocol (used by Node.js SDK) | -| 5775 | UDP | agent | (deprecated) accept zipkin.thrift over compact Thrift protocol (used by legacy clients only) | -| 5778 | HTTP | agent | serve configs (sampling, etc.) | -| 16686 | HTTP | query | serve frontend | -| 4317 | HTTP | collector | accept OpenTelemetry Protocol (OTLP) over gRPC, if enabled | -| 4318 | HTTP | collector | accept OpenTelemetry Protocol (OTLP) over HTTP, if enabled | -| 14268 | HTTP | collector | accept jaeger.thrift directly from clients | -| 14250 | HTTP | collector | accept model.proto | -| 9411 | HTTP | collector | Zipkin compatible endpoint (optional) | - -With Service Performance Monitoring (SPM)