Update docs in preparation for Matrix Public Archive being generally available (#194)

README.md

@@ -6,12 +6,8 @@ In the vein of [feature parity with
 Gitter](https://github.com/vector-im/roadmap/issues/26), the goal is to make a
 public archive site for `world_readable` Matrix rooms like Gitter's archives
 which search engines can index and keep all of the content accessible/available.
-There is already https://view.matrix.org/
-(https://github.com/matrix-org/matrix-static) but there is some desire to make
-something with more Element-feeling polish and loading faster (avoid the slow
-502's errors that are frequent on `view.matrix.org`).
 
-![](https://user-images.githubusercontent.com/558581/179578263-e224ed59-dbba-464e-8b34-89a72ee0ae71.png)
+![A reference for how the Matrix Public Archive looks. Showing off a day of messages in `#gitter:matrix.org` on 2021-08-06. There is a date picker calendar in the right sidebar and a traditional chat app layout on the left.](https://user-images.githubusercontent.com/558581/234765275-28c70c49-c27f-473a-88ba-f4392ddae871.png)
 
 ## Demo videos
 
@@ -31,17 +27,21 @@ messages from a Matrix homeserver.
 Re-using Hydrogen gets us pretty and native(to Element) looking styles and keeps
 the maintenance burden of supporting more event types in Hydrogen.
 
+## FAQ
+
+See the [FAQ page](docs/faq.md).
+
 ## Setup
 
 ### Prerequisites
 
-- Node.js v16
+- [Node.js](https://nodejs.org/) v16
   - We only need v16 because it includes
     [`require('crypto').webcrypto.subtle`](https://nodejs.org/docs/latest-v16.x/api/webcrypto.html#cryptosubtle)
     for [Matrix encryption (olm) which can't be disabled in
     Hydrogen](https://github.com/vector-im/hydrogen-web/issues/579) yet.
 - A Matrix homeserver that supports [MSC3030's](https://github.com/matrix-org/matrix-spec-proposals/pull/3030) `/timestamp_to_event` endpoint
-  - Synapse 1.73.0+
+  - [Synapse](https://matrix.org/docs/projects/server/synapse) 1.73.0+
 
 ### Get the app running
 
@@ -53,8 +53,6 @@ $ npm install
 $ cp config/config.default.json config/config.user-overrides.json
 
 $ npm run start
-# To enable tracing, add the `--tracing` flag
-$ npm run start -- --tracing
 ```
 
 ## Development
@@ -71,7 +69,8 @@ $ cp config/config.default.json config/config.user-overrides.json
 
 # This will watch for changes, rebuild bundles and restart the server
 $ npm run start-dev
-# To enable tracing, add the `--tracing` flag
+# To enable tracing, add the `--tracing` flag (you will also need setup a
+# Jaeger instance and set the `jaegerTracesEndpoint` config option)
 $ npm run start-dev -- --tracing
 ```
 
@@ -96,4 +95,8 @@ $ npm link hydrogen-view-sdk
 
 ### Running tests
 
-See the [testing readme](./test/README.md).
+See the [testing documentation](./docs/testing.md).
+
+### Tracing
+
+See the [tracing documentation](./docs/tracing.md).

docs/faq.md

@@ -0,0 +1,49 @@
+# FAQ
+
+## Can I run my own instance?
+
+Yes! We host a public canonical version of the Matrix Public Archive at
+[archive.matrix.org](https://archive.matrix.org/) that everyone can use but feel free to
+also run your own instance (setup instructions in the [readme](../README.md)).
+
+## How is this different from [`view.matrix.org`](https://view.matrix.org/)?
+
+https://view.matrix.org/ (https://github.com/matrix-org/matrix-static) already existed
+before the Matrix Public Archive but there was some desire to make something with more
+Element-feeling polish and loading faster (avoid the slow 502's errors that are frequent
+on `view.matrix.org`).
+
+## How do I opt out and keep my room from being indexed by search engines?
+
+All public Matrix rooms are accessible to view in the Matrix Public Archive. But only
+rooms with history visibility set to `world_readable` will be indexed by search engines.
+
+Also see https://github.com/matrix-org/matrix-public-archive/issues/47
+
+## Technical details
+
+The main readme has an [technical overview](../README.md#technical-overview) of the
+project. Here are a few more details.
+
+### How do I figure out what version of the Matrix Public Archive is running?
+
+Just visit the `/health-check` endpoint which will return information like the following:
+
+```
+{
+  "ok": true,
+  "commit": "954b22995a44bf11bfcd5850b62e206e46ee2db9",
+  "version": "main",
+  "versionDate": "2023-04-05T09:26:12.524Z",
+  "packageVersion": "0.0.0"
+}
+```
+
+### How does the archive room URL relate to what is displayed on the page?
+
+We start the end of the date/time specified in the URL looking backward up to the limit.
+
+### Why does the time selector only appear for some pages?
+
+The time selector only appears for pages that have a lot of messages on a given
+day/hour/minute/second (more than the configured `archiveMessageLimit`).

docs/testing.md

@@ -4,12 +4,15 @@
 
 If you haven't setup `matrix-public-archive` yet, see the [_Setup_ section in the root `README.md`](../README.md#setup)
 
+Then we need to setup the federation cluster of homeservers that we will test with.
 Sorry, this isn't automated yet when you run the tests 🙇
 
 ```sh
+# Build the test homeserver image that are pre-configured to federate with each other
 $ docker pull matrixdotorg/synapse:latest
 $ docker build -t matrix-public-archive-test-homeserver -f test/dockerfiles/Synapse.Dockerfile test/dockerfiles/
 
+# Start the test homeservers
 $ docker-compose --project-name matrix_public_archive_test -f test/docker-compose.yml up -d --no-recreate
 ```
 
@@ -19,14 +22,23 @@ $ docker-compose --project-name matrix_public_archive_test -f test/docker-compos
 $ npm run test
 ```
 
-Or if you want to keep `matrix-public-archive` server running after the tests run and explore the output from the interactive URL's printed on the screen, use:
+Or if you want to keep the Matrix Public Archive server running after the tests run and
+explore the UI from the interactive URL's printed on the screen to better debug, use:
 
 ```sh
 $ npm run test-interactive
 ```
 
+Caveat: You might not see the same result that a test is seeing when visiting the
+interactive URL. Some tests set config like the `archiveMessageLimit` which is reset
+after each test case unless you are using `npm run test-interactive` and visiting the
+interactive URL for a failed test. Otherwise, we reset config between each test case so
+they don't leak and contaminate each other.
+
 ### Developer utility
 
+Some copy-pasta to help you manage the Docker containers for the test homeservers:
+
 ```sh
 $ docker ps --all | grep test_hs
 $ docker logs -f --tail 10 matrix_public_archive_test_hs1_1

docs/tracing.md

@@ -0,0 +1,55 @@
+# Tracing
+
+Tracing allows you to see the flow of a request through the system and where time is
+taken up in functions. This is useful for debugging and performance analysis.
+
+<img src="https://user-images.githubusercontent.com/558581/180586026-ff6c653e-a54d-4cf4-abc8-a8c51971aad5.png" width="612">
+
+## Setup
+
+1. Get the all-in-one Jaeger Docker container running (via https://www.jaegertracing.io/docs/1.35/getting-started/)
+   ```
+   docker run -d --name jaeger \
+     -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
+     -e COLLECTOR_OTLP_ENABLED=true \
+     -p 6831:6831/udp \
+     -p 6832:6832/udp \
+     -p 5778:5778 \
+     -p 5775:5775/udp \
+     -p 16686:16686 \
+     -p 4317:4317 \
+     -p 4318:4318 \
+     -p 14250:14250 \
+     -p 14268:14268 \
+     -p 14269:14269 \
+     -p 9411:9411 \
+     jaegertracing/all-in-one:1.35
+   ```
+1. Add `jaegerTracesEndpoint` to your `config.json`:
+   ```json5
+   {
+     // ...
+     jaegerTracesEndpoint: 'http://localhost:14268/api/traces',
+   }
+   ```
+
+## Run the app with the OpenTelemetry tracing enabled
+
+```
+npm run start -- --tracing
+# or
+npm run start-dev -- --tracing
+```
+
+Manually:
+
+```
+node --require './server/tracing.js' server/server.js
+```
+
+## Viewing traces in Jaeger
+
+Once you have the all-in-one Jaeger Docker container running, just visit
+http://localhost:16686 to see a dashboard of the collected traces and dive in.
+
+Traces are made up of many spans. Each span defines a `traceId` which it is associated with.

server/README.md

@@ -1,58 +0,0 @@
-## Tracing
-
-Run the app with the OpenTelemetry tracing.
-
-```
-npm run start -- --tracing
-# or
-npm run start-dev -- --tracing
-```
-
-Manually:
-
-```
-node --require './server/tracing.js' server/server.js
-```
-
----
-
-Traces are made up of many spans. Each span defines a `traceId` which it is associated with.
-
-## Viewing traces in Jaeger
-
-- Dashboard where users can see visualisations -> http://localhost:16686
-
-via https://www.jaegertracing.io/docs/1.35/getting-started/
-
-```
-docker run -d --name jaeger \
-  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
-  -e COLLECTOR_OTLP_ENABLED=true \
-  -p 6831:6831/udp \
-  -p 6832:6832/udp \
-  -p 5778:5778 \
-  -p 5775:5775/udp \
-  -p 16686:16686 \
-  -p 4317:4317 \
-  -p 4318:4318 \
-  -p 14250:14250 \
-  -p 14268:14268 \
-  -p 14269:14269 \
-  -p 9411:9411 \
-  jaegertracing/all-in-one:1.35
-```
-
-| Port  | Protocol | Component | Function                                                                                     |
-| ----- | -------- | --------- | -------------------------------------------------------------------------------------------- |
-| 6831  | UDP      | agent     | accept jaeger.thrift over Thrift-compact protocol (used by most SDKs)                        |
-| 6832  | UDP      | agent     | accept jaeger.thrift over Thrift-binary protocol (used by Node.js SDK)                       |
-| 5775  | UDP      | agent     | (deprecated) accept zipkin.thrift over compact Thrift protocol (used by legacy clients only) |
-| 5778  | HTTP     | agent     | serve configs (sampling, etc.)                                                               |
-| 16686 | HTTP     | query     | serve frontend                                                                               |
-| 4317  | HTTP     | collector | accept OpenTelemetry Protocol (OTLP) over gRPC, if enabled                                   |
-| 4318  | HTTP     | collector | accept OpenTelemetry Protocol (OTLP) over HTTP, if enabled                                   |
-| 14268 | HTTP     | collector | accept jaeger.thrift directly from clients                                                   |
-| 14250 | HTTP     | collector | accept model.proto                                                                           |
-| 9411  | HTTP     | collector | Zipkin compatible endpoint (optional)                                                        |
-
-With Service Performance Monitoring (SPM)