cyberes
/
matrix-public-archive-stealth
mirror of https://github.com/Cyberes/matrix-public-archive-stealth.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update docs in preparation for Matrix Public Archive being generally available (#194)

This commit is contained in:
Eric Eastwood 2023-04-27 00:22:41 -05:00 committed by GitHub
parent ab9154bec2
commit 53a1d4b43b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 131 additions and 70 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
README.md
View File

@ -6,12 +6,8 @@ In the vein of [feature parity with
Gitter](https://github.com/vector-im/roadmap/issues/26), the goal is to make a
public archive site for `world_readable` Matrix rooms like Gitter's archives
which search engines can index and keep all of the content accessible/available.
There is already https://view.matrix.org/
(https://github.com/matrix-org/matrix-static) but there is some desire to make
something with more Element-feeling polish and loading faster (avoid the slow
502's errors that are frequent on `view.matrix.org`).
![](https://user-images.githubusercontent.com/558581/179578263-e224ed59-dbba-464e-8b34-89a72ee0ae71.png)
![A reference for how the Matrix Public Archive looks. Showing off a day of messages in `#gitter:matrix.org` on 2021-08-06. There is a date picker calendar in the right sidebar and a traditional chat app layout on the left.](https://user-images.githubusercontent.com/558581/234765275-28c70c49-c27f-473a-88ba-f4392ddae871.png)
## Demo videos
@ -31,17 +27,21 @@ messages from a Matrix homeserver.
Re-using Hydrogen gets us pretty and native(to Element) looking styles and keeps
the maintenance burden of supporting more event types in Hydrogen.
## FAQ
See the [FAQ page](docs/faq.md).
## Setup
### Prerequisites
- Node.js v16
- [Node.js](https://nodejs.org/) v16
- We only need v16 because it includes
[`require('crypto').webcrypto.subtle`](https://nodejs.org/docs/latest-v16.x/api/webcrypto.html#cryptosubtle)
for [Matrix encryption (olm) which can't be disabled in
Hydrogen](https://github.com/vector-im/hydrogen-web/issues/579) yet.
- A Matrix homeserver that supports [MSC3030's](https://github.com/matrix-org/matrix-spec-proposals/pull/3030) `/timestamp_to_event` endpoint
- Synapse 1.73.0+
- [Synapse](https://matrix.org/docs/projects/server/synapse) 1.73.0+
### Get the app running
@ -53,8 +53,6 @@ $ npm install
$ cp config/config.default.json config/config.user-overrides.json
$ npm run start
# To enable tracing, add the `--tracing` flag
$ npm run start -- --tracing
```
## Development
@ -71,7 +69,8 @@ $ cp config/config.default.json config/config.user-overrides.json
# This will watch for changes, rebuild bundles and restart the server
$ npm run start-dev
# To enable tracing, add the `--tracing` flag
# To enable tracing, add the `--tracing` flag (you will also need setup a
# Jaeger instance and set the `jaegerTracesEndpoint` config option)
$ npm run start-dev -- --tracing
```
@ -96,4 +95,8 @@ $ npm link hydrogen-view-sdk
### Running tests
See the [testing readme](./test/README.md).
See the [testing documentation](./docs/testing.md).
### Tracing
See the [tracing documentation](./docs/tracing.md).

49
docs/faq.md Normal file
View File

@ -0,0 +1,49 @@
# FAQ
## Can I run my own instance?
Yes! We host a public canonical version of the Matrix Public Archive at
[archive.matrix.org](https://archive.matrix.org/) that everyone can use but feel free to
also run your own instance (setup instructions in the [readme](../README.md)).
## How is this different from [`view.matrix.org`](https://view.matrix.org/)?
https://view.matrix.org/ (https://github.com/matrix-org/matrix-static) already existed
before the Matrix Public Archive but there was some desire to make something with more
Element-feeling polish and loading faster (avoid the slow 502's errors that are frequent
on `view.matrix.org`).
## How do I opt out and keep my room from being indexed by search engines?
All public Matrix rooms are accessible to view in the Matrix Public Archive. But only
rooms with history visibility set to `world_readable` will be indexed by search engines.
Also see https://github.com/matrix-org/matrix-public-archive/issues/47
## Technical details
The main readme has an [technical overview](../README.md#technical-overview) of the
project. Here are a few more details.
### How do I figure out what version of the Matrix Public Archive is running?
Just visit the `/health-check` endpoint which will return information like the following:
```
{
"ok": true,
"commit": "954b22995a44bf11bfcd5850b62e206e46ee2db9",
"version": "main",
"versionDate": "2023-04-05T09:26:12.524Z",
"packageVersion": "0.0.0"
}
```
### How does the archive room URL relate to what is displayed on the page?
We start the end of the date/time specified in the URL looking backward up to the limit.
### Why does the time selector only appear for some pages?
The time selector only appears for pages that have a lot of messages on a given
day/hour/minute/second (more than the configured `archiveMessageLimit`).

14
test/README.md → docs/testing.md
View File

@ -4,12 +4,15 @@
If you haven't setup `matrix-public-archive` yet, see the [_Setup_ section in the root `README.md`](../README.md#setup)
Then we need to setup the federation cluster of homeservers that we will test with.
Sorry, this isn't automated yet when you run the tests 🙇
```sh
# Build the test homeserver image that are pre-configured to federate with each other
$ docker pull matrixdotorg/synapse:latest
$ docker build -t matrix-public-archive-test-homeserver -f test/dockerfiles/Synapse.Dockerfile test/dockerfiles/
# Start the test homeservers
$ docker-compose --project-name matrix_public_archive_test -f test/docker-compose.yml up -d --no-recreate
```
@ -19,14 +22,23 @@ $ docker-compose --project-name matrix_public_archive_test -f test/docker-compos
$ npm run test
```
Or if you want to keep `matrix-public-archive` server running after the tests run and explore the output from the interactive URL's printed on the screen, use:
Or if you want to keep the Matrix Public Archive server running after the tests run and
explore the UI from the interactive URL's printed on the screen to better debug, use:
```sh
$ npm run test-interactive
```
Caveat: You might not see the same result that a test is seeing when visiting the
interactive URL. Some tests set config like the `archiveMessageLimit` which is reset
after each test case unless you are using `npm run test-interactive` and visiting the
interactive URL for a failed test. Otherwise, we reset config between each test case so
they don't leak and contaminate each other.
### Developer utility
Some copy-pasta to help you manage the Docker containers for the test homeservers:
```sh
$ docker ps --all | grep test_hs
$ docker logs -f --tail 10 matrix_public_archive_test_hs1_1

55
docs/tracing.md Normal file
View File

@ -0,0 +1,55 @@
# Tracing
Tracing allows you to see the flow of a request through the system and where time is
taken up in functions. This is useful for debugging and performance analysis.
<img src="https://user-images.githubusercontent.com/558581/180586026-ff6c653e-a54d-4cf4-abc8-a8c51971aad5.png" width="612">
## Setup
1. Get the all-in-one Jaeger Docker container running (via https://www.jaegertracing.io/docs/1.35/getting-started/)
```
docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
-e COLLECTOR_OTLP_ENABLED=true \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 5775:5775/udp \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
-p 14250:14250 \
-p 14268:14268 \
-p 14269:14269 \
-p 9411:9411 \
jaegertracing/all-in-one:1.35
```
1. Add `jaegerTracesEndpoint` to your `config.json`:
```json5
{
// ...
jaegerTracesEndpoint: 'http://localhost:14268/api/traces',
}
```
## Run the app with the OpenTelemetry tracing enabled
```
npm run start -- --tracing
# or
npm run start-dev -- --tracing
```
Manually:
```
node --require './server/tracing.js' server/server.js
```
## Viewing traces in Jaeger
Once you have the all-in-one Jaeger Docker container running, just visit
http://localhost:16686 to see a dashboard of the collected traces and dive in.
Traces are made up of many spans. Each span defines a `traceId` which it is associated with.

58
server/README.md
View File

@ -1,58 +0,0 @@
## Tracing
Run the app with the OpenTelemetry tracing.
```
npm run start -- --tracing
# or
npm run start-dev -- --tracing
```
Manually:
```
node --require './server/tracing.js' server/server.js
```
---
Traces are made up of many spans. Each span defines a `traceId` which it is associated with.
## Viewing traces in Jaeger
- Dashboard where users can see visualisations -> http://localhost:16686
via https://www.jaegertracing.io/docs/1.35/getting-started/
```
docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
-e COLLECTOR_OTLP_ENABLED=true \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 5775:5775/udp \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
-p 14250:14250 \
-p 14268:14268 \
-p 14269:14269 \
-p 9411:9411 \
jaegertracing/all-in-one:1.35
```
| Port | Protocol | Component | Function |
| ----- | -------- | --------- | -------------------------------------------------------------------------------------------- |
| 6831 | UDP | agent | accept jaeger.thrift over Thrift-compact protocol (used by most SDKs) |
| 6832 | UDP | agent | accept jaeger.thrift over Thrift-binary protocol (used by Node.js SDK) |
| 5775 | UDP | agent | (deprecated) accept zipkin.thrift over compact Thrift protocol (used by legacy clients only) |
| 5778 | HTTP | agent | serve configs (sampling, etc.) |
| 16686 | HTTP | query | serve frontend |
| 4317 | HTTP | collector | accept OpenTelemetry Protocol (OTLP) over gRPC, if enabled |
| 4318 | HTTP | collector | accept OpenTelemetry Protocol (OTLP) over HTTP, if enabled |
| 14268 | HTTP | collector | accept jaeger.thrift directly from clients |
| 14250 | HTTP | collector | accept model.proto |
| 9411 | HTTP | collector | Zipkin compatible endpoint (optional) |
With Service Performance Monitoring (SPM)