Commit Graph

44 Commits

Author SHA1 Message Date
Eric Eastwood 07bc094890
Enable tracing by config so we can enable from argv, env variable, or config file (#41) 2022-07-14 11:26:53 -05:00
Eric Eastwood ddfe94beab
OpenTelemetry tracing so we can see spans where the app is taking time (#27)
OpenTelemetry tracing so we can see spans where the app is taking time.
For the user, we specifically show the spans for the external API HTTP requests
that are slow (so we know when the Matrix API is being slow).

Enable tracing:

 - `npm run start -- --tracing`
 - `npm run start-dev -- --tracing`

What does this PR change:

 - Adds OpenTelemetry tracing with some of the automatic instrumentation (includes HTTP and express)
    - We ignore traces for serving static assets (just noise)
 - Adds `X-Trace-Id` to the response headers
 - Adds `window.tracingSpansForRequest` which includes the external HTTP API requests made during the request
 - Adds a fancy 504 timeout page that includes trace details and lists the slow HTTP requests
 - Adds `jaegerTracesEndpoint` configuration to export tracing spans to Jaeger
 - Related to, https://github.com/matrix-org/matrix-public-archive/issues/26
2022-07-14 11:08:50 -05:00
Eric Eastwood 13eb92b067
Make sure we finish sending the HTML payload before we exit the process (#38)
I encountered a page which responded successfully but all of the Hydrogen HTML was missing. It just had the boilerplate around it.

What I am guessing happened is that since `process.send` is async, with a sufficiently large
payload and race condition, `process.exit(0)` was being called before it finished sending.

Related:
 - https://stackoverflow.com/questions/34627546/process-send-is-sync-async-on-nix-windows
 - 56d9584a0e
 - https://github.com/nodejs/node/issues/6767
2022-07-06 19:24:29 -05:00
Eric Eastwood 7eaa103a28
Fix large option payloads throwing E2BIG and ENAMETOOLONG (#37)
Follow-up to https://github.com/matrix-org/matrix-public-archive/pull/36
2022-07-05 18:00:29 -05:00
Eric Eastwood f738dbc1da
Stop Hydrogen from running in the background after we get our SSR HTML render data (#36)
We now run the Hydrogen render in a `child_process` so we can exit the whole render process. We still use the `vm` to setup the browser-like globals. With a `vm`, everything continues to run even after it returns and there isn't a way to clean up, stop, kill, terminate the vm script or context so we need this extra `child_process` now to clean up. I don't like the complexity necessary for this though. I wish the `vm` API allowed for this use case. The only way to stop a `vm` is the `timeout` and we want to stop as soon as we return.

Fix https://github.com/matrix-org/matrix-public-archive/issues/34
2022-07-05 17:30:52 -05:00
Eric Eastwood 9c8f980e2a Remove random log 2022-07-05 11:21:04 -05:00
Eric Eastwood 17f2c399dd
Expose arguments for `renderHydrogenToString` (pure function) so we can reproduce when error occurs while we render Hydrogen (#35)
`renderHydrogenToString` is a pure function (probably) which means it will give the same output given the same input. This means, that if we give it a certain input and an error occurs, we should be able to reproduce it again if we have the arguments. This PR exposes those arguments in the logged error so we can investigate what's going wrong.

Added so we can investigate https://github.com/matrix-org/matrix-public-archive/issues/34 better and reproduce locally.
2022-07-05 10:30:12 -05:00
Eric Eastwood cfbd6182a9 Explain why parallel 2022-06-29 13:57:06 +02:00
Eric Eastwood 4690349816
Remove unneeded `include_redundant_members` from `/messages` `filter` and test that member state is still visible (#29)
Follow-up to https://github.com/matrix-org/matrix-public-archive/pull/28#discussion_r909428366
2022-06-29 06:56:13 -05:00
Eric Eastwood 57174db6e0
Fix archive not responding in big rooms because homeserver times out on `/context` request
Add `filter={"lazy_load_members":true}` so that `/context` responds without timing out by returning just the state for the sender of the included event. Otherwise, the homeserver returns all state in the room at that point in time which in big rooms, can be 100k member events that we don't care about anyway. Synapse seems to timeout at about the ~5k state event mark.


## Dev notes

Without `filter={"lazy_load_members":true}`, Synapse can only handle ~4k member events (probably state in general) before `/context` times out before it ever responds in the 180 second window. I'm only looking at the member count as a rough proxy but the number of member events + state will be more.


Loads | Member count | Alias | Matrix.to | API request
--- | --- | --- | --- | ---
 | 39k | `#matrix:matrix.org` | [🔗 ](https://matrix.to/#/!OGEhHVWSdvArJzumhm:matrix.org/$xo-tESRNP1Vg1RxLXIfmMeO6dA6-u9XuE2lv6toeKcw?via=matrix.org) | `https://matrix-client.matrix.org/_matrix/client/r0/rooms/!OGEhHVWSdvArJzumhm:matrix.org/context/$xo-tESRNP1Vg1RxLXIfmMeO6dA6-u9XuE2lv6toeKcw?limit=0`
 | 31k |  `#openwisp_general:gitter.im` | [🔗 ](https://matrix.to/#/!RBzfoBeqYcCwLAAenz:gitter.im/$YFjmbmH0NRPmfVsPyxhM0jcK4RFR_CdCigtHSgCTLSc?via=matrix.org) |`https://matrix-client.matrix.org/_matrix/client/r0/rooms/!RBzfoBeqYcCwLAAenz:gitter.im/context/$YFjmbmH0NRPmfVsPyxhM0jcK4RFR_CdCigtHSgCTLSc?limit=0`
 | 16k | `#raspberrypi:matrix.org` | [🔗 ](https://matrix.to/#/!wOlkWNmgkAZFxbTaqj:matrix.org/$WMh_QauoyW6cjFDuUeNgsnUSgDn7C2XlrUnlijhdmdk?via=matrix.org) | `https://matrix-client.matrix.org/_matrix/client/r0/rooms/!wOlkWNmgkAZFxbTaqj:matrix.org/context/$WMh_QauoyW6cjFDuUeNgsnUSgDn7C2XlrUnlijhdmdk?limit=0`
Loads with warm cache | 7.2k | `#element-android:matrix.org` | [🔗 ](https://matrix.to/#/!AZozoWghOYSIAfaZjJ:matrix.org/$Qmb8vmaD91PIM6ROfh-2jApPoJgH2Q7NnjiwRdiEPZE?via=matrix.org) | `https://matrix-client.matrix.org/_matrix/client/r0/rooms/!AZozoWghOYSIAfaZjJ:matrix.org/context/$Qmb8vmaD91PIM6ROfh-2jApPoJgH2Q7NnjiwRdiEPZE?limit=0`
  | 4k | `#nim-science:envs.net` | [🔗 ](https://matrix.to/#/!IpFtPSbgfrZrVcVyti:envs.net/$bDRop4yvjFOl3HMouAyx4mtau-JaAxvBUJ-MqftAm7E?via=matrix.org) | `https://matrix-client.matrix.org/_matrix/client/r0/rooms/!IpFtPSbgfrZrVcVyti:envs.net/context/$bDRop4yvjFOl3HMouAyx4mtau-JaAxvBUJ-MqftAm7E?limit=0`
  | 2.3k | `#nim-webdev:matrix.org` | [🔗 ](https://matrix.to/#/!EoyccMaVGwdqyfKMAL:matrix.org/$VdRyNb1F-gLdDP9nDYK7xAh_LcE822fEfDYWz5re8dE?via=matrix.org) | `https://matrix-client.matrix.org/_matrix/client/r0/rooms/!EoyccMaVGwdqyfKMAL:matrix.org/context/$VdRyNb1F-gLdDP9nDYK7xAh_LcE822fEfDYWz5re8dE?limit=0`
2022-06-29 05:37:40 -05:00
Eric Eastwood bd5c14242e
Make sure container is able to start up (#23)
Follow-up to https://github.com/matrix-org/matrix-public-archive/pull/22
2022-06-15 17:12:44 -05:00
Eric Eastwood 952a3acd4a
Explain how config inherits (#17) 2022-06-09 21:37:07 -05:00
Eric Eastwood 9fc71a3412
Remove `matrix-bot-sdk` usage in tests (#15)
Remove `matrix-bot-sdk` usage in tests because it didn't have timestamp massaging `?ts` and it's not really necessary to rely on since we can just call the API directly 🤷. `matrix-bot-sdk` is also very annoying having to build rust crypto packages.

We're now using direct `fetch` requests against the Matrix API and lightweight `client` object.

All 3 current tests pass 
2022-06-09 20:44:57 -05:00
Eric Eastwood 940c73868f
Organize file and add more nodemon logging (#14)
Spawned from https://github.com/vitejs/vite/issues/8492
2022-06-07 20:27:22 -05:00
Eric Eastwood cc958326ea
Use new Hydrogen light-theme asset directly instead of legacy style.css (#13) 2022-06-07 20:21:56 -05:00
Eric Eastwood 40f9d2ea5a
Update to work with latest `hydrogen-view-sdk@0.0.12` (#10)
Get this project running again after a few months of changes
from `hydrogen-view-sdk` and now finally after
https://github.com/vector-im/hydrogen-web/pull/693
merged to make the SDK friendly to locally link and develop on.
2022-06-06 18:58:45 -05:00
Eric Eastwood c34ed9ea0e Fix last event in day not being shown 2022-02-24 13:06:19 -06:00
Eric Eastwood 2153fa4852 Use file that will resolve in new `./assets/*` export
Because we had to update the `assets` `export` to avoid the following deprecation warning:

```
(node:133896) [DEP0148] DeprecationWarning: Use of deprecated folder mapping "./assets/" in the "exports" field module resolution of the package at C:\Users\MLM\Documents\GitHub\element\matrix-public-archive\node_modules\hydrogen-view-sdk\package.json.
Update this package.json to use a subpath pattern like "./assets/*".
(Use `node --trace-deprecation ...` to show where the warning was created)
```

`hydrogen-view-sdk` `package.json` before:
```json
{
  "exports": {
      ".": {
          "import": "./lib-build/hydrogen.es.js",
          "require": "./lib-build/hydrogen.cjs.js"
      },
      "./paths/vite": "./paths/vite.js",
      "./style.css": "./style.css",
      "./assets/": "./asset-build/assets/"
  }
}
```

`hydrogen-view-sdk` `package.json` after:

```json
{
  "exports": {
      ".": {
          "import": "./lib-build/hydrogen.es.js",
          "require": "./lib-build/hydrogen.cjs.js"
      },
      "./paths/vite": "./paths/vite.js",
      "./style.css": "./style.css",
      "./assets/*": "./asset-build/assets/*"
  }
}
```
2022-02-24 13:02:55 -06:00
Eric Eastwood db6d3797d7 Working e2e test 2022-02-24 03:27:53 -06:00
Eric Eastwood 839e31a35e E2E test but still failing because fetching from start of day before test events happened 2022-02-23 21:25:05 -06:00
Eric Eastwood 0f493a241e Federated homeservers in Docker for e2e tests 2022-02-22 20:25:24 -06:00
Eric Eastwood aea382e4f8 WIP: start of tests 2022-02-22 16:06:29 -06:00
Eric Eastwood 7d95681611 Make sure input value is zero padded to parse properly 2022-02-18 12:34:21 -06:00
Eric Eastwood fe3f515862 Calendar styles and server hydrogen assets 2022-02-17 16:56:54 -06:00
Eric Eastwood b401cbbc3a Add active date to calendar 2022-02-16 23:08:18 -06:00
Eric Eastwood 6a5d011a45 Link calendar days 2022-02-16 19:58:32 -06:00
Eric Eastwood d9af4b6b7c Reload when locally linked hydrogen-web changes 2022-02-16 14:55:52 -06:00
Eric Eastwood 56f7d17126 Ensure deps available 2022-02-16 14:13:39 -06:00
Eric Eastwood 5386c5aac7 Point to the right file path 2022-02-16 14:13:30 -06:00
Eric Eastwood 166857e0de Fix lints 2022-02-15 21:33:31 -06:00
Eric Eastwood 6c1cf6d46a Stop restarting server when the client bundle updates 2022-02-15 17:30:30 -06:00
Eric Eastwood e0279e080e Add npm run start-dev to build vite and run server at once 2022-02-15 17:17:14 -06:00
Eric Eastwood 72a6297ae5 Re-mount on client to get JS interactivity 2022-02-15 01:33:16 -06:00
Eric Eastwood 0aab8383c5 Pass data by window which is also serialized on the client 2022-02-14 23:20:25 -06:00
Eric Eastwood 82ba92b806 Add custom RightPanel support 2022-02-10 02:21:49 -06:00
Eric Eastwood c0a2a65c2f Add support for the room header (RoomView) 2022-02-10 01:42:02 -06:00
Eric Eastwood 2378ed72c7 Add in date routes 2022-02-09 20:29:18 -06:00
Eric Eastwood f80f9fbbb6 Make displayname and avatars show up and handle avatar changes 2022-02-09 02:07:16 -06:00
Eric Eastwood 1c7c6bf440 Pass config so images render with correct media repo 2022-02-09 01:49:02 -06:00
Eric Eastwood a49657f9e3 Fetch and render from homeserver 2022-02-07 19:55:11 -06:00
Eric Eastwood 4ab26ef2d1 Strip out PoC pieces (just leave the SSR stuff) 2022-02-05 04:35:02 -06:00
Eric Eastwood e95c8ad4e6 Add CSS support 2022-02-04 01:54:12 -06:00
Eric Eastwood 4e881d6487 Working render (without CSS)
See ![](https://user-images.githubusercontent.com/558581/152488518-a93d320e-608e-4bf4-90a6-43787134894f.png)
2022-02-04 01:24:31 -06:00
Eric Eastwood 3a54d3b652 SSR with linkedom
Currently doesn't work because the `window` object doesn't include
things like `window.crypto`.
2022-02-03 23:44:50 -06:00