Happens to address part of https://github.com/matrix-org/matrix-public-archive/issues/271
but made primarily as a follow-up to https://github.com/matrix-org/matrix-public-archive/pull/239
---
Only 42% rooms on the `matrix.org` room directory are `world_readable` which means we will get pages of rooms that are half-empty most of the time if we just naively fetch 9 rooms at a time.
Ideally, we would be able to just add a filter directly to `/publicRooms` in order to only grab the `world_readable` rooms and still get full pages but the filter option doesn't allow us to slice by `world_readable` history visibility.
Instead, we have to paginate until we get a full grid of 9 rooms, then make a final `/publicRooms` request to backtrack to the exact continuation point so next page won't skip any rooms in between.
---
We had empty spaces in the grid before because some rooms in the room directory are private which we filtered out before. But that was a much more rare experience since only 2% of rooms were private .
Only `world_readable` can be considered as opting into having history publicly on the web. Anything else must not be archived until there's a dedicated state event for opting into archiving.
- Default to a nice `[matrix]` banner
- There is room for improvement here when the Matrix Public Archive gets it's own logo (https://github.com/matrix-org/matrix-public-archive/issues/94) and maybe says "Matrix Public Archive" somewhere in the banner.
- This is good enough for now (and certainly better than downstream previews using the first image on the page).
- For rooms, it will use the room avatar
Part of https://github.com/matrix-org/matrix-public-archive/issues/202
Image is sized to 1200x630 to match conventions of `og:image`.
Crafted the banner image by modifying the header on the room directory homepage and taking a node screenshot. Page zoom @ 175%
Set `X-Date-Temporal-Context: [past|present|future]` header for easy cache rules:
- Cache `past` things heavily
- Cache `present`/`future` things for 5 minutes
This accomplishes the goal we set out for:
> - We can cache all responses except for the latest UTC day (and anything in the future). ex. `/!aMzLHLvScQCGKDNqCB:gitter.im/date/2022/10/13`
> - For the latest day, we could set the cache expire after 5 minutes or so
>
> *-- [Matrix Public Archive deployment issue](https://github.com/vector-im/sre-internal/issues/2079)*
And this way we don't have to do any fancy date parsing and comparison from the URL which is probably not even possible Cloudflare cache rules.
- Rename `public` -> `client` so it doesn't get copied automagically as-is (without hashes which we want for cache busting), https://vitejs.dev/guide/assets.html#the-public-directory
- We still build the version files to `public/` so their copied as-is and Vite handles it for us (so we can use `emptyOutDir`)
- Use a multiple entrypoint `.js` Vite build so things can be more intelligently bundled and take less time
- We aren't using library mode because it doesn't minify or bundle assets
- Using hash asset tags for cache busting. Hash of the file included in the file name
- We lookup these hashed assets from `manifest.json` that Vite builds (https://vitejs.dev/guide/backend-integration.html) to serve and preload
- In terms of optimized bundles, I know the current output isn't great now but will have to opt to fix that up separately in the future. Tracked by https://github.com/matrix-org/matrix-public-archive/issues/176
We already read it once for the `/health-check` endpoint and cached the response but this way we can use `getVersionTags()` everywhere without worrying about it.
Also, it's no longer `async` so we can use it in things like Express route paths and CDN asset tags more easily.
Fix https://github.com/matrix-org/matrix-public-archive/issues/59
Other updates:
- Update tests to use `/roomid/room1/date/2022/01/03` format instead of trying to retrofit the weird alias stuff on there. Which also makes the fancy to actual URL utilities much more simple.
- Update to specify `archiveMessageLimit` in the test case because pages have different number of events depending on if we are against a boundary, hidden events, etc.
- Less test bulk
- Single source of truth: there is no mismatch between the comment and the expectations (we already caught a few mistakes in the conversion thanks to this benefit)
- Easier to maintain and update
- Fix https://github.com/matrix-org/matrix-public-archive/issues/7
- A URL with time looks like
- `/r/too-many-messages-on-day:my.synapse.server/date/2022/11/16T23:59`
- Or when more precision is required (seconds): `/r/too-many-messages-on-day:my.synapse.server/date/2022/11/16T23:59:59`
- Add new custom time picker/scrubber (pictured below) with momentum scrubbing
- Native built-in `<input type="time">` for easier picking if you prefer that and accessibility.
- Uses localized time strings
- Design inspired by Thiago Sanchez's *Time Zone Translate* concept, https://dribbble.com/shots/14590546-Time-Zone-Translate
This change enables CORS support in the archive — to allow web developers to create web applications with frontend JavaScript code that can fetch pages from the archive (for example, for scraping content from chat logs).
Otherwise, without this change, web developers can’t create web apps with frontend JavaScript that can fetch chat logs from the archive and then consume the content of the logs.
It’s imaginable that web developers may find use cases for consuming the chat logs in the archive from frontend JavaScript code — at the simplest level, web apps that fetch and scrape logs to get data out of them or to pull out particular snippets from the logs.
Developers can anyway already scrape the contents of the archive — by using server-side programming languages or by using `curl` or whatever from the command line. They just can’t do the same from frontend JavaScript code, unless CORS support is enabled.
Fix https://github.com/matrix-org/matrix-public-archive/issues/46
Follow-up to https://github.com/matrix-org/matrix-public-archive/pull/71
Summary:
- Changes the "Jump to next activity in room" to actually continue you to the next 100 messages ahead. Previously, it only jumped you to the single next event in the room which meant a lot of backwards overlap each time.
- Jumping this direction will also start your scroll position at the top of the timeline to continue reading seamlessly `?continue=top`
- Adds "Jump to previous activity in room" to the top of the timeline to continue reading the previous part of the conversation.
[1]: There is a caveat with seamless here which is also commented on in the code:
> XXX: This is flawed in the fact that when we go `/messages?dir=b` it could backfill messages which will fill up the response before we perfectly connect and continue from the position they were jumping from before. When `/messages?dir=f` backfills, we won't have this problem anymore because any messages backfilled in the forwards direction would be picked up the same going backwards.
(need forwards fill MSC)
This helps when someone just pastes a room alias on the end of the domain,
- `/#room-alias:server` -> `/r/room-alias:server`
- `/r/#room-alias:server/date/2022/10/27` -> `/r/room-alias:server/date/2022/10/27`
Since these redirects happen on the client, we can't write any e2e tests. Those e2e tests do everything but run client-side JavaScript.
Follow-up to https://github.com/matrix-org/matrix-public-archive/pull/107
Part of https://github.com/matrix-org/matrix-public-archive/issues/25
Also does friendly redirects if you don't exactly use the right URL pattern.
For example, if you paste the full room ID with the `!` like `/roomid/!foo:bar`,
it will properly redirect you to `/roomid/foo:bar`. It also does this sort of
thing for URL encoded room ID's and aliases.
Fix https://github.com/matrix-org/matrix-public-archive/issues/25
Page-load with the correct homeserver selected (according to `?homeserver`).
Fix https://github.com/matrix-org/matrix-public-archive/issues/92
Also makes sure that the `?homeserver` is always available somewhere in the list; whether that be in the available homeserver list or the added homeserver list depending on it someone cleared it out or never had it because they visited from someone else's link.
Fix https://github.com/matrix-org/matrix-public-archive/issues/80
```
RethrownError: Unable to fetch rooms from room directory (homeserver=http://localhost:8008/)
searchTerm=, paginationToken=undefined, limit=9
at matrix-public-archive\server\routes\room-directory-routes.js:55:13
--- Original Error ---
Error: HTTP Error Response: 500 Internal Server Error: {"errcode":"M_UNKNOWN","error":"Internal server error"}
URL=http://localhost:8008/_matrix/client/v3/publicRooms?
at checkResponseStatus (matrix-public-archive\server\lib\fetch-endpoint.js:21:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async fetchEndpoint (matrix-public-archive\server\lib\fetch-endpoint.js:38:3)
at async fetchEndpointAsJson (matrix-public-archive\server\lib\fetch-endpoint.js:63:15)
at async fetchPublicRooms (matrix-public-archive\server\lib\matrix-utils\fetch-public-rooms.js:26:26)
at async matrix-public-archive\server\tracing\trace-utilities.js:31:24
at async matrix-public-archive\server\routes\room-directory-routes.js:45:62
```
Regressed in https://github.com/matrix-org/matrix-public-archive/pull/61 where we tried to serve this under `/css/hydrogen-styles.css` but it doesn't work because all of the image and font references in the CSS file expect it to be at the domain root so just reverted back to serving at the root `/`.
1. Add surrounding messages to the given messages so we have a full screen of content to make it feel lively even in quiet rooms
- As you scroll around the timeline across different days, the date changes in the URL, calendar, etc
2. Add summary item to the bottom of the timeline that explains if we couldn't find any messages in the specific day requested
- Also allows you to the jump to the next activity in the room. Adds `/:roomId/jump?ts=xxx&dir=[f|b]` to facilitate this.
- Part of https://github.com/matrix-org/matrix-public-archive/issues/46
3. Add developer options modal which is linked from the bottom of the right-panel
- Adds an option so you can debug the `IntersectionObserver` and how it's selecting the active day from the top-edge of the scroll viewport.
- In the future, this will also include a nice little visualization of the backend timing traces
Follow-up to https://github.com/matrix-org/matrix-public-archive/pull/36
Render pipeline separation of concerns:
1. Run in `child_process`
2. Hydrogen render
It's now just a generic `child_process` runner that runs the Hydrogen render in it. This eliminates the windy path of the 1-4 steps that was only held together by the file names themselves.
Split off from https://github.com/matrix-org/matrix-public-archive/pull/43
Listen to `process.on('uncaughtException', ...)` and handle the async errors ourselves so it no longer fails the child process.
And if the process does exit with status code 1 (error), we have those underlying errors serialized and shown.
Eric Eastwood