Commit Graph

43 Commits

Author SHA1 Message Date
Eric Eastwood f91b8f07f3
Try and make the seamless pagination test more robust (wait for Synapse to settle) (#281)
Previously was seeing some failures like this locally
```
  1 failing

  1) matrix-viewer
       Matrix Viewer
         Room directory
           pagination is seamless:

      AssertionError [ERR_ASSERTION]: Make sure we saw all visible rooms paginating through the directory
      + expected - actual

         "planet-1689366398300-room-29"
         "planet-1689366398300-room-31"
         "planet-1689366398300-room-32"
         "planet-1689366398300-room-34"
      -  "planet-1689366398300-room-34"
         "planet-1689366398300-room-35"
         "planet-1689366398300-room-37"
         "planet-1689366398300-room-38"
         "planet-1689366398300-room-4"

      at Context.<anonymous> (test/e2e-tests.js:2835:16)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
```
2023-07-14 16:04:05 -05:00
Eric Eastwood 5d444d5044
Rename project from "Matrix Public Archive" to "Matrix Viewer" (#280)
Mentioned in the recent blog post: https://matrix.org/blog/2023/07/what-happened-with-the-archive/#next-steps

The canonical `view.matrix.org` instance is not live yet but this is pre-requisite work for that to happen.
2023-07-14 15:52:35 -05:00
Eric Eastwood dd2cd9126d
Only show `world_readable` rooms in the room directory (#276)
Happens to address part of https://github.com/matrix-org/matrix-public-archive/issues/271
but made primarily as a follow-up to https://github.com/matrix-org/matrix-public-archive/pull/239

---

Only 42% rooms on the `matrix.org` room directory are `world_readable` which means we will get pages of rooms that are half-empty most of the time if we just naively fetch 9 rooms at a time.

Ideally, we would be able to just add a filter directly to `/publicRooms` in order to only grab the `world_readable` rooms and still get full pages but the filter option doesn't allow us to slice by `world_readable` history visibility.

Instead, we have to paginate until we get a full grid of 9 rooms, then make a final `/publicRooms` request to backtrack to the exact continuation point so next page won't skip any rooms in between.

---

We had empty spaces in the grid before because some rooms in the room directory are private which we filtered out before. But that was a much more rare experience since only 2% of rooms were private .
2023-06-30 03:08:32 -05:00
Eric Eastwood 59c9d3180e
Fix `18+` false positives with NSFW check (#279)
Was noticing false positives with our test room names like: `planet-1688081266353-room-18`

Before:
```regex
/(\b|_)18+(\b|_)/i
```

After:
```regex
/(\b|_|-|\s|^)18\+(\b|_|-|\s|$)/i
```
2023-06-29 18:58:53 -05:00
Tulir Asokan 1d3e930fbd
Don't allow previewing `shared` history rooms (#239)
Only `world_readable` can be considered as opting into having history publicly on the web. Anything else must not be archived until there's a dedicated state event for opting into archiving.
2023-06-27 16:56:58 -05:00
Eric Eastwood 1dd63212c0
Add reason why the archive bot is joining the room (#262)
Using the join `reason` added in [MSC2367](https://github.com/matrix-org/matrix-spec-proposals/pull/2367). Unfortunately, this PR doesn't have much effect because it doesn't look like many clients support it yet (Element doesn't support it for example).

Part of https://github.com/matrix-org/matrix-public-archive/issues/257
2023-06-09 16:05:20 -05:00
Eric Eastwood a9aa08f24a
Catch NSFW rooms with underscores (#231)
`\b` includes `_` which is a bit unexpected since we expect a room that looks like `NSFW_foo` to be caught in safe search
2023-05-23 12:29:19 -05:00
Eric Eastwood 198e8c09be
Mark NSFW room pages with `<meta name="rating" content="adult">` (#216)
Related docs:

 - https://developers.google.com/search/docs/crawling-indexing/safesearch
 - https://developers.google.com/search/docs/crawling-indexing/special-tags
2023-05-05 15:36:26 -05:00
Eric Eastwood b10884505a
Fix time selector showing when less than the page limit of messages (#213)
Fix https://github.com/matrix-org/matrix-public-archive/issues/211
2023-05-04 20:50:43 -05:00
Eric Eastwood 9b067f8637
Set `X-Date-Temporal-Context` header for easy cache rules (#209)
Set `X-Date-Temporal-Context: [past|present|future]` header for easy cache rules:

 - Cache `past` things heavily
 - Cache `present`/`future` things for 5 minutes
 
This accomplishes the goal we set out for:

> - We can cache all responses except for the latest UTC day (and anything in the future). ex. `/!aMzLHLvScQCGKDNqCB:gitter.im/date/2022/10/13`
>    - For the latest day, we could set the cache expire after 5 minutes or so
>
> *-- [Matrix Public Archive deployment issue](https://github.com/vector-im/sre-internal/issues/2079)*

And this way we don't have to do any fancy date parsing and comparison from the URL which is probably not even possible Cloudflare cache rules.
2023-05-04 13:42:59 -05:00
Eric Eastwood b70439e95b
Add safe search filter for NSFW rooms (#208)
Fix https://github.com/matrix-org/matrix-public-archive/issues/89
2023-05-03 04:45:33 -05:00
Eric Eastwood 9078abf4f1
Timeout requests and stop processing further (#204)
Fix https://github.com/matrix-org/matrix-public-archive/issues/148
Fix https://github.com/matrix-org/matrix-public-archive/issues/40

 - Apply timeout middleware to all room directory and room routes
 - Stop messing with the response after we timeout. Fix https://github.com/matrix-org/matrix-public-archive/issues/148
    - This also involves cancelling any `async/await` things like requests in the routes so we throw an abort error instead of continuing on. Fix https://github.com/matrix-org/matrix-public-archive/issues/40
 - Also abort the route if we see that the user closed the request before we could respond to them
 - Bumps minimum supported Node.js version to v18 because we're now using the built-in native `fetch` in Node.js vs `node-fetch`. This gives us the custom `signal.reason` that we aborted with instead of a generic `AbortError`.
    - This also means we had to add some instrumentation for `fetch` which uses `undici` under the hood. Settled on some unofficial instrumentation: [`opentelemetry-instrumentation-fetch-node`](https://www.npmjs.com/package/opentelemetry-instrumentation-fetch-node)
2023-05-02 00:39:01 -05:00
Eric Eastwood 53a1d4b43b
Update docs in preparation for Matrix Public Archive being generally available (#194) 2023-04-27 00:22:41 -05:00
Eric Eastwood 0f26dc94d3
Migrate from `eslint-plugin-node` to `eslint-plugin-n` (#179) 2023-04-25 00:39:59 -05:00
Eric Eastwood 551b4e72d1
Follow tombstone and predecessor history (#167)
Fix https://github.com/matrix-org/matrix-public-archive/issues/59

Other updates:

 - Update tests to use `/roomid/room1/date/2022/01/03` format instead of trying to retrofit the weird alias stuff on there. Which also makes the fancy to actual URL utilities much more simple.
 - Update to specify `archiveMessageLimit` in the test case because pages have different number of events depending on if we are against a boundary, hidden events, etc.
2023-04-19 01:26:15 -05:00
Eric Eastwood 6c789eae69
Do our best to get the user to the right place and try joining `via` derived server name (#168)
Split out from https://github.com/matrix-org/matrix-public-archive/pull/167
2023-04-11 15:09:44 -05:00
Eric Eastwood 5aafa01f38
Update linting so we don't need omit the context filling name of some variables (#165)
`"no-unused-vars": ["error", { "destructuredArrayIgnorePattern": "^_" }],` was only [introduced in `eslint@8.11.0`](0fd6bb213a/CHANGELOG.md) so we had to update
2023-04-07 13:01:20 -05:00
Eric Eastwood 57d2cb3dd3
Refactor tests to use single source of truth ASCII diagram (#164)
- Less test bulk
 - Single source of truth: there is no mismatch between the comment and the expectations (we already caught a few mistakes in the conversion thanks to this benefit)
 - Easier to maintain and update
2023-04-07 12:52:41 -05:00
Eric Eastwood 954b22995a
Add a way to select time of day (#139)
- Fix https://github.com/matrix-org/matrix-public-archive/issues/7
 - A URL with time looks like
    - `/r/too-many-messages-on-day:my.synapse.server/date/2022/11/16T23:59`
    - Or when more precision is required (seconds): `/r/too-many-messages-on-day:my.synapse.server/date/2022/11/16T23:59:59`
 - Add new custom time picker/scrubber (pictured below) with momentum scrubbing
    - Native built-in `<input type="time">` for easier picking if you prefer that and accessibility.
    - Uses localized time strings
    - Design inspired by Thiago Sanchez's *Time Zone Translate* concept, https://dribbble.com/shots/14590546-Time-Zone-Translate
2023-04-05 04:25:31 -05:00
Eric Eastwood 026a08a77a
Jump forward and backward seamlessly (#121)
Fix https://github.com/matrix-org/matrix-public-archive/issues/120
Follow-up to https://github.com/matrix-org/matrix-public-archive/pull/114

 - Uses event permalinking (`?at=$xxx`) to continue the scroll where you should start reading again.
 - When we jump forwards, we make sure that we go a day back to ensure there isn't more than the page limit between where we jumped from and the day so we don't lose any messages in a gap.
2022-11-03 05:06:53 -05:00
Eric Eastwood 2dff7ecea5
Refactor `fetchEndpointAsText`/`fetchEndpointAsJson` to return `res` alongside `data` (#122)
Split out of https://github.com/matrix-org/matrix-public-archive/pull/121
where we needed to use `res.url`.
2022-11-03 04:12:00 -05:00
Eric Eastwood 08254cbb49
Add a way to jump forwards and backwards to more activity in the room (seamless navigation) (#114)
Fix https://github.com/matrix-org/matrix-public-archive/issues/46
Follow-up to https://github.com/matrix-org/matrix-public-archive/pull/71

Summary:

 - Changes the "Jump to next activity in room" to actually continue you to the next 100 messages ahead. Previously, it only jumped you to the single next event in the room which meant a lot of backwards overlap each time.
    - Jumping this direction will also start your scroll position at the top of the timeline to continue reading seamlessly `?continue=top`
 - Adds "Jump to previous activity in room" to the top of the timeline to continue reading the previous part of the conversation.

[1]: There is a caveat with seamless here which is also commented on in the code:

> XXX: This is flawed in the fact that when we go `/messages?dir=b` it could  backfill messages which will fill up the response before we perfectly connect and  continue from the position they were jumping from before. When `/messages?dir=f`  backfills, we won't have this problem anymore because any messages backfilled in  the forwards direction would be picked up the same going backwards.

(need forwards fill MSC)
2022-11-02 04:27:30 -05:00
Eric Eastwood 7a88ea0c19
Add support for room aliases (#107)
Also does friendly redirects if you don't exactly use the right URL pattern.
For example, if you paste the full room ID with the `!` like `/roomid/!foo:bar`,
it will properly redirect you to `/roomid/foo:bar`. It also does this sort of
thing for URL encoded room ID's and aliases.

Fix https://github.com/matrix-org/matrix-public-archive/issues/25
2022-10-27 01:09:13 -05:00
Eric Eastwood 1e89179f09
Page-load with the correct homeserver selected (#98)
Page-load with the correct homeserver selected (according to `?homeserver`).

Fix https://github.com/matrix-org/matrix-public-archive/issues/92

Also makes sure that the `?homeserver` is always available somewhere in the list; whether that be in the available homeserver list or the added homeserver list depending on it someone cleared it out or never had it because they visited from someone else's link.
2022-10-21 02:09:26 -05:00
Eric Eastwood b34c1b817d
Add homeserver selector to room directory landing page (#87)
Opting for the simple solution and using `include_all_networks` instead of needing to fetch the information about the third-party networks.

Fix https://github.com/matrix-org/matrix-public-archive/issues/6 (last piece done with this PR)
2022-10-20 02:06:43 -05:00
Eric Eastwood be837515fe
Show surrounding messages for a full screen of content (#71)
1. Add surrounding messages to the given messages so we have a full screen of content to make it feel lively even in quiet rooms
    - As you scroll around the timeline across different days, the date changes in the URL, calendar, etc
 2. Add summary item to the bottom of the timeline that explains if we couldn't find any messages in the specific day requested 
    - Also allows you to the jump to the next activity in the room. Adds `/:roomId/jump?ts=xxx&dir=[f|b]` to facilitate this.
    - Part of https://github.com/matrix-org/matrix-public-archive/issues/46
 3. Add developer options modal which is linked from the bottom of the right-panel
    - Adds an option so you can debug the `IntersectionObserver` and how it's selecting the active day from the top-edge of the scroll viewport.
    - In the future, this will also include a nice little visualization of the backend timing traces
2022-09-20 16:02:09 -05:00
Eric Eastwood 92668996d7
Add search to room directory landing page (#70)
Part of https://github.com/matrix-org/matrix-public-archive/issues/6
2022-09-15 20:41:55 -05:00
Eric Eastwood 32c77ecffe
Only show `world_readable` or `public` rooms in the archive. Only index `world_readable` (#66)
Only show `world_readable` or `public` rooms in the archive. Only allow `world_readable` rooms to be indexed by search engines.

Related to https://github.com/matrix-org/matrix-public-archive/issues/47
2022-09-08 19:15:07 -05:00
Eric Eastwood 65a371910a
Redirect to last day with message history (#65)
Redirect to last day with message history: `/:roomIdOrAlias` -> `/:roomIdOrAlias/date/:yyyy/:mm/:dd`

Fix https://github.com/matrix-org/matrix-public-archive/issues/60
2022-09-08 02:18:18 -05:00
Eric Eastwood 36925cd603
Add test to make sure the archive doesn't fail when event for event relation is missing and not included in list of provided events (#43)
Add test to make sure the archive doesn't fail when event for event relation is missing and not included in list of provided events. Like if someone is replying to an event that was from long ago out of our range.

In the case of missing relations, Hydrogen does `_loadContextEntryNotInTimeline` because it can't find the event locally which throws an `uncaughtException`. Before https://github.com/matrix-org/matrix-public-archive/pull/51, the `uncaughtException` killed the Hydrogen `child_process` before it could pass back the HTML. Now this PR mainly just adds a test to make sure it works.
```
TypeError: Cannot read properties of undefined (reading 'storeNames')
    at TimelineReader.readById (hydrogen-web\target\lib-build\hydrogen.cjs.js:12483:33)
    at Timeline._getEventFromStorage (hydrogen-web\target\lib-build\hydrogen.cjs.js:12762:46)
    at Timeline._loadContextEntryNotInTimeline (hydrogen-web\target\lib-build\hydrogen.cjs.js:12747:35)
    at Timeline._loadContextEntriesWhereNeeded (hydrogen-web\target\lib-build\hydrogen.cjs.js:12741:14)
    at Timeline.addEntries (hydrogen-web\target\lib-build\hydrogen.cjs.js:12699:10)
    at mountHydrogen (4-hydrogen-vm-render-script.js:204:12)
    at 4-hydrogen-vm-render-script.js:353:1
    at Script.runInContext (node:vm:139:12)
    at _renderHydrogenToStringUnsafe (matrix-public-archive\server\hydrogen-render\3-render-hydrogen-to-string-unsafe.js:102:41)
    at async process.<anonymous> (matrix-public-archive\server\hydrogen-render\2-render-hydrogen-to-string-fork-script.js:18:27)
```
2022-08-29 19:42:18 -05:00
Eric Eastwood e9d13db911
Add test for joining a new federated room (#31)
Add test for joining a new federated room and making sure the messages are available (homeserver should backfill).

Synapse changes: https://github.com/matrix-org/synapse/pull/13205, https://github.com/matrix-org/synapse/pull/13320
2022-08-29 18:56:31 -05:00
Eric Eastwood d508521171
Use stable test selectors from Hydrogen (`data-testid`)
Follow-up to https://github.com/matrix-org/matrix-public-archive/pull/29#discussion_r909492946

---

Updated Hydrogen to add the consistent `data-testid` attribute selectors,
https://github.com/vector-im/hydrogen-web/pull/773

```
npm install hydrogen-view-sdk@npm:@mlm/hydrogen-view-sdk@0.0.13-scratch
```
2022-07-05 06:23:47 -05:00
Eric Eastwood 4690349816
Remove unneeded `include_redundant_members` from `/messages` `filter` and test that member state is still visible (#29)
Follow-up to https://github.com/matrix-org/matrix-public-archive/pull/28#discussion_r909428366
2022-06-29 06:56:13 -05:00
Eric Eastwood 780600e3dd
Containerize app (make a Docker image for `matrix-public-archive`) (#22)
Containerize app (make a Docker image for `matrix-public-archive`)

```
ghcr.io/matrix-org/matrix-public-archive/matrix-public-archive
```

... with a variety of tags!


Packages viewable on https://github.com/matrix-org/matrix-public-archive/pkgs/container/matrix-public-archive%2Fmatrix-public-archive
2022-06-15 02:28:18 -05:00
Eric Eastwood cda2d98df0
Add testing to CI (#20)
1. Build test homeserver Docker images which can federate with each other
 2. Run end-to-end (e2e) tests

#### Dev notes

Sharing variables across jobs when the `services` field can't access the `env` context, https://github.community/t/how-to-use-env-with-container-image/17252/24
```yaml
env:
  FOO: bar

jobs:
  set_env:
    outputs:
      var: ${{ steps.save_var.outputs.var }}
    steps:
      - id: save_var
         run: echo "::set-output name=var::${{ env.FOO }}"
  actual_job:
    needs: set_env
    container:
      image: ...whatever_you_need_here...${{ needs.set_env.outputs.var }}
```
2022-06-15 00:59:41 -05:00
Eric Eastwood cead1796b8
Better explain how to get the project and tests to run at this point (#19) 2022-06-10 18:52:12 -05:00
Eric Eastwood 9fc71a3412
Remove `matrix-bot-sdk` usage in tests (#15)
Remove `matrix-bot-sdk` usage in tests because it didn't have timestamp massaging `?ts` and it's not really necessary to rely on since we can just call the API directly 🤷. `matrix-bot-sdk` is also very annoying having to build rust crypto packages.

We're now using direct `fetch` requests against the Matrix API and lightweight `client` object.

All 3 current tests pass 
2022-06-09 20:44:57 -05:00
Eric Eastwood c34ed9ea0e Fix last event in day not being shown 2022-02-24 13:06:19 -06:00
Eric Eastwood db6d3797d7 Working e2e test 2022-02-24 03:27:53 -06:00
Eric Eastwood 839e31a35e E2E test but still failing because fetching from start of day before test events happened 2022-02-23 21:25:05 -06:00
Eric Eastwood 40fa71df03 Clean up federation test 2022-02-22 20:55:42 -06:00
Eric Eastwood 0f493a241e Federated homeservers in Docker for e2e tests 2022-02-22 20:25:24 -06:00
Eric Eastwood aea382e4f8 WIP: start of tests 2022-02-22 16:06:29 -06:00