Happens to address part of https://github.com/matrix-org/matrix-public-archive/issues/271
but made primarily as a follow-up to https://github.com/matrix-org/matrix-public-archive/pull/239
---
Only 42% rooms on the `matrix.org` room directory are `world_readable` which means we will get pages of rooms that are half-empty most of the time if we just naively fetch 9 rooms at a time.
Ideally, we would be able to just add a filter directly to `/publicRooms` in order to only grab the `world_readable` rooms and still get full pages but the filter option doesn't allow us to slice by `world_readable` history visibility.
Instead, we have to paginate until we get a full grid of 9 rooms, then make a final `/publicRooms` request to backtrack to the exact continuation point so next page won't skip any rooms in between.
---
We had empty spaces in the grid before because some rooms in the room directory are private which we filtered out before. But that was a much more rare experience since only 2% of rooms were private .
1. Build test homeserver Docker images which can federate with each other
2. Run end-to-end (e2e) tests
#### Dev notes
Sharing variables across jobs when the `services` field can't access the `env` context, https://github.community/t/how-to-use-env-with-container-image/17252/24
```yaml
env:
FOO: bar
jobs:
set_env:
outputs:
var: ${{ steps.save_var.outputs.var }}
steps:
- id: save_var
run: echo "::set-output name=var::${{ env.FOO }}"
actual_job:
needs: set_env
container:
image: ...whatever_you_need_here...${{ needs.set_env.outputs.var }}
```
Remove `matrix-bot-sdk` usage in tests because it didn't have timestamp massaging `?ts` and it's not really necessary to rely on since we can just call the API directly 🤷. `matrix-bot-sdk` is also very annoying having to build rust crypto packages.
We're now using direct `fetch` requests against the Matrix API and lightweight `client` object.
All 3 current tests pass ✅
Eric Eastwood