The idea is to remove as many dependencies as possible for
low-level ScriptletFilteringEngine in order to make it easier
to reuse the module outside uBO itself.
The high-level derived class takes care of caching and
injection of scriptlets into documents, which requires
more knowledge about the environment in which scriptlets
are to be used.
Also improve scriptlet cache usage to minimize overhead of
retrieving scriptlets.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2969
Changes:
Use browser.alarms to trigger selfie creation. Presence of a selfie
improve markedly time to readiness when uBO is unsuspended.
Mirror content of storage.local to (in-memory) storage.session for
faster load to readiness when uBO is ususpended.
Broadcast channels are more suited to uBO than DOM events to dispatch
notifications to different parts of uBO.
DOM events can only be dispatched to local context, broadcast channels
dispatch to all contexts (i.e. background process, workers, auxiliary
pages) -- this last behavior is better suited to uBO to communicate
internal changes to all potential listeners, not just those in the local
context.
Additionally, broadcasting to content scripts is now done through
tabs.sendMessage() instead of through potentially opened message
ports, this simplifies broadcasting to content scripts, and this
doesn't require to have long-lived message ports in content
scripts.
When the target world of a scriptlet is the ISOLATED one,
skip Blob-based injection in Firefox, as the current world
is always the ISOLATED one. This should make ISOLATED
world-based scriptlets more reliable (i.e. execute sooner)
in Firefox.
The `urltransform` option allows to redirect a non-blocked network
request to another URL. There are restrictions on its usage:
- require a trusted source -- thus uBO-maintained lists or user
filters
- the `urltransform` value must start with a `/`
If at least one of these conditions is not fulfilled, the filter
will be invalid and rejected.
The requirement to start with `/` is to enforce that only the path
part of a URL can be modified, thus ensuring the network request
is redirected to the same scheme and authority (as defined at
https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Syntax).
Usage example (redirect requests for CSS resources to a non-existing
resource, for demonstration purpose):
||iana.org^$css,urltransform=/notfound.css
Name of this option is inspired from DNR API:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/declarativeNetRequest/URLTransform
This commit required to bring the concept of "trusted source" to
the static network filtering engine.
As per discussion with uBO volunteers.
Volunteers offering support for uBO will be able to craft links with
specially formed URLs, which once clicked will cause uBO to automatically
force an update of specified filter lists.
The URL must be crafted as shown in the example below:
https://ublockorigin.github.io/uAssets/update-lists.html?listkeys=ublock-filters,easylist
Where the `listkeys` parameter is a comma-separated list of tokens
corresponding to filter lists. If a token does not match an enabled
filter list, it will be ignored.
The ability to update filter lists through a specially crafted link
is available only on uBO's own support sites:
- https://github.com/uBlockOrigin/
- https://reddit.com/r/uBlockOrigin/
- https://ublockorigin.github.io/
Additionally, a visual cue has been added in the "Filter lists" pane
to easily spot the filter lists which have been recently updated, where
"recently" is currently defined as less than an hour ago.
Additionally, finalize versioning scheme for uBOL. Since most updates
will be simply related to update rulesets, the version will from now
on reflects the date at which the extension package was created:
year.month.day.minutes
So for example:
2023.8.19.690
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2773
The `randomize` paramater introduced in https://github.com/gorhill/uBlock/commit/418087de9c
is now named `directive`, and beside the `true` value which is meant
to respond with a random 10-character string, it can now take the
following value:
war:[web_accessible_resource name]
In order to mock the XHR response with a web accessible resource. For
example:
piquark6046.github.io##+js(no-xhr-if, adsbygoogle.js, war:googlesyndication_adsbygoogle.js)
Will cause the XHR performed by the webpage to resolve to the content
of `/web_accessible_resources/googlesyndication_adsbygoogle.js`.
Should the resource not exist, the empty string will be returned.
Additionally:
Use `export UBO_VERSION=local` at the console to build MV3 extension using
current version of uBO code base. By default, the version is taken from
`./platform/mv3/ubo-version' and usually set to last stable release.
Since in uBOL filter lists from various sources are combined into
a single list, there must be a way to turn on/off trust level
inside the resulting combined filter list so as to be able to
validate the trust level of filters requiring trust.
This commit adds new parser directives understood only by MV3
compiler to turn on/off trust flag internally.
See `managed_storage.json` for available settings. Currently
only `noFiltering` setting is availale.
`noFiltering` is an array of strings, each being a domain for
which no filtering should occur.
Related discussion:
- https://github.com/uBlockOrigin/uBOL-issues/discussions/35
Avoid using updateContentScripts() as it suffers from an unexpected
behavior, causing injected content scripts to lose proper order
at injection time. The order in which content scripts are injected
is key for uBOL content scripts. Potential out of order injection
was causing cosmetic filtering to be broken.
Use actual storage API to persist data across service worker
wake-ups and browser launches. uBOL was trying to avoid using
storage API, at the cost of somewhat hacky code (using DNR API
to persist settings).
Make use of session storage if available, to speed up
initialization of waking up the service worker (which at this
point is necessary to properly implement cosmetic filtering).
Related issues:
- https://github.com/uBlockOrigin/uBOL-issues/issues/5#issuecomment-1575425913
- https://github.com/w3c/webextensions/issues/403
Currently, there is no other way to inject CSS user styles than to
wake up the service worker, so that it can inject the CSS styles
itself using the `scripting.insertCSS()` method.
If ever the MV3 API supports injecting CSS user styles directly
from a content script, uBOL will be back to be fully declarative.
At this point the service worker is very lightweight since the
filtering is completely declarative, so this is not too much of
an issue performance-wise except for the fact that waking up the
service worker for the sole purpose of injecting CSS user styles
and nothing else introduces a pointless overhead.
Hopefully the MV3 API will mature to address such inefficiency.
Specifically, avoid long list of hostnames for the `matches`
property[1] when registering the content scripts, as this was causing
whole browser freeze for long seconds in Chromium-based browsers
(reason unknown).
The content scripts themselves will sort out which cosmetic filters to
apply on which websites.
This change makes it now possible to support annoyances-related lists,
and thus two lists have been added:
- EasyList -- Annoyances
- EasyList -- Cookies
Related issue:
- https://github.com/uBlockOrigin/uBOL-issues/issues/5
These annoyances-related lists contains many thousands of specific
cosmetic filters and as a result, before the above change this was
causing long seconds of whole browser freeze when simply modifying
the blocking mode of a specific site via the slider in the popup
panel.
It is now virtually instantaneous, at the cost of injecting larger
cosmetic filtering-related content scripts (which typically should
be garbage-collected within single-digit milliseconds).
Also, added support for entity-based cosmetic filters. (They were
previously discarded).
---
[1] https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/scripting/RegisteredContentScript
Source code of scriplets is now fetched directly from uBO
project, so there is no longer the need to keep duplicate
versions of scriplet code.
All scriplet filters are now supported.
Some filters with entity-based domain option can be salvaged
when there are non-entity-based domain option, but since we are
throwing away the entity-based entries, we are only partially
converting to DNR. This commit will log a warning about this
in log.txt. Before this commit, only non-salvageable filters
were logged.
This reflects the _world_ of the MV3 scripting API:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/scripting/ExecutionWorld
MAIN: page's world
ISOLATED: extension's content script world
Some scriptlets are best executed in either world, so this
commit allows to pick in which world a scriptlet should execute
(default to MAIN).
For instance, the new sed.js scriptlet will now execute in
the ISOLATED world.
Also added additional safeguard against sticky unprocessed-request
status: all unprocessed-request statuses will be cleared after a
minute elapsed following intialization. The idea is that if the
user hasn't care to force a reload of the page, then it's assumed
to be by choice and uBO should stop informing about the status.
What does not work at the time of commit:
Cosmetic filtering does not work:
The content scripts responsible for cosmetic filtering fail when
trying to inject the stylesheets through document.adoptedStyleSheets,
with the following error message:
XrayWrapper denied access to property Symbol.iterator
(reason: object is not safely Xrayable).
See https://developer.mozilla.org/en-US/docs/Xray_vision for more
information. ... css-declarative.js:106:8
A possible solution is to inject those content scripts in the
MAIN world. However Firefox scripting API does not support MAIN
world injection at the moment.
Scriptlet-filtering does not work:
Because scriptlet code needs to be injected in the MAIN world,
and this is currently not supported by Firefox's scripting API,
see https://bugzilla.mozilla.org/show_bug.cgi?id=1736575
There is no count badge on the toolbar icon in Firefox, as it
currently does not support the `DNR.setExtensionActionOptions`
method.
Other than the above issues, it does appear uBO is blocking
properly with no error reported in the dev console.
The adoptedStyleSheets issue though is worrisome, as the
cosmetic filtering content scripts were designed with ISOLATED
world injection in mind. Being forced to inject in MAIN world
(when available) make things a bit more complicated as uBO
has to ensure it's global variables do not leak into the page.
uBO will now verify that at least one unprocessed network requests
at launch should have been blocked in order to warn users of
unprocessed network requests through the `!` toolbar icon badge.
For example, with default filter lists, there is nothing to block
on `wikipedia.org`, and hence in this case it's not useful to
present the user with the `!` badge.
Therefore uBO will not show the badge *only* when at least one
unprocessed network requests should have been blocked had uBO been
ready when it was fired by the browser.
Related commit:
- https://github.com/gorhill/uBlock/commit/769b8da664be
Related feedback:
- 769b8da664 (commitcomment-104695781)
The incomplete filtering status of a given tab at browser launch
will be carried over visually as a yellowish `!` badge until the
web page in the tab is force reloaded, navigated away, or closed.
The purpose is to make it obvious to end users that a web page
has not been filtered properly and to avoid issue reports
related to this.
It is expected that Firefox should never be affected by cases of
yellowish badge -- that is unless the setting "Suspend network
activity [...]" has been disabled, in which case the new behavior
will also be useful to those who disabled the setting.
Related discussion:
- https://github.com/uBlockOrigin/uAssets/discussions/16939
Until uBO's filtering engines are properly initialized, there will
be a distinct toolbar icon to help users understand that uBO may
not be fully initialized when a webpage is loaded -- often the
cause of improper filtering of trackers/ads at browser launch.
This commit is a rewrite of the static filtering parser into a
tree-based data structure, for easier maintenance and better
abstraction of parsed filters.
This simplifies greatly syntax coloring of filters and also
simplify extending filter syntax.
The minimum version of Chromium-based browsers has been raised
to version 73 because of usage of String.matchAll().
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1861
The "exceptor" feature has been rewritten, with the following
changes as a result:
- The excepted filters cease to exist when closing the logger
- It's now possible to temporary except network filters
When toggling on/off a temporary exception, filter lists are now
fully reloaded. This simplified managing temporary exceptions, and
made it easy to implement temporary exception for network filters,
but this also means there might be a perceptible delay when
adding/removing temporary exceptions. At this point I consider
this an acceptable side-effect just to bring the ability to easily
create temporary exception for network filters, while this
simplified the existing temporary exception code throughout.
Raymond Hill