The main nodejs flavor is "npm", which is to be used to
lint/test and the publication of an official npm
package -- and by design it has dependencies on mocha,
eslint, etc.
A new flavor "dig" has been created with minimal
dependencies and which purpose is to easily allow to
write specialized code to investigate local code changes
in uBO -- and it's not meant for publication.
Consequently, "make nodejs" has been replaced with
"make npm", and a new "dig" target has been added to the
makefile, to be used for instrumenting local code changes
for investigation purpose.
The code exported to nodejs package was revised to use modern
JavaScript syntax. A few issues were fixed at the same time.
The exported classes are:
- DynamicHostRuleFiltering
- DynamicURLRuleFiltering
- DynamicSwitchRuleFiltering
These related to the content the of "My rules" pane in the
uBlock Origin extension.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
The changes are enough to fulfill the related issue.
A new platform has been added in order to allow for building
a NodeJS package. From the root of the project:
./tools/make-nodejs
This will create new uBlock0.nodejs directory in the
./dist/build directory, which is a valid NodeJS package.
From the root of the package, you can try:
node test
This will instantiate a static network filtering engine,
populated by easylist and easyprivacy, which can be used
to match network requests by filling the appropriate
filtering context object.
The test.js file contains code which is typical example
of usage of the package.
Limitations: the NodeJS package can't execute the WASM
versions of the code since the WASM module requires the
use of fetch(), which is not available in NodeJS.
This is a first pass at modularizing the codebase, and
while at it a number of opportunistic small rewrites
have also been made.
This commit requires the minimum supported version for
Chromium and Firefox be raised to 61 and 60 respectively.
Though Firefox shares a lot of WebExtensions code with Chromium,
these platforms have their own specific code paths, for various
reasons.
The reorganization here makes it clear that Chromium platform is
just one flavor of WebExtensions, and as such all Chromium-specific
code paths should no longer be automatically pulled by other
platforms where these code paths are not needed.
Given that the filepath of many files changed, here is the
parent commit to quickly browse back to the previous directory
layout:
ec7db30b2f
The pseudo user styles code served only browsers based
on Chromium 65 and earlier -- Chromium 66 supports
native user styles and was first released more than two
years ago.
In Chromium-based browsers, the pseudo user styles code
is being unconditionally injected in every page/frame
just in case the browser is version 65 or earlier.
Removing pseudo user styles reduce uBO's main content
script in Chromium-based browsers by more than 20K.
Related thread:
- https://github.com/NanoAdblocker/NanoCore/issues/348#issuecomment-653646507
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1116
- https://github.com/uBlockOrigin/uBlock-issues/issues/984
It has been found that the two malware lists uBO uses by
default appear to be essentially no longer maintained.
urlhaus list[1] has been identified by the community as
being well maintained and as being actually useful as a
default malware-related list in uBO, since it's being updated
every day from a database of top domains and specific URLs
identified as serving malicious content. Additionally, the
maintainer of urlhaus list has taken steps to increase
compatibility with uBO[2].
The decision has been to replace the current two malware-
related lists with urlhaus list, which will be enabled by
default in uBO -- and this means that list will be part of
uBO's package from now on.
For those who have the two removed malware lists enabled,
these will be moved to the custom lists section -- they
will still be enabled. It is suggested users remove them from
their selection of lists as their usefulness at this point
is questionable.
[1] https://gitlab.com/curben/urlhaus-filter
[2] 859dfd03c6
New advanced setting: `benchmarkDatasetURL`
Default value: `unset`
To specify a URL from where the benchmark dataset will be
fetched. This allows to launch benchmark operations from
within published versions of uBO, rather than from just
a locally built version.
With hindsight, I revised decisions made earlier during
this development cycle:
Un-redirectable scriptlets have been removed from
/web_accessible_resources and instead put in the new
/assets/resources/scriptlets.js, which contains all
scriptlets used for web page injection purpose.
uBO will no longer fetch a remote version of built-in
resources.
Advanced setting `userResourcesLocation` will still be
honoured by uBO, and if set, will be fetched every
time at least one asset is updated.
This is a first step, the ultimate goal is to remove
the need for resources.txt, or at least to reduce to
only hotfixes or for trivial resources targeting very
specific websites.
Most resources will become immutable, i.e. they will
be part of uBO's code base. Advantages include easier
code maintenance (jshint, syntax highlight), and to
make scriptlets more easy to code review by external
parties (for example extension store reviewers).
TODO:
- More scriptlets need to be imported before next
release.
- Need to make legacy versions of uBO use a legacy
version of resources.txt, as all the now obsolete
scriptlets will have to be removed once uBO's
next release become widespread.
- Possibly need to add code to load binary
resources so that they can be injected as
data: URI. So far it's unclear whether this is
really needed. For example, this would be needed
if a xmlhttprequest is redirected to an image
resource.
This works only for platforms supporting the return of
Promise by network listeners, i.e. only Firefox at this
point.
When filter lists are reloaded[1], there is a small
time window in which some network requests which should
have normally been blocked are not being blocked
because the static network filtering engine may not
have yet loaded all the filters in memory
This is now addressed by suspending the network request
handler when filter lists are reloaded -- again, this
works only on supported platforms.
[1] Examples: when a filter list update session
completes; when user filters change, when
adding/removing filter lists.
As seen at:
https://whotracks.me/blog/adblockers_performance_study.html
The requests.json.gz file can be downloaded from:
https://cdn.cliqz.com/adblocking/requests_top500.json.gz
Copy the file into ./tmp/requests.json.gz
If the file is present when you build uBO using `make-[target].sh` from
the shell, the resulting package will contain `./assets/requests.json`,
which will be looked-up by the method below to launch a benchmark
session.
From uBO's dev console, launch the benchmark:
µBlock.staticNetFilteringEngine.benchmark();
The usual browser dev tools can be used to obtain useful profiling
data, i.e. start the profiler, call the benchmark method from the
console, then stop the profiler when it completes.
Keep in mind that the measurements at the blog post above where obtained
with ONLY EasyList. The CPU reportedly used was:
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-6600U+%40+2.60GHz&id=2608
Rename ./tmp/requests.json.gz to something else if you no longer want
./assets/requests.json in the build.
The Promise chain was not properly designed for WASM module
loading. This became apparent when removing WASM modules
from Opera build[1].
The problem was that errors thrown by fetch() -- used to
load WASM modules -- were not properly handled.
[1] Opera refuses updating uBO if there are unrecognized file
types in the package, and `.wasm`/`.wat` files are not
recognized by Opera uploader.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/407
Both flavors will be stitched together into a single
`vapi-qebrequest.js` file.
The decision of which flavor to use will be made at runtime,
according to the browser environment.
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/194
This is a first iteration, which purpose is to merely allow
uBO to load properly. Many things are known to not work,
quite probably due to the early Thunderbird support of the
WebExtensions framework.
Permission which had to be removed:
- contextMenus
Manifest entries which had to be removed:
- commands
- sidebar_action
uBO's webRequest listeners are not being called when loading a
feed item in the preview pane, *except* for resources fetched
from embedded iframes.
uBO appears to function properly when a feed item is opened in
its own tab.
When using paths, platform implementations of setIcon typically
will fetch the resource then convert to image data internally.
It is preferable for uBO to do this conversion itself as it can
be done only once at launch time.
With chromium-based browsers, using image data eliminate the
incessant network traffic to fetch browser icons as reported
in the extension's dev tool, meaning a good chunk of overhead
is eliminated.
Also, use optimal icon sizes, as of now both chromium and firefox
prefers 16px instead of 19px, and 32px instead of 38px.
Raymond Hill