Commit Graph

38 Commits

Author SHA1 Message Date
Raymond Hill daa62c3081
Update README for npm package 2024-10-07 08:56:34 -04:00
Raymond Hill 818cb2d801
Fix npm test suite
Related commits:
- https://github.com/gorhill/uBlock/commit/02cba63331
- https://github.com/gorhill/uBlock/commit/41693407b2
2024-10-06 11:15:08 -04:00
Raymond Hill 985ea24e82
[mv3] Add support for redirect= filters
This adds support for `redirect=` filters. As with `removeparam=`
filters, `redirect=` filters can only be enforced when the
default filtering mode is set to Optimal or Complete, since these
filters require broad host permissions to be enforced by the DNR
engine.

`redirect-rule=` filters are not supported since there is no
corresponding DNR syntax.

Additionally, fixed the dropping of whole network filters even though
those filters are still useful despite not being completely
enforceable -- for example a filter with a single (unsupported) domain
using entity syntax in its `domain=` option should not be wholly
dropped when there are other valid domains in the list.
2022-10-16 12:05:24 -04:00
Raymond Hill a71b71e4c8
New cosmetic filter parser using CSSTree library
The new parser no longer uses the browser DOM to validate
that a cosmetic filter is valid or not, this is now done
through a JS library, CSSTree.

This means filter list authors will have to be more careful
to ensure that a cosmetic filter is really valid, as there is
no more guarantee that a cosmetic filter which works for a
given browser/version will still work properly on another
browser, or different version of the same browser.

This change has become necessary because of many reasons,
one of them being the flakiness of the previous parser as
exposed by many issues lately:

- https://github.com/uBlockOrigin/uBlock-issues/issues/2262
- https://github.com/uBlockOrigin/uBlock-issues/issues/2228

The new parser introduces breaking changes, there was no way
to do otherwise. Some current procedural cosmetic filters will
be shown as invalid with this change. This occurs because the
CSSTree library gets confused with some syntax which was
previously allowed by the previous parser because it was more
permissive.

Mainly the issue is with the arguments passed to some procedural
cosmetic filters, and these issues can be solved as follow:

Use quotes around the argument. You can use either single or
double-quotes, whichever is most convenient. If your argument
contains a single quote, use double-quotes, and vice versa.

Additionally, try to escape a quote inside an argument using
backslash. THis may work, but if not, use quotes around the
argument.

When the parser encounter quotes around an argument, it will
discard them before trying to process the argument, same with
escaped quotes inside the argument. Examples:

Breakage:

    ...##^script:has-text(toscr')

Fix:

    ...##^script:has-text(toscr\')

Breakage:

    ...##:xpath(//*[contains(text(),"VPN")]):upward(2)

Fix:

    ...##:xpath('//*[contains(text(),"VPN")]'):upward(2)

There are not many filters which break in the default set of
filter lists, so this should be workable for default lists.

Unfortunately those fixes will break the filter for previous
versions of uBO since these to not deal with quoted argument.
In such case, it may be necessary to keep the previous filter,
which will be discarded as broken on newer version of uBO.

THis was a necessary change as the old parser was becoming
more and more flaky after being constantly patched for new
cases arising, The new parser should be far more robust and
stay robist through expanding procedural cosmetic filter
syntax.

Additionally, in the MV3 version, filters are pre-compiled
using a Nodejs script, i.e. outside the browser, so validating
cosmetic filters using a live DOM no longer made sense.

This new parser will have to be tested throughly before stable
release.
2022-09-23 16:03:13 -04:00
Raymond Hill a559f5f271
Add experimental mv3 version
This create a separate Chromium extension, named
"uBO Minus (MV3)".

This experimental mv3 version supports only the blocking of
network requests through the declarativeNetRequest API, so as
to abide by the stated MV3 philosophy of not requiring broad
"read/modify data" permission. Accordingly, the extension
should not trigger the warning at installation time:

    Read and change all your data on all websites

The consequences of being permission-less are the following:

- No cosmetic filtering (##)
- No scriptlet injection (##+js)
- No redirect= filters
- No csp= filters
- No removeparam= filters

At this point there is no popup panel or options pages.

The default filterset correspond to the default filterset of
uBO proper:

Listset for 'default':
  https://ublockorigin.github.io/uAssets/filters/badware.txt
  https://ublockorigin.github.io/uAssets/filters/filters.txt
  https://ublockorigin.github.io/uAssets/filters/filters-2020.txt
  https://ublockorigin.github.io/uAssets/filters/filters-2021.txt
  https://ublockorigin.github.io/uAssets/filters/filters-2022.txt
  https://ublockorigin.github.io/uAssets/filters/privacy.txt
  https://ublockorigin.github.io/uAssets/filters/quick-fixes.txt
  https://ublockorigin.github.io/uAssets/filters/resource-abuse.txt
  https://ublockorigin.github.io/uAssets/filters/unbreak.txt
  https://easylist.to/easylist/easylist.txt
  https://easylist.to/easylist/easyprivacy.txt
  https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-online.txt
  https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=1&mimetype=plaintext

The result of the conversion of the filters in all these
filter lists is as follow:

Ruleset size for 'default': 22245
  Good: 21408
  Maybe good (regexes): 127
  redirect-rule= (discarded): 458
  csp= (discarded): 85
  removeparams= (discarded): 22
  Unsupported: 145

The fact that the number of DNR rules are far lower than the
number of network filters reported in uBO comes from the fact
that lists-to-rulesets converter does its best to coallesce
filters into minimal set of rules. Notably, the DNR's
requestDomains condition property allows to create a single
DNR rule out of all pure hostname-based filters.

Regex-based rules are dynamically added at launch time since
they must be validated as valid DNR regexes through
isRegexSupported() API call.

At this point I consider being permission-less the limiting
factor: if broad "read/modify data" permission is to be used,
than there is not much point for an MV3 version over MV2, just
use the MV2 version if you want to benefit all the features
which can't be implemented without broad "read/modify data"
permission.

To locally build the MV3 extension:

    make mv3

Then load the resulting extension directory in the browser
using the "Load unpacked" button.

From now on there will be a uBlock0.mv3.zip package available
in each release.
2022-09-06 13:47:52 -04:00
Manish Jethani d959c7aabe
Remove globals.js (#3849) 2021-08-23 10:54:16 -04:00
Manish Jethani b19393d8dc
Add tasks.js module (#3839) 2021-08-19 07:19:20 -04:00
Raymond Hill 087da6407d
Add support for nodejs flavors
The main nodejs flavor is "npm", which is to be used to
lint/test and the publication of an official npm
package -- and by design it has dependencies on mocha,
eslint, etc.

A new flavor "dig" has been created with minimal
dependencies and which purpose is to easily allow to
write specialized code to investigate local code changes
in uBO -- and it's not meant for publication.

Consequently, "make nodejs" has been replaced with
"make npm", and a new "dig" target has been added to the
makefile, to be used for instrumenting local code changes
for investigation purpose.
2021-08-15 10:43:36 -04:00
Manish Jethani bd8cb8d50f
Use local tmp directory (#3821) 2021-08-14 12:55:20 -04:00
Manish Jethani bafe824f09
Fix $TMPDIR check in make-nodejs.sh (#3820) 2021-08-14 12:31:17 -04:00
Manish Jethani 6c2856cb7e
Add package-lock.json (#3819) 2021-08-14 11:59:14 -04:00
Raymond Hill 6c4472ce7f
Save/restore node_modules dependencies if present
To avoid hacing to fetch them all over again each time we
launch the test suite.
2021-08-14 11:27:07 -04:00
Manish Jethani 806fe5dbe1
Add Mocha tests (#3815) 2021-08-13 13:08:13 -04:00
Manish Jethani a56f43f007
Add .npmrc to stop generating package lock file (#3814) 2021-08-13 11:49:52 -04:00
Manish Jethani daff6b3a34
Add eslint dependency within Node.js package (#3813) 2021-08-13 06:43:20 -04:00
Raymond Hill e112f6e3bd
Publish tgz package instead of zip
Related feedback:
- https://github.com/gorhill/uBlock/pull/3812#issuecomment-897057648
2021-08-12 08:33:43 -04:00
Manish Jethani 089c28861c
Use npm pack (#3812) 2021-08-12 07:57:56 -04:00
Manish Jethani ab9f5ae100
Run build script via tools/make-nodejs.sh (#3808) 2021-08-09 20:02:58 -04:00
Raymond Hill 5eb99a4dd6
... 2021-08-05 14:36:13 -04:00
Raymond Hill fe7ce8f576
Need to bump version to publish README 2021-08-05 14:32:07 -04:00
Manish Jethani 4ea0d134ad
Move linting out of tools/make-nodejs.sh (#3803) 2021-08-04 14:40:20 -04:00
Raymond Hill 314d6082f2
Fix broken GitHub actions
Related discussion:
- ad69c760fb (commitcomment-54363709)
2021-08-03 12:47:26 -04:00
Raymond Hill 89c5653bc6
Export the rule-based filtering engines to the nodejs package
The code exported to nodejs package was revised to use modern
JavaScript syntax. A few issues were fixed at the same time.

The exported classes are:
- DynamicHostRuleFiltering
- DynamicURLRuleFiltering
- DynamicSwitchRuleFiltering

These related to the content the of "My rules" pane in the
uBlock Origin extension.
2021-08-03 12:19:25 -04:00
Manish Jethani 6b85b7f2b1
Create build folder in npm install script (#3799) 2021-08-03 07:02:48 -04:00
Manish Jethani ad69c760fb
Run ESLint during Node.js package generation (#3798) 2021-08-02 16:55:03 -04:00
Manish Jethani 0b764934e0
Serialize PSL in Node.js install script (#3793) 2021-08-01 16:42:05 -04:00
Raymond Hill c6d275674d
Revert "Make the nodejs package load a serialized version of the PSL"
This reverts commit 46c6ff8708.
2021-08-01 15:42:46 -04:00
Raymond Hill 46c6ff8708
Make the nodejs package load a serialized version of the PSL
Related discussion:
- https://github.com/cliqz-oss/adblocker/pull/2091#issuecomment-890545926
2021-08-01 12:58:21 -04:00
Manish Jethani be68a6b5b4
Use set -e in Bash scripts (#3791) 2021-07-31 17:34:25 -04:00
Raymond Hill 6fa8ae619d
Revert "Pass -e flag to bash (#3790)"
This reverts commit bb20159495.
2021-07-31 14:49:32 -04:00
Manish Jethani bb20159495
Pass -e flag to bash (#3790) 2021-07-31 12:15:32 -04:00
Raymond Hill de1feb0948
Fix make-nodejs
Broken due to renaming text-iterators to text-utils.
2021-07-31 08:59:55 -04:00
Raymond Hill d5da88ea87
Simplify loading of WASM modules in nodejs 2021-07-31 07:06:11 -04:00
Raymond Hill 98fc66bb1b
Add support for enabling WASM code paths in NodeJS package
See `test.js` for reference on how to enable WASM code
paths (which are disabled by default).
2021-07-29 16:54:51 -04:00
Manish Jethani 63591ef2aa Make uAssets a submodule 2021-07-29 19:36:49 +05:30
Raymond Hill 09db8803c5
Remove requirement of uAssets presence to build a package
Related discussion:
- https://github.com/cliqz-oss/adblocker/pull/2075#discussion_r678654293
2021-07-28 19:40:11 -04:00
Raymond Hill e1222d1643
Use `require` instead of `fs`
Related feedback:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664#issuecomment-888332409
2021-07-28 10:38:24 -04:00
Raymond Hill 22022f636f
Modularize codebase with export/import
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664

The changes are enough to fulfill the related issue.

A new platform has been added in order to allow for building
a NodeJS package. From the root of the project:

    ./tools/make-nodejs

This will create new uBlock0.nodejs directory in the
./dist/build directory, which is a valid NodeJS package.

From the root of the package, you can try:

    node test

This will instantiate a static network filtering engine,
populated by easylist and easyprivacy, which can be used
to match network requests by filling the appropriate
filtering context object.

The test.js file contains code which is typical example
of usage of the package.

Limitations: the NodeJS package can't execute the WASM
versions of the code since the WASM module requires the
use of fetch(), which is not available in NodeJS.

This is a first pass at modularizing the codebase, and
while at it a number of opportunistic small rewrites
have also been made.

This commit requires the minimum supported version for
Chromium and Firefox be raised to 61 and 60 respectively.
2021-07-27 17:26:04 -04:00