Mirrors
/
uBlock
mirror of https://github.com/gorhill/uBlock.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
uBlock/tools/make-nodejs.sh

45 lines
1.5 KiB
Bash
Raw Permalink Normal View History

Revert "Pass -e flag to bash (#3790)" This reverts commit bb2015949516346edb137384882022f839a187d2.
2021-07-31 12:49:32 -06:00
#!/usr/bin/env bash
Modularize codebase with export/import Related issue: - https://github.com/uBlockOrigin/uBlock-issues/issues/1664 The changes are enough to fulfill the related issue. A new platform has been added in order to allow for building a NodeJS package. From the root of the project: ./tools/make-nodejs This will create new uBlock0.nodejs directory in the ./dist/build directory, which is a valid NodeJS package. From the root of the package, you can try: node test This will instantiate a static network filtering engine, populated by easylist and easyprivacy, which can be used to match network requests by filling the appropriate filtering context object. The test.js file contains code which is typical example of usage of the package. Limitations: the NodeJS package can't execute the WASM versions of the code since the WASM module requires the use of fetch(), which is not available in NodeJS. This is a first pass at modularizing the codebase, and while at it a number of opportunistic small rewrites have also been made. This commit requires the minimum supported version for Chromium and Firefox be raised to 61 and 60 respectively.
2021-07-25 08:55:35 -06:00
#
# This script assumes a linux environment
Use set -e in Bash scripts (#3791)
2021-07-31 15:34:25 -06:00
set -e
Add support for nodejs flavors The main nodejs flavor is "npm", which is to be used to lint/test and the publication of an official npm package -- and by design it has dependencies on mocha, eslint, etc. A new flavor "dig" has been created with minimal dependencies and which purpose is to easily allow to write specialized code to investigate local code changes in uBO -- and it's not meant for publication. Consequently, "make nodejs" has been replaced with "make npm", and a new "dig" target has been added to the makefile, to be used for instrumenting local code changes for investigation purpose.
2021-08-15 08:43:36 -06:00
DES=$1
Create build folder in npm install script (#3799)
2021-08-03 05:02:48 -06:00
Modularize codebase with export/import Related issue: - https://github.com/uBlockOrigin/uBlock-issues/issues/1664 The changes are enough to fulfill the related issue. A new platform has been added in order to allow for building a NodeJS package. From the root of the project: ./tools/make-nodejs This will create new uBlock0.nodejs directory in the ./dist/build directory, which is a valid NodeJS package. From the root of the package, you can try: node test This will instantiate a static network filtering engine, populated by easylist and easyprivacy, which can be used to match network requests by filling the appropriate filtering context object. The test.js file contains code which is typical example of usage of the package. Limitations: the NodeJS package can't execute the WASM versions of the code since the WASM module requires the use of fetch(), which is not available in NodeJS. This is a first pass at modularizing the codebase, and while at it a number of opportunistic small rewrites have also been made. This commit requires the minimum supported version for Chromium and Firefox be raised to 61 and 60 respectively.
2021-07-25 08:55:35 -06:00
mkdir -p $DES/js
cp src/js/base64-custom.js $DES/js
cp src/js/biditrie.js $DES/js
Export the rule-based filtering engines to the nodejs package The code exported to nodejs package was revised to use modern JavaScript syntax. A few issues were fixed at the same time. The exported classes are: - DynamicHostRuleFiltering - DynamicURLRuleFiltering - DynamicSwitchRuleFiltering These related to the content the of "My rules" pane in the uBlock Origin extension.
2021-08-03 10:19:25 -06:00
cp src/js/dynamic-net-filtering.js $DES/js
Modularize codebase with export/import Related issue: - https://github.com/uBlockOrigin/uBlock-issues/issues/1664 The changes are enough to fulfill the related issue. A new platform has been added in order to allow for building a NodeJS package. From the root of the project: ./tools/make-nodejs This will create new uBlock0.nodejs directory in the ./dist/build directory, which is a valid NodeJS package. From the root of the package, you can try: node test This will instantiate a static network filtering engine, populated by easylist and easyprivacy, which can be used to match network requests by filling the appropriate filtering context object. The test.js file contains code which is typical example of usage of the package. Limitations: the NodeJS package can't execute the WASM versions of the code since the WASM module requires the use of fetch(), which is not available in NodeJS. This is a first pass at modularizing the codebase, and while at it a number of opportunistic small rewrites have also been made. This commit requires the minimum supported version for Chromium and Firefox be raised to 61 and 60 respectively.
2021-07-25 08:55:35 -06:00
cp src/js/filtering-context.js $DES/js
Export the rule-based filtering engines to the nodejs package The code exported to nodejs package was revised to use modern JavaScript syntax. A few issues were fixed at the same time. The exported classes are: - DynamicHostRuleFiltering - DynamicURLRuleFiltering - DynamicSwitchRuleFiltering These related to the content the of "My rules" pane in the uBlock Origin extension.
2021-08-03 10:19:25 -06:00
cp src/js/hnswitches.js $DES/js
Modularize codebase with export/import Related issue: - https://github.com/uBlockOrigin/uBlock-issues/issues/1664 The changes are enough to fulfill the related issue. A new platform has been added in order to allow for building a NodeJS package. From the root of the project: ./tools/make-nodejs This will create new uBlock0.nodejs directory in the ./dist/build directory, which is a valid NodeJS package. From the root of the package, you can try: node test This will instantiate a static network filtering engine, populated by easylist and easyprivacy, which can be used to match network requests by filling the appropriate filtering context object. The test.js file contains code which is typical example of usage of the package. Limitations: the NodeJS package can't execute the WASM versions of the code since the WASM module requires the use of fetch(), which is not available in NodeJS. This is a first pass at modularizing the codebase, and while at it a number of opportunistic small rewrites have also been made. This commit requires the minimum supported version for Chromium and Firefox be raised to 61 and 60 respectively.
2021-07-25 08:55:35 -06:00
cp src/js/hntrie.js $DES/js
[mv3] Add support for redirect= filters This adds support for `redirect=` filters. As with `removeparam=` filters, `redirect=` filters can only be enforced when the default filtering mode is set to Optimal or Complete, since these filters require broad host permissions to be enforced by the DNR engine. `redirect-rule=` filters are not supported since there is no corresponding DNR syntax. Additionally, fixed the dropping of whole network filters even though those filters are still useful despite not being completely enforceable -- for example a filter with a single (unsupported) domain using entity syntax in its `domain=` option should not be wholly dropped when there are other valid domains in the list.
2022-10-16 10:05:24 -06:00
cp src/js/redirect-resources.js $DES/js
Add experimental mv3 version This create a separate Chromium extension, named "uBO Minus (MV3)". This experimental mv3 version supports only the blocking of network requests through the declarativeNetRequest API, so as to abide by the stated MV3 philosophy of not requiring broad "read/modify data" permission. Accordingly, the extension should not trigger the warning at installation time: Read and change all your data on all websites The consequences of being permission-less are the following: - No cosmetic filtering (##) - No scriptlet injection (##+js) - No redirect= filters - No csp= filters - No removeparam= filters At this point there is no popup panel or options pages. The default filterset correspond to the default filterset of uBO proper: Listset for 'default': https://ublockorigin.github.io/uAssets/filters/badware.txt https://ublockorigin.github.io/uAssets/filters/filters.txt https://ublockorigin.github.io/uAssets/filters/filters-2020.txt https://ublockorigin.github.io/uAssets/filters/filters-2021.txt https://ublockorigin.github.io/uAssets/filters/filters-2022.txt https://ublockorigin.github.io/uAssets/filters/privacy.txt https://ublockorigin.github.io/uAssets/filters/quick-fixes.txt https://ublockorigin.github.io/uAssets/filters/resource-abuse.txt https://ublockorigin.github.io/uAssets/filters/unbreak.txt https://easylist.to/easylist/easylist.txt https://easylist.to/easylist/easyprivacy.txt https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-online.txt https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=1&mimetype=plaintext The result of the conversion of the filters in all these filter lists is as follow: Ruleset size for 'default': 22245 Good: 21408 Maybe good (regexes): 127 redirect-rule= (discarded): 458 csp= (discarded): 85 removeparams= (discarded): 22 Unsupported: 145 The fact that the number of DNR rules are far lower than the number of network filters reported in uBO comes from the fact that lists-to-rulesets converter does its best to coallesce filters into minimal set of rules. Notably, the DNR's requestDomains condition property allows to create a single DNR rule out of all pure hostname-based filters. Regex-based rules are dynamically added at launch time since they must be validated as valid DNR regexes through isRegexSupported() API call. At this point I consider being permission-less the limiting factor: if broad "read/modify data" permission is to be used, than there is not much point for an MV3 version over MV2, just use the MV2 version if you want to benefit all the features which can't be implemented without broad "read/modify data" permission. To locally build the MV3 extension: make mv3 Then load the resulting extension directory in the browser using the "Load unpacked" button. From now on there will be a uBlock0.mv3.zip package available in each release.
2022-09-06 11:47:52 -06:00
cp src/js/static-dnr-filtering.js $DES/js
Modularize codebase with export/import Related issue: - https://github.com/uBlockOrigin/uBlock-issues/issues/1664 The changes are enough to fulfill the related issue. A new platform has been added in order to allow for building a NodeJS package. From the root of the project: ./tools/make-nodejs This will create new uBlock0.nodejs directory in the ./dist/build directory, which is a valid NodeJS package. From the root of the package, you can try: node test This will instantiate a static network filtering engine, populated by easylist and easyprivacy, which can be used to match network requests by filling the appropriate filtering context object. The test.js file contains code which is typical example of usage of the package. Limitations: the NodeJS package can't execute the WASM versions of the code since the WASM module requires the use of fetch(), which is not available in NodeJS. This is a first pass at modularizing the codebase, and while at it a number of opportunistic small rewrites have also been made. This commit requires the minimum supported version for Chromium and Firefox be raised to 61 and 60 respectively.
2021-07-25 08:55:35 -06:00
cp src/js/static-filtering-parser.js $DES/js
cp src/js/static-net-filtering.js $DES/js
cp src/js/static-filtering-io.js $DES/js
Add tasks.js module (#3839)
2021-08-19 05:19:20 -06:00
cp src/js/tasks.js $DES/js
Fix make-nodejs Broken due to renaming text-iterators to text-utils.
2021-07-31 06:59:55 -06:00
cp src/js/text-utils.js $DES/js
Modularize codebase with export/import Related issue: - https://github.com/uBlockOrigin/uBlock-issues/issues/1664 The changes are enough to fulfill the related issue. A new platform has been added in order to allow for building a NodeJS package. From the root of the project: ./tools/make-nodejs This will create new uBlock0.nodejs directory in the ./dist/build directory, which is a valid NodeJS package. From the root of the package, you can try: node test This will instantiate a static network filtering engine, populated by easylist and easyprivacy, which can be used to match network requests by filling the appropriate filtering context object. The test.js file contains code which is typical example of usage of the package. Limitations: the NodeJS package can't execute the WASM versions of the code since the WASM module requires the use of fetch(), which is not available in NodeJS. This is a first pass at modularizing the codebase, and while at it a number of opportunistic small rewrites have also been made. This commit requires the minimum supported version for Chromium and Firefox be raised to 61 and 60 respectively.
2021-07-25 08:55:35 -06:00
cp src/js/uri-utils.js $DES/js
Export the rule-based filtering engines to the nodejs package The code exported to nodejs package was revised to use modern JavaScript syntax. A few issues were fixed at the same time. The exported classes are: - DynamicHostRuleFiltering - DynamicURLRuleFiltering - DynamicSwitchRuleFiltering These related to the content the of "My rules" pane in the uBlock Origin extension.
2021-08-03 10:19:25 -06:00
cp src/js/url-net-filtering.js $DES/js
Modularize codebase with export/import Related issue: - https://github.com/uBlockOrigin/uBlock-issues/issues/1664 The changes are enough to fulfill the related issue. A new platform has been added in order to allow for building a NodeJS package. From the root of the project: ./tools/make-nodejs This will create new uBlock0.nodejs directory in the ./dist/build directory, which is a valid NodeJS package. From the root of the package, you can try: node test This will instantiate a static network filtering engine, populated by easylist and easyprivacy, which can be used to match network requests by filling the appropriate filtering context object. The test.js file contains code which is typical example of usage of the package. Limitations: the NodeJS package can't execute the WASM versions of the code since the WASM module requires the use of fetch(), which is not available in NodeJS. This is a first pass at modularizing the codebase, and while at it a number of opportunistic small rewrites have also been made. This commit requires the minimum supported version for Chromium and Firefox be raised to 61 and 60 respectively.
2021-07-25 08:55:35 -06:00
mkdir -p $DES/lib
New cosmetic filter parser using CSSTree library The new parser no longer uses the browser DOM to validate that a cosmetic filter is valid or not, this is now done through a JS library, CSSTree. This means filter list authors will have to be more careful to ensure that a cosmetic filter is really valid, as there is no more guarantee that a cosmetic filter which works for a given browser/version will still work properly on another browser, or different version of the same browser. This change has become necessary because of many reasons, one of them being the flakiness of the previous parser as exposed by many issues lately: - https://github.com/uBlockOrigin/uBlock-issues/issues/2262 - https://github.com/uBlockOrigin/uBlock-issues/issues/2228 The new parser introduces breaking changes, there was no way to do otherwise. Some current procedural cosmetic filters will be shown as invalid with this change. This occurs because the CSSTree library gets confused with some syntax which was previously allowed by the previous parser because it was more permissive. Mainly the issue is with the arguments passed to some procedural cosmetic filters, and these issues can be solved as follow: Use quotes around the argument. You can use either single or double-quotes, whichever is most convenient. If your argument contains a single quote, use double-quotes, and vice versa. Additionally, try to escape a quote inside an argument using backslash. THis may work, but if not, use quotes around the argument. When the parser encounter quotes around an argument, it will discard them before trying to process the argument, same with escaped quotes inside the argument. Examples: Breakage: ...##^script:has-text(toscr') Fix: ...##^script:has-text(toscr\') Breakage: ...##:xpath(//*[contains(text(),"VPN")]):upward(2) Fix: ...##:xpath('//*[contains(text(),"VPN")]'):upward(2) There are not many filters which break in the default set of filter lists, so this should be workable for default lists. Unfortunately those fixes will break the filter for previous versions of uBO since these to not deal with quoted argument. In such case, it may be necessary to keep the previous filter, which will be discarded as broken on newer version of uBO. THis was a necessary change as the old parser was becoming more and more flaky after being constantly patched for new cases arising, The new parser should be far more robust and stay robist through expanding procedural cosmetic filter syntax. Additionally, in the MV3 version, filters are pre-compiled using a Nodejs script, i.e. outside the browser, so validating cosmetic filters using a live DOM no longer made sense. This new parser will have to be tested throughly before stable release.
2022-09-23 14:03:13 -06:00
cp -R src/lib/csstree $DES/lib/
Modularize codebase with export/import Related issue: - https://github.com/uBlockOrigin/uBlock-issues/issues/1664 The changes are enough to fulfill the related issue. A new platform has been added in order to allow for building a NodeJS package. From the root of the project: ./tools/make-nodejs This will create new uBlock0.nodejs directory in the ./dist/build directory, which is a valid NodeJS package. From the root of the package, you can try: node test This will instantiate a static network filtering engine, populated by easylist and easyprivacy, which can be used to match network requests by filling the appropriate filtering context object. The test.js file contains code which is typical example of usage of the package. Limitations: the NodeJS package can't execute the WASM versions of the code since the WASM module requires the use of fetch(), which is not available in NodeJS. This is a first pass at modularizing the codebase, and while at it a number of opportunistic small rewrites have also been made. This commit requires the minimum supported version for Chromium and Firefox be raised to 61 and 60 respectively.
2021-07-25 08:55:35 -06:00
cp -R src/lib/punycode.js $DES/lib/
cp -R src/lib/regexanalyzer $DES/lib/
Simplify loading of WASM modules in nodejs
2021-07-31 05:06:11 -06:00
cp -R src/lib/publicsuffixlist $DES/lib/
# Convert wasm modules into json arrays
mkdir -p $DES/js/wasm
Add experimental mv3 version This create a separate Chromium extension, named "uBO Minus (MV3)". This experimental mv3 version supports only the blocking of network requests through the declarativeNetRequest API, so as to abide by the stated MV3 philosophy of not requiring broad "read/modify data" permission. Accordingly, the extension should not trigger the warning at installation time: Read and change all your data on all websites The consequences of being permission-less are the following: - No cosmetic filtering (##) - No scriptlet injection (##+js) - No redirect= filters - No csp= filters - No removeparam= filters At this point there is no popup panel or options pages. The default filterset correspond to the default filterset of uBO proper: Listset for 'default': https://ublockorigin.github.io/uAssets/filters/badware.txt https://ublockorigin.github.io/uAssets/filters/filters.txt https://ublockorigin.github.io/uAssets/filters/filters-2020.txt https://ublockorigin.github.io/uAssets/filters/filters-2021.txt https://ublockorigin.github.io/uAssets/filters/filters-2022.txt https://ublockorigin.github.io/uAssets/filters/privacy.txt https://ublockorigin.github.io/uAssets/filters/quick-fixes.txt https://ublockorigin.github.io/uAssets/filters/resource-abuse.txt https://ublockorigin.github.io/uAssets/filters/unbreak.txt https://easylist.to/easylist/easylist.txt https://easylist.to/easylist/easyprivacy.txt https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-online.txt https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=1&mimetype=plaintext The result of the conversion of the filters in all these filter lists is as follow: Ruleset size for 'default': 22245 Good: 21408 Maybe good (regexes): 127 redirect-rule= (discarded): 458 csp= (discarded): 85 removeparams= (discarded): 22 Unsupported: 145 The fact that the number of DNR rules are far lower than the number of network filters reported in uBO comes from the fact that lists-to-rulesets converter does its best to coallesce filters into minimal set of rules. Notably, the DNR's requestDomains condition property allows to create a single DNR rule out of all pure hostname-based filters. Regex-based rules are dynamically added at launch time since they must be validated as valid DNR regexes through isRegexSupported() API call. At this point I consider being permission-less the limiting factor: if broad "read/modify data" permission is to be used, than there is not much point for an MV3 version over MV2, just use the MV2 version if you want to benefit all the features which can't be implemented without broad "read/modify data" permission. To locally build the MV3 extension: make mv3 Then load the resulting extension directory in the browser using the "Load unpacked" button. From now on there will be a uBlock0.mv3.zip package available in each release.
2022-09-06 11:47:52 -06:00
cp src/js/wasm/* $DES/js/wasm/
Simplify loading of WASM modules in nodejs
2021-07-31 05:06:11 -06:00
node -pe "JSON.stringify(Array.from(fs.readFileSync('src/js/wasm/hntrie.wasm')))" \
> $DES/js/wasm/hntrie.wasm.json
node -pe "JSON.stringify(Array.from(fs.readFileSync('src/js/wasm/biditrie.wasm')))" \
> $DES/js/wasm/biditrie.wasm.json
node -pe "JSON.stringify(Array.from(fs.readFileSync('src/lib/publicsuffixlist/wasm/publicsuffixlist.wasm')))" \
> $DES/lib/publicsuffixlist/wasm/publicsuffixlist.wasm.json
Modularize codebase with export/import Related issue: - https://github.com/uBlockOrigin/uBlock-issues/issues/1664 The changes are enough to fulfill the related issue. A new platform has been added in order to allow for building a NodeJS package. From the root of the project: ./tools/make-nodejs This will create new uBlock0.nodejs directory in the ./dist/build directory, which is a valid NodeJS package. From the root of the package, you can try: node test This will instantiate a static network filtering engine, populated by easylist and easyprivacy, which can be used to match network requests by filling the appropriate filtering context object. The test.js file contains code which is typical example of usage of the package. Limitations: the NodeJS package can't execute the WASM versions of the code since the WASM module requires the use of fetch(), which is not available in NodeJS. This is a first pass at modularizing the codebase, and while at it a number of opportunistic small rewrites have also been made. This commit requires the minimum supported version for Chromium and Firefox be raised to 61 and 60 respectively.
2021-07-25 08:55:35 -06:00
...
2021-08-05 12:36:13 -06:00
cp platform/nodejs/*.js $DES/
cp platform/nodejs/README.md $DES/
cp LICENSE.txt $DES/