Commit Graph

300 Commits

Author SHA1 Message Date
Simon Sawicki 088add9567
[cleanup] Misc
Authored by: Grub4K
2023-09-24 02:35:23 +02:00
Simon Sawicki de015e9307
[core] Prevent RCE when using `--exec` with `%q` (CVE-2023-40581)
The shell escape function is now using `""` instead of `\"`. `utils.Popen` has been patched to properly quote commands.

Prior to this fix using `--exec` together with `%q` when on Windows could cause remote code to execute. See https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg for reference.

Authored by: Grub4K
2023-09-24 02:29:01 +02:00
bashonly 52414d64ca
[utils] `js_to_json`: Handle `Array` objects
Authored by: Grub4K, std-move

Co-authored-by: std-move <26625259+std-move@users.noreply.github.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
2023-09-21 16:51:57 -05:00
pukkandan 3f7965105d
[utils] HTTPHeaderDict: Handle byte values 2023-07-30 03:18:10 +05:30
coletdjnz 4bf912282a
[networking] Remove dot segments during URL normalization (#7662)
This implements RFC3986 5.2.4 remove_dot_segments during the URL normalization process.

Closes #3355, #6526

Authored by: coletdjnz
2023-07-28 22:40:20 +00:00
bashonly af86873218
[utils] Improve `parse_duration`
Authored by: bashonly
2023-07-20 08:40:31 -05:00
coletdjnz 227bf1a33b
[networking] Rewrite architecture (#2861)
New networking interface consists of a `RequestDirector` that directs
each `Request` to appropriate `RequestHandler` and returns the
`Response` or raises `RequestError`. The handlers define adapters to
transform its internal Request/Response/Errors to our interfaces.

User-facing changes:
- Fix issues with per request proxies on redirects for urllib
- Support for `ALL_PROXY` environment variable for proxy setting
- Support for `socks5h` proxy
   - Closes https://github.com/yt-dlp/yt-dlp/issues/6325, https://github.com/ytdl-org/youtube-dl/issues/22618, https://github.com/ytdl-org/youtube-dl/pull/28093
- Raise error when using `https` proxy instead of silently converting it to `http`

Authored by: coletdjnz
2023-07-15 16:18:35 +05:30
pukkandan c365dba843
[networking] Add module (#2861)
No actual changes - code is only moved around
2023-07-15 16:18:34 +05:30
Mahmoud Abdel-Fattah 2af4eeb772
[utils] `clean_podcast_url`: Handle more trackers (#7556)
Authored by: mabdelfattah, bashonly
Closes #7544
2023-07-11 06:30:38 +05:30
pukkandan 4823ec9f46
Update to ytdl-commit-d1c6c5
[YouTube] [core] Improve platform debug log, based on yt-dlp
d1c6c5c4d6

Except:
    * 6ed34338285f722d0da312ce0af3a15a077a3e2a [jsinterp] Add short-cut evaluation for common expression
        * There was no performance improvement when tested with https://github.com/ytdl-org/youtube-dl/issues/30641
    * e8de54bce50f6f77a4d7e8e80675f7003d5bf630 [core] Handle `/../` sequences in HTTP URLs
        * We plan to implement this differently
2023-05-24 23:30:43 +05:30
Simon Sawicki b079c26f0a
[utils] `traverse_obj`: More fixes (#6959)
- Fix result when branching with `traverse_string`
- Fix `slice` path on `dict`s
- Fix tests and docstrings from 21b5ec86c2
- Add `is_iterable_like` helper function

Authored by: Grub4K
2023-04-30 19:50:22 +02:00
Simon Sawicki 21b5ec86c2
[utils] `traverse_obj`: Allow iterables in traversal (#6902)
Authored by: Grub4K
2023-04-24 19:56:35 +02:00
Simon Sawicki 0898c5c8cc
[utils] `js_to_json`: Implement template strings (#6623)
Authored by: Grub4K
2023-03-25 19:41:28 +01:00
Simon Sawicki 6839ae1f6d
[utils] `traverse_obj`: Fix more bugs
and cleanup uses of `default=[]`

Continued from b1bde57bef
2023-02-10 19:36:55 +05:30
Simon Sawicki b1bde57bef
[utils] `traverse_obj`: Fix several behavioral problems
See #6180 for further info

Authored by: Grub4K
2023-02-08 04:11:08 +01:00
Simon Sawicki 776995bc10
[utils] `traverse_obj`: Various improvements
- Add `set` key for transformations/filters
- Add `re.Match` group names
- Fix behavior for `expected_type` with `dict` key
- Raise for filter function signature mismatch in debug

Authored by: Grub4K
2023-02-02 06:40:19 +01:00
pukkandan fbb7383306
Add `weba` to known extensions 2022-12-30 15:32:47 +05:30
ChillingPepper d5f043d127
[utils] js_to_json: Fix bug in f55523c (#5771)
Authored by: ChillingPepper, pukkandan
2022-12-30 12:08:38 +05:30
Simon Sawicki a71b812f53
[utils] `js_to_json`: Improve escape handling (#5217)
Authored by: Grub4K
2022-10-13 01:52:17 +05:30
Matthew 4c9a1a3ba5
[extractor/wordpress:mb.miniAudioPlayer] Add embed extractor (#5087)
Closes https://github.com/yt-dlp/yt-dlp/issues/4994

Authored by: coletdjnz
2022-10-09 05:55:26 +00:00
Simon Sawicki 7b0127e1e1
[utils] `traverse_obj`: Allow `re.Match` objects (#5174)
Authored by: Grub4K
2022-10-09 07:01:37 +05:30
Simon Sawicki f99bbfc983
[utils] `traverse_obj`: Always return list when branching (#5170)
Fixes #5162
Authored by: Grub4K
2022-10-09 06:57:32 +05:30
Simon Sawicki ab029d7e92
[utils] `traverse_obj`: Rewrite, document and add tests (#5024)
Authored by: Grub4K
2022-09-26 02:33:19 +05:30
Elyse 7657ec7ed6
[utils] `base_url`: URL paths can contain `&` (#4841)
Authored by: elyse0
Closes #4187
2022-09-04 08:39:45 +05:30
pukkandan 8f53dc44a0
[jsinterp] Handle new youtube signature functions
Closes #4635
2022-08-14 05:12:32 +05:30
pukkandan 97d9c79e92
Fix tests for 989a01c261 2022-08-05 00:07:05 +05:30
Lauren N. Liberda fc61aff41b
Determine merge container better (See desc) (#1482)
* Determine the container early. Closes #4069
* Use codecs instead of just file extensions
* Obey `--prefer-free-formats`
* Allow fallbacks in `--merge-output`

Authored by: pukkandan, selfisekai
2022-08-04 06:12:12 +05:30
nixxo 47304e07dc
[extractor/rai] Add raisudtirol extractor (#4524)
Closes #4206
Authored by: nixxo
2022-08-02 00:55:48 +05:30
pukkandan 88f60feb32
Fix a904a7f8c6 2022-07-15 21:45:05 +05:30
Lesmiscore a904a7f8c6
Allow users to specify encoding in each config files (#4357)
Authored by: Lesmiscore
2022-07-15 20:52:14 +09:00
pukkandan f5ea47488a
[cleanup] Minor fixes 2022-07-11 02:24:36 +05:30
pukkandan 54007a45f1
[cleanup] Consistent style for file heads 2022-06-25 00:08:58 +05:30
pukkandan ac66811112
[compat] Remove more functions
Removing any more will require changes to a large number of extractors
2022-06-25 00:08:55 +05:30
pukkandan 0f06bcd759
[cleanup] Minor fixes (See desc)
* [youtube] Fix `--youtube-skip-dash-manifest`
* [build] Use `$()` in `Makefile`. Closes #3684
* Fix bug in 385ffb467b
* Fix bug in 43d7f5a5d0
* [cleanup] Remove unnecessary `utf-8` from `str.encode`/`bytes.decode`
* [utils] LazyList: Expose unnecessarily "protected" attributes
and other minor cleanup
2022-05-09 17:59:26 +05:30
pukkandan 19a0394044
[cleanup] Misc cleanup and refactor (#2173) 2022-04-18 02:28:28 +05:30
pukkandan f82711587c
[cleanup] Sort imports
Using https://github.com/PyCQA/isort

    isort -m VERTICAL_HANGING_INDENT --py 36 -l 80 --rr -n --tc .
2022-04-12 05:32:52 +05:30
pukkandan 86e5f3ed2e
[cleanup] Upgrade syntax
Using https://github.com/asottile/pyupgrade

1. `__future__` imports and `coding: utf-8` were removed
2. Files were rewritten with `pyupgrade --py36-plus --keep-percent-format`
3. f-strings were cherry-picked from `pyupgrade --py36-plus`

Extractors are left untouched (except removing header) to avoid unnecessary merge conflicts
2022-04-12 05:32:51 +05:30
felix cfb0511d82
[cleanup] Remove unused code paths (#2173)
Notes:

* `_windows_write_string`: Fixed in 3.6
  * https://bugs.python.org/issue1602
  * PEP: https://www.python.org/dev/peps/pep-0528

* Windows UTF-8 fix: Fixed in 3.3
  * https://bugs.python.org/issue13216

* `__loader__`: is always present in 3.3+
  * https://bugs.python.org/issue14646

* `workaround_optparse_bug9161`: Fixed in 2.7
  * https://bugs.python.org/issue9161

Authored by: fstirlitz
2022-04-12 05:32:50 +05:30
pukkandan b506289fe2
[test] Add `test_locked_file` 2022-04-07 12:05:44 +05:30
coletdev 1c1b2f96ae
[youtube:tab] Fix duration extraction for shorts (#3171)
Related: https://github.com/TeamNewPipe/NewPipe/issues/8034
Authored-by: coletdjnz
2022-03-28 00:49:42 +00:00
pukkandan 5c3895fff1
[outtmpl] Limit changes during sanitization
Closes #2761
2022-03-27 11:18:35 +05:30
s0u1h eeb2a770f3
[utils] `format_decimal_suffix`: Fix for very large numbers (#3109)
Authored by: s0u1h
2022-03-18 14:03:09 -07:00
pukkandan 28469edd7d
Release 2022.02.03 2022-02-03 23:14:46 +05:30
pukkandan 8bd1c00bf3
[utils] Handle `ss:xxx` in `parse_duration`
Closes #2388
2022-01-19 18:57:29 +05:30
Zenon Mousmoulas 0254f16274
[utils] Improve `get_elements_text_and_html_by_attribute` regex (#2280)
Authored by: zmousm, pukkandan
2022-01-09 23:44:56 +05:30
Zenon Mousmoulas 6f32a0b5b7
[utils] Improve parsing for nested HTML elements (#2129)
and add functions to return the HTML of elements

Authored by: zmousm
2022-01-06 00:07:49 +05:30
pukkandan 06e57990f7
Allow multiple and nested configuration files 2022-01-04 00:26:58 +05:30
Pierre Mdawar f02d24d8d2
[utils] Fix `format_bytes` output for Bytes (#2132)
Authored by: pukkandan, mdawar
2021-12-28 03:42:19 +05:30
pukkandan 352d5da812
[utils] Improve `parse_count` 2021-12-25 04:07:19 +05:30
pukkandan 282f570918
[utils] Fix error when copying `LazyList` 2021-11-20 08:33:55 +05:30