WGet & Wget2

Table of Content GNU Wget2 - Introduction Features Links Build Requirements Downloading and building from tarball Building from git Download project and prepare sources with Build Wget2 with In Haiku build Wget2 with Test the functionality Install Wget2 and libwget Build static wget2.exe on GNU/Linux for Windows License GNU Wget2 - Introduction GNU Wget2 is the successor of GNU Wget, a file and recursive website downloader. Designed and written from scratch it wraps around libwget, that provides the basic functions needed by a web client. Wget2 works multi-threaded and uses many features to allow fast operation. In many cases Wget2 downloads much faster than Wget1.x due to HTTP2, HTTP compression, parallel connections and use of If-Modified-Since HTTP header. GNU Wget2 is licensed under GPLv3+. Libwget is licensed under LGPLv3+. Features A non-exhaustive list of features Support for HTTP/1.1 and HTTP/2.0 protocol brotli decompression support (Accept-Encoding: br) zstandard decompression support, RFC8478 (Accept-Encoding: zstd) lzip decompression support, (Accept-Encoding: lzip) HPKP - HTTP Public Key Pinning (RFC7469) with persistent database TCP Fast Open for plain text and for HTTPS TLS Session Resumption including persistent session data cache TLS False Start (with GnuTLS >= 3.5.0) HTTP2 support via nghttp2 and GnuTLS ALPN including streaming/pipelining OCSP stapling + OCSP server querying as a fallback (experimental, needs GnuTLS >= 3.3.11) Use libpsl for cookie domain checking (using Public Suffix List) Support link conversion (-k/--convert-links and -K/--backup-converted) Support for RFC 6266 compliant Content-Disposition RFC 6797 HSTS (HTTP Strict Transport Security) Support for bzip2 Content-Encoding / Accept-Encoding compression type New Year 2014 gimmick: added support for XZ Content-Encoding / Accept-Encoding compression type Character encoding of input files may be specified despite from local and remote encoding (--input-encoding) Support scanning RSS 2.0 feeds from local files (--force-rss -i <filename>) Support scanning RSS 2.0 feeds. Support scanning Atom 1.0 feeds from local files (--force-atom -i <filename>) Support scanning Atom 1.0 feeds. Support scanning URLs from local Sitemap XML file (--force-sitemap -i <filename>) Support scanning sitemap files given in robots.txt (Sitemap XML, gzipped Sitemap XML, plain text) including sitemap index files. Support arbitrary number of proxies for parallel downloads Multithreaded download of single files (option --chunk-size) Internationalized Domain Names in Applications (compile-selectable IDNA2008 or IDNA2003) ICEcast / SHOUTcast support via library (see examples/getstream.c) respect /robots.txt "Robot Exclusion Standard" and <META name="robots" ...> new option --secure-protocol=PFS to have TLS only plus forcing Perfect Forward Secrecy (PFS) IDN support for international domains autotools support proxy support cookies (session/non-session), detection of supercookies via Mozilla Public Suffix List (use the new option --cookie-suffixes <filename>, better: put it into ~/.wgetrc) recursive download of websites with or without spanning hosts download of single web pages / resources zlib/gzip compressed HTTP/HTTPS downloads (gzip, deflate) number of parallel download threads is adjustable include directive for config files (wildcards allowed) support for keep-alive connections included CSS, HTML, XML parser needed for recursive downloads gettext support HTTPS via libgnutls (and basic WolfSSL support) support for Metalink RFC 6249 (Metalink/HTTP: Mirrors and Hashes) support for Metalink RFC 5854 (Metalink Download Description Format / .meta4 files) support for Metalink 3 Metalink checksumming via libgnutls DNS lookup cache IPv4 and IPv6 support built and tested on Linux, OSX, OpenBSD, FreeBSD, Solaris, Windows Links Online Docs Mailing List Bug Tracker Development Code Coverage Fuzz Code Coverage Build Requirements The following packages are needed to build the software autotools (autoconf, autogen, automake, autopoint, libtool) python (recommended for faster bootstrap) rsync tar makeinfo (part of texinfo) pkg-config >= 0.28 (recommended) doxygen (for creating the documentation) pandoc (for creating the wget2 man page) gettext >= 0.18.2 libiconv (needed for IRI and IDN support) libz >= 1.2.3 (the distribution may call the package zlib*, eg. zlib1g on Debian) liblzma >= 5.1.1alpha (optional, if you want HTTP lzma decompression) libbz2 >= 1.0.6 (optional, if you want HTTP bzip2 decompression) libbrotlidec/libbrotli >= 1.0.0 (optional, if you want HTTP brotli decompression) libzstd >= 1.3.0 (optional, if you want HTTP zstd decompression) libgnutls (3.3, 3.5 or 3.6) libidn2 >= 0.14 (libidn >= 1.25 if you don't have libidn2) flex >= 2.5.35 libpsl >= 0.5.0 libnghttp2 >= 1.3.0 (optional, if you want HTTP/2 support) libmicrohttpd >= 0.9.51 (optional, if you want to run the test suite) lzip (optional, if you want to build distribution tarballs) lcov (optional, for coverage reports) libgpgme >= 0.4.2 (optional, for automatic signature verification) libpcre | libpcre2 (optional, for filtering by PCRE|PCRE2 regex) libhsts (optional, to support HSTS preload lists) libwolfssl (optional, to support WolfSSL instead of GnuTLS) The versions are recommended, but older versions may also work. Downloading and building from tarball wget https://gnuwget.gitlab.io/wget2/wget2-latest.tar.gz tar xf wget2-latest.tar.gz cd wget2-* ./configure make make check sudo make install Building from git Download project and prepare sources with git clone https://gitlab.com/gnuwget/wget2.git cd wget2 ./bootstrap # on shell failure try 'bash ./bootstrap' Build Wget2 with ./configure make In Haiku build Wget2 with setarch x86 ./configure --prefix=/boot/home/config/non-packaged rm /boot/home/config/non-packaged/wget2 && mv /boot/home/config/non-packaged/wget2_noinstall /boot/home/config/non-packaged/wget2 Test the functionality make check Install Wget2 and libwget sudo make install (or su -c "make install") Build static wget2.exe on GNU/Linux for Windows (cd contrib; docker build -t wget2/static -f Dockerfile.win32.static .) docker run --rm -v $PWD:/tmp wget2/static cp /usr/local/wget2/src/wget2.exe /tmp # now you have `wget2.exe` with debug symbols in your current directory # optional: remove debug symbols with strip wget2.exe # optional: pack executable upx wget2.exe License Copyright (C) 2015-2024 Free Software Foundation, Inc. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. A copy of the GNU General Public License can be found in the file LICENSE in the top directory of the official source distribution. The license is also available in several formats through the World Wide Web, or via http://www.gnu.org/licenses/licenses.html#GPL . ================================================== GNU Wget2 NEWS -- history of user-visible changes. Copyright (C) 2012-2015 Tim Ruehsen Copyright (C) 2015-2024 Free Software Foundation, Inc. See the end for copying conditions. Please send GNU Wget2 bug reports to <bug-wget@gnu.org>. 24.11.2024 Release v2.2.0 * Don't truncate file when -c and -O are combined * Don't log URI userinfo to logs * Fix downloading multiple files via HTTP/2 * Fix redirections with --no-parent * Support connecting with HTTP/1.0 proxies * Ignore 1xx HTTP responses for HTTP/1.1 * Fix status 8 for failed redirection of robots.txt * Fix ignoring connect timeout (regression) * Disable TCP Fast Open by default * Fix --dns-cache-preload for IPv6 * Accept --progress=dot:... for backwards compatibility * Disable explicit OCSP requests by default * Fix segfault when OCSP response is missing * Fix OCSP verification of first intermediate certificate * Allow option --no-tcp-fastopen to work on Linux kernels >= 4.11 * Limit cases where methods are redirected to GET * Fix possible deadlock when combining --no-clobber and --no-parent * Fix xattr reading of user.mime_type * Fix --no-parent for denormalized paths * Fix robots.txt parser * Several improvements to the WolfSSL code * Fix IPv6 address representation * Fix --restrict-file-names to be backwards compatible with wget 1.x * Add fetchmail compatibility for user/password in .netrc * Improve suport for non-standard cookie timestamps * Add libproxy support * Add instruction on how to cross-build wget2.exe via docker * Don't request preferred mime type for single file downloads * Slightly improved compatibility with LibreSSL 31.08.2023 Release v2.1.0 * New option --follow-sitemaps * New option --dane (cert validation via DNS) * Implement --check-certificate=quiet * Support proxies on non-default ports * Added CIDR support for no_proxy (IPv4 and IPv6) * Improve recursive RSS/Atom processing * Improve default cert/bundle paths for Windows * Improve Windows and MSVC compatibility * Use CONNECT for https_proxy * Add decoding numeric XML entities * Improve OpenSSL code * Improve WolfSSL code * Improve the progress bar * New function wget_xml_decode_entities_inline() * Support compilation of wget.h from C++ * Handle comments in robots.txt correctly * Fix parsing HTML/XML entities in URLs from HTML/XML * Fix use-after-free when updating blacklist entries * Don't try setting file timestamps on ttys * Fix arguments parsing for --filter-urls * Fix removing fragments when converting links * Fix duplicate downloads for Link headers with rel=duplicate * Fix segmentation fault (NULL dereference when no HTTP header has been received) * Change arguments of wget_iri_compare to const * Fix memory leak in wget_hashmap_clear() * Extend network error messages with hostname and IP address * Fix status code for 5xx errors * Fix issue in wget_buffer_trim() * Improve tests, documentation, building 27.05.2022 Release v2.0.1 * Fix escaping space in query part * Set EXIT_STATUS_NETWORK on error for the last try * Fix -k/--convert-links fragment * Fix escapng URLs with -k/--convert-links * Fix false reporting of a PSL error * Fix --directory-prefix with --content-disposition * Allow spaces and \ escaping in passwords in .netrc * Fix download abortion on some versions of Windows * Fix --unlink behavior * Fix deflate decompression when server omits the header * WolfSSL: Fix buffer overflow in SHA512 hashing * WolfSSL: Fix memory leak * Add support for unquoted HTML attribute values * OpenSSL: Fix several OCSP issues * Use keep-alive for HTTP/1.1 and higher as default * Don't create core dumps on CTRL-c * Fix replacing Content-Type: headers * Fix NULL pointer read / segfault * Fix several build issues * Fix several documentation issues 12.09.2021 Release v2.0.0 * OpenSSL: Fix CRL checking * OpenSSL: Implement ALPN * OpenSSL: Fix memory leaks * Update license information * Fix for reproducible builds * Add Accept-Encoding: lzip * Rename --retry-on-http-status to --retry-on-http-error * Limit to page requisites only for leaf pages * Allow list of tokens for Connection: header * Fix directory clash with --no-clobber * Fix NULL dereference with --convert-links * Using --robots=off downloads robots.txt * Add pkg-config support for GPGME * Fix conversion (-k) in combination with -E * Fix cookie file header to be recognized by 'file' command * Fix loading CA certs when 'system' is not supported * Add option --method, improving backward compatibility * Add option --body-data, improving backward compatibility * Add option --body-file, improving backward compatibility * Add option --ignore-length, improving backward compatibility * Add option --convert-file-only, improving backward compatibility * Add option --download-attr to make use of the HTML5 'download' attribute * Support terminal hyperlinks in output * Configure switch --disable-manylibs to disable building small libraries * Support --background on Windows * Add option --bind-interface * Add HTTP2 uploading * Support the HTML download attribute (for a and area tags) * Add option --download-attr=[strippath|usepath] to control download attribute support * OpenSSL: Add OCSP support * OpenSSL: Implement OCSP stapling * Support data: URL in srcset attribute * Fixed diverse issues * Improved code, documentation, building, testing, CI, ... 30.08.2019 Release 1.99.2 (beta) * Improve docs * Improve build system * More continuous integration testing * Add functionality tests * Add examples * Add HTTP/2 support for test suite (GSOC project) * Add OCSP responder for test suite (GSOC project) * Add new option --keep-extension * Add new option --retry-on-http-status * Add new option --dns-cache-preload * Add -X/--exclude-directories and -I/--include-directories * Add new option --save-content-on * Add new option --limit-rate * Add new option --unlink (Wget1.x compatibility) * Add new option --start-pos (Wget1.x compatibility) * Add new option --no-if-modified-since * Add new option --ocsp-server * Add new option --ocsp-nonce * Add new option --ocsp-date * Add bitmap type to libwget * Add support for Chromium's HSTS Preload List * Add zstd decompression (RFC8478) * Add WolfSSL as alternative TLS backend * Add OpenSSL as alternative TLS backend * Add arguments fail / nofail to --verify-sig * Add TLSv1_x to --secure-protocol * Add support for TCP FastOpen Linux 4.11+ style * Add basic HTML entity decoding (RFC1866) * Add TLS 1.3 post-handshake authentication * Add XDG Desktop Specification support for config files * Remove support for libidn2 < 0.14 and libunistring * Remove option --gnutls-options (use --secure-protocol instead) * Rename --dns-caching to --dns-cache (Wget1.x compatibility) * Skip -np/--no-parent for CSS URLs * Enable syncing with translationproject.org while bootstrapping * Fix HTTP/2 slowness issue * Fix xattr behavior (CVE-2018-20483) * Fixed a pile of bugs * Made many changes to the libwget API 30.05.2018 Release 1.99.1 (alpha) * Enhance docs * Enhance building on many platforms * Enhance statistics * Add --http2-request-window=SIZE * Add --https-enforce=TYPE * Add --compression=TYPE * Add --retry-connrefused * Add more fuzzing * Add more tests * Add more CI testing * Fix bugs 11.01.2018 Release 1.99.0 (alpha) * Add HTTP/2.0 * Add IDNA 2008 / TR46 * Add Travis-CI .yml file for Linux and MacOS * Build with gnulib * Moved to Doxygen (from GTK-Doc) * Support Shoutcast response (ICY) * Use colors for debug and error messages * Use TCP Fast Open (TFO) for TLS handshakes * Add TLS False Start * Add TLS Session resumption * Add sanitizer configure options * Add 'make check-coverage' for viewing test code coverage * Add Public Key Pinning (HPKP) * Add brotli (br) compression method * Add mime type filtering * Add GPG signature checking * Add fuzzing * Add plugins * Add statistic functions * Use libmicrohttpd (MHD) for testing *** Previous Mget releases *** 02.05.2015 New release v0.1.9 Made cross-compilable with MinGW on Linux Fix for 'unsigned char' architectures Add --follow-tags and --ignore-tags Added multithreaded progress bar (--progress=bar) Added --backups Added --post-data and --post-file Add port to Referer HTTP header Fixed race condition when having a filename / directory clash Added --crl-file option to respect CRLs Added OCSP (stapling + responder request) Added --ocsp and --ocsp-stapling Added OCSP response caching Added pkg-config support for library searching Added wildcard support for -D/--domains and --exclude-domains Removed --load-hsts and --save-hsts (will be automatically done) Compilable on Solaris Fixed lots of Coverity scan issues Fixed several small bugs 27.08.2014 New release v0.1.8 forgot to bump up version 27.08.2014 New release v0.1.7 added -m / --mirror added descriptions of options to --help 15.08.2014 New release v0.1.6 Check cookie domains via libpsl against the Public Sufix List Fixed -p/--page-requisites Fixed HTML parsing of uppercase attributes (fixes issue #21) Added --accept, --reject, --ignore-case Added -k/--convert-links and -K/--backup-converted 08.03.2014 Fixed sigfault in cookie handling, introduced in v0.1.5. 02.03.2014 Added --gnutls-options to directly set a GnuTLS priority string for secure connections. 26.02.2014 New release v0.1.5 19.02.2014 Added support for RFC 6266 Content-Disposition 17.02.2014 Fixed a race condition that could lead to sigfaults. Fixed waiting for threads at program exit. 16.02.2014 Made HTTPS test work with GnuTLS 2.12. Added option -/t--tries to Wget. 12.02.2014 Added HTTPS server code, used in test suite. Added new test test-i-https. 06.02.2014 Changed the naming convention of types. Support pkg-config. Fixed generating of man pages. 01.02.2014 Added support for RFC 6797 HTTP Strict Transport Security (HSTS). Do not download robots.txt twice for HTTP and HTTPS if --protocol-directories is not given. 24.01.2014 New release v0.1.4 Use libnettle for checksumming if libgnutls is not available or < 2.10.0. 21.01.2014 Gtk-doc is not required any more to configure and compile the project. Made project configurable and compilable on Cygwin. 17.01.2014 Release first public version v0.1.3 (master branch) Created new branch 'develop' for further hacking. 02.01.2014 Added support for bzip2 Content-Encoding / Accept-Encoding type. It was easy going after having gzip and lzma decompression implemented. Created a mailing list for discussions around Wget. Subscribe by sending a mail to wget-bugs+subscribe@googlegroups.com. 31.12.2013 Added support for XZ / LZMA Content-Encoding / Accept-Encoding type (LZMA2 compression). See https://wiki.mozilla.org/LZMA2_Compression for setting up an Apache to support it. See https://bugzilla.mozilla.org/show_bug.cgi?id=366559 for Firefox/Mozilla discussion. 30.12.2013 Extended --adjust-extension to attach .rss resp. .atom to RSS and Atom Feeds. Implemented Wget options -w/--wait, --random-wait, -t/--tries. With these you can (randomly) slow down website downloading. 25.12.2013 Added URL scanning of RSS 2.0 feeds using the internal XML parser. 15.12.2013 Added URL scanning of Atom 1.0 feeds using the internal XML parser. 24.11.2013 Added URL scanning of sitemaps using the internal XML parser. ---------------------------------------------------------------------- Copyright information: Copyright (C) 2015-2024 Free Software Foundation, Inc. Permission is granted to anyone to make or distribute verbatim copies of this document as received, in any medium, provided that the copyright notice and this permission notice are preserved, thus giving the recipient permission to redistribute in turn. Permission is granted to distribute modified versions of this document, or of portions of it, under the above conditions, provided also that they carry prominent notices stating who last changed them. ================================================== v2.2.0: * First public release