xref: /wget/
Name Date Size

..29-Nov-20194 KiB

.gitignoreH A D30-May-20191.6 KiB

.gitlab-ci.ymlH A D21-Nov-20199.8 KiB

.gitmodulesH A D26-Oct-201887

.lgtm.ymlH A D19-Oct-2018130

.mailmapH A D19-Nov-2019779

.travis.ymlH A D13-Dec-2018971

.x-sc_po_checkH A D13-Aug-201711

.x-sc_trailing_blankH A D13-Aug-2017133

AUTHORSH A D21-Nov-20192.3 KiB

bootstrapH A D10-Jan-202032.8 KiB

bootstrap.confH A D10-Jan-20203.1 KiB

build-aux/H10-Jan-20204 KiB

cfg.mkH A D10-Jan-20201.2 KiB

ChangeLogH A D13-Aug-20170

ChangeLog-2014-12-10H A D13-Aug-2017666.9 KiB

configure.acH A D10-Jan-202031.6 KiB

contrib/H28-Dec-20184 KiB

CONTRIBUTING.mdH A D28-Apr-20192.5 KiB

COPYINGH A D14-Jan-201834.3 KiB

doc/H10-Jan-20204 KiB

fuzz/H10-Jan-20204 KiB

gnulib/H13-Aug-20174 KiB

lib/H03-May-20194 KiB

m4/H10-Jan-20204 KiB

MAILING-LISTH A D14-Jan-20181.1 KiB

Makefile.amH A D10-Jan-20202.9 KiB

msdos/H13-Aug-20174 KiB

NEWSH A D10-Jan-202039.2 KiB

po/H10-Jan-20204 KiB

READMEH A D10-Jan-20203.9 KiB

README.checkoutH A D10-Jan-202010.2 KiB

src/H10-Jan-20204 KiB

testenv/H10-Jan-20204 KiB

tests/H10-Jan-20204 KiB

util/H10-Jan-20204 KiB

vms/H13-Aug-20174 KiB

README

1                                                          -*- text -*-
2GNU Wget
3========
4                  Current Web home: https://www.gnu.org/software/wget/
5
6GNU Wget is a free utility for non-interactive download of files from
7the Web.  It supports HTTP, HTTPS, and FTP protocols, as well as
8retrieval through HTTP proxies.
9
10It can follow links in HTML pages and create local versions of remote
11web sites, fully recreating the directory structure of the original
12site.  This is sometimes referred to as "recursive downloading."
13While doing that, Wget respects the Robot Exclusion Standard
14(/robots.txt).  Wget can be instructed to convert the links in
15downloaded HTML files to the local files for offline viewing.
16
17Recursive downloading also works with FTP, where Wget can retrieves a
18hierarchy of directories and files.
19
20With both HTTP and FTP, Wget can check whether a remote file has
21changed on the server since the previous run, and only download the
22newer files.
23
24Wget has been designed for robustness over slow or unstable network
25connections; if a download fails due to a network problem, it will
26keep retrying until the whole file has been retrieved.  If the server
27supports regetting, it will instruct the server to continue the
28download from where it left off.
29
30If you are behind a firewall that requires the use of a socks style
31gateway, you can get the socks library and compile wget with support
32for socks.
33
34Most of the features are configurable, either through command-line
35options, or via initialization file .wgetrc.  Wget allows you to
36install a global startup file (/usr/local/etc/wgetrc by default) for
37site settings.
38
39Wget works under almost all Unix variants in use today and, unlike
40many of its historical predecessors, is written entirely in C, thus
41requiring no additional software, such as Perl.  The external software
42it does work with, such as OpenSSL, is optional.  As Wget uses the GNU
43Autoconf, it is easily built on and ported to new Unix-like systems.
44The installation procedure is described in the INSTALL file.
45
46As with other GNU software, the latest version of Wget can be found at
47the master GNU archive site ftp.gnu.org, and its mirrors.  Wget
48resides at <ftp://ftp.gnu.org/pub/gnu/wget/>.
49
50Please report bugs in Wget to <bug-wget@gnu.org>.
51
52See the file `MAILING-LIST' for information about Wget mailing lists.
53Wget's home page is at <https://www.gnu.org/software/wget/>.
54
55If you would like to contribute code for Wget, please read
56CONTRIBUTING.md.
57
58Wget was originally written and mainained by Hrvoje Niksic.  Please see
59the file AUTHORS for a list of major contributors, and the ChangeLogs
60for a detailed listing of all contributions.
61
62
63Copyright (C) 1995-2020 Free Software Foundation, Inc.
64
65This program is free software; you can redistribute it and/or modify
66it under the terms of the GNU General Public License as published by
67the Free Software Foundation; either version 3 of the License, or
68(at your option) any later version.
69
70This program is distributed in the hope that it will be useful,
71but WITHOUT ANY WARRANTY; without even the implied warranty of
72MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
73GNU General Public License for more details.
74
75You should have received a copy of the GNU General Public License
76along with this program; if not, write to the Free Software
77Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301
78USA.
79
80Additional permission under GNU GPL version 3 section 7
81
82If you modify this program, or any covered work, by linking or
83combining it with the OpenSSL project's OpenSSL library (or a
84modified version of that library), containing parts covered by the
85terms of the OpenSSL or SSLeay licenses, the Free Software Foundation
86grants you additional permission to convey the resulting work.
87Corresponding Source for a non-source form of such a combination
88shall include the source code for the parts of OpenSSL used as well
89as that of the covered work.
90

README.checkout

1Compiling From Repository Sources
2
3   To reduce bandwidth and needless updates, the source code repository
4   does not contain automatically-generated files, even when these are
5   normally present in the distribution tarballs. Therefore, to build GNU
6   Wget from the sources in the repository, you'll need to have one or
7   more of the following (note that gettext, OpenSSL, GnuTLS, libidn2,
8   libiconv, libpsl, libpcre, pkg-config, libmetalink and GnuPG are not
9   absolutely required):
10
11     * [20]autoconf (currently, GNU Wget requires version 2.61). This is
12       needed to generate the configure script from configure.in. This is
13       not required when building from a tarball distribution; only when
14       building from repository sources.
15
16     * [21]automake (currently, GNU Wget requires version 1.10.1). This is
17       needed for generating the Makefile.in templates that the configure
18       script uses to generate the Makefiles. As with autoconf, it is not
19       required when building from a tarball distribution; only when
20       building from repository sources.
21
22     * [22]flex is needed to generate the CSS-parsing code.
23
24     * [23]Perl, if you wish to generate the wget(1) manpage, or run the
25       tests in the tests/ sub directory. Tarball distributions include an
26       already-generated wget.1 manual. The command "make check" runs the
27       test suite written in perl and python (see below). To execute all the tests
28       you need libwww-perl and libio-socket-ssl-perl perl library.
29       GNU/Linux distros likely have these two libraries packaged.
30       If not, install them directly with "perl -MCPAN -e 'install Bundle::LWP'"[24].
31       If that fails then you most likely don't have the CPAN module installed.
32       First download [25]CPAN and [26]install it, then try again the above command.
33       Now "make check" should pass most of the tests in the test suite.
34
35     * [45]Python3, if you want to run the tests in the testenv/ subdirectory.
36       Keep in mind that "make check" will try to run all the Perl and Python3
37       tests. More information about the test suite below in the section
38       "Testing and development".
39
40     * [27]texinfo in order to generate Info, PostScript and/or HTML
41       documentation. You don't need texinfo in order to generate the
42       wget(1) manpage; however, note that the manpage does not include
43       the full documentation. Tarball distributions include the
44       already-generated documentation in these formats.
45
46     * [28]gettext, if you wish to compile with NLS (Native Language
47       Support), which is enabled by default. If you do not have gettext,
48       you can compile without it by specifying --disable-nls to the
49       ./configure script. This is true regardless of where you obtained
50       the source you're building. NOTE: if you get errors about
51       AM_GNU_GETTEXT and/or AM_INTL_SUBDIR, you probably have a buggy
52       version of GNU m4. Upgrade to the latest version. You may also need
53       to export M4=<new m4 path>, to be sure that autoconf/automake use
54       it instead of the old one.
55
56     * [29]GnuTLS to allow encrypted data transfer (HTTPS).
57       You need the header files and the library installed.
58       As an alternative, you can use [30]OpenSSL by specifying
59       --with-ssl=openssl to the ./configure script.
60       If you do not want HTTPS support, specify --without-ssl to the
61       ./configure script.
62       If you want to compile+link a non-system library version use
63       --with-libgnutls-prefix (or if having pkg-config: see description
64       below).
65
66     * [30]OpenSSL to allow encrypted data transfer (HTTPS) an
67       alternative to [31]GnuTLS.
68       You need the header files and the library installed.
69       If you want to compile+link a non-system library version use
70       --with-libssl-prefix (or if having pkg-config: see description below).
71
72     * [31]libidn2 is required for IDN/IRI support (non-ASCII characters
73       within what would otherwise be URLs).
74
75     * [32]libiconv is required on non-GNU systems, for IDN/IRI support.
76       On GNU systems, the functionality provided by libiconv is already
77       present in the system libraries.
78
79     * [33]git is used to fetch gnulib files trough the bootstrap.sh script.
80
81     * [34]libpsl is required for using a public suffix list to check for valid
82       cookie domains. You need the header files and the library installed.
83
84     * [35]libpcre is required for using Perl-compatible regular expressions
85       with --accept-regex and --reject-regex. You need the header files and
86       the library installed to compile and link Wget with PCRE support.
87
88     * [36]pkg-config helps the ./configure script to find installed libraries.
89       Most libraries provide a pkg-config file (.pc extension) with
90       information about dependencies, header file and library locations.
91       Distributions deliver their specific .pc file to each library.
92       If you want to compile+link against your own library version, make a
93       copy of the appropriate .pc file and amend it to your needs (e.g. edit
94       the line starting with prefix=). Before you execute the ./configure
95       script, set (and export) PKG_CONFIG_PATH to the directory where you
96       saved the .pc file. Example:
97       $ PKG_CONFIG_PATH="." ./configure
98
99     * [46]libmetalink is needed to enable Metalink files support.
100
101     * [47]GnuPG with GPGME is used to verify GPG-signed Metalink resources.
102
103     * [48]libcares is needed to bind DNS resolving to a given IP address.
104       The command line options --dns-servers and --bind-dns-address are
105       only available when configured with --with-cares.
106
107     * [49]Gpef is required by some gnulib modules, most notably "unicase". You
108	   need the application installed on the machine to compile Wget from git.
109
110
111   For those who might be confused as to what to do once they check out
112   the source code, considering configure and Makefile do not yet exist at
113   that point, a shell script called bootstrap has been provided. After
114   calling ./bootstrap you're ready to build GNU Wget in the normal
115   fashion, with ./configure and make.
116
117   So, to sum up, after checking out the source code as described above,
118   you may proceed as follows:
119    1. Change to the topmost GNU Wget directory:
120 $  cd wget        # assumes you've cloned a repository to "./wget"
121    2. Generate all the automatically-generated files required prior to
122       configuring the package:
123 $  ./bootstrap (or: bash ./bootstrap)
124    3. Configure the package and compile it:
125 $  ./configure --enable-assert [some_parameters]
126 $  make
127    4. Hack, compile, test, hack, compile, test...
128 $  src/wget --version
129 GNU Wget 1.12-devel (9cb2563197bc)
130
131Testing and development
132
133 All developers are requested to enable the assertions on their development
134 builds to ensure a stable codebase. Assertions are added to state certain
135 assumptions about the code and its data which all developers should be mindful
136 of. To enable assertions, run the configure command with the --enable-assert
137 option, like this:
138
139 $ ./configure --enable-assert [other configure options]
140
141 Both the Perl and Python test suites (test/ and testenv/) include support for GDB and Valgrind.
142 The environment variables GDB_TESTS and VALGRIND_TESTS are available to
143 enable such wrappers. If specified, Wget would be run through either of them
144 during the test. For example:
145
146 $ cd testenv
147 $ VALGRIND_TESTS=1 ./Test-O.py
148
149 That would execute Test-O.py test case, but running Wget through Valgrind.
150
151 GDB has preference over Valgrind. If both variables have been asserted,
152 Wget would be run through GDB.
153
154 If you run a test case through GDB, please bear in mind that it could give
155 a false negative. This is because some tests that expect Wget to fail
156 rely on Wget's return code. However, when run through GDB, its return code
157 will always be zero, causing the test to claim failure. This wrapper for
158 GDB is, however, very useful to tackle bugs, allowing one to write a test case
159 for some specific bug and then using GDB to fix it more easily. Otherwise,
160 a dedicated server would have to be set up and write a custom CGI just to reproduce
161 that bug, which might be tedious. Tests should only be run through GDB for that purpose.
162
163
164 Copyright (C) 2008, 2010, 2014-2015, 2018-2020 Free Software
165 Foundation, Inc.
166 License GPLv3+: GNU GPL version 3 or later
167 <https://www.gnu.org/licenses/gpl.html>.
168 This is free software: you are free to change and redistribute it.
169 There is NO WARRANTY, to the extent permitted by law.
170
171 Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
172
173     * [36]Edit
174     * [37]Comments
175     * [38]Info
176     * [39]Attachments
177     * More Actions:
178       [Raw Text................] Do
179
180     * [40]MoinMoin Powered
181     * [41]Python Powered
182     * [42]GPL licensed
183     * [43]Valid HTML 4.01
184     __________________________________________________________________
185
186   All content (C) 2007 Free Software Foundation. For terms of use,
187   redistribution, and modification, please see the [45]WikiLicense page.
188
189References
190
191  20. https://www.gnu.org/software/autoconf/
192  21. https://www.gnu.org/software/automake/
193  22. https://github.com/westes/flex
194  23. https://www.perl.org/
195  24. http://search.cpan.org/dist/libwww-perl/lib/Bundle/LWP.pm
196  25. http://search.cpan.org/CPAN/authors/id/A/AN/ANDK/CPAN-1.9402.tar.gz
197  26. http://apthorpe.cynistar.net/code/configuring_cpan.html
198  27. https://www.gnu.org/software/texinfo/
199  28. https://www.gnu.org/software/gettext/
200  29. https://www.gnu.org/software/gnutls/
201  30. https://www.openssl.org/
202  31. https://www.gnu.org/software/libidn/#libidn2
203  32. https://www.gnu.org/software/libiconv/
204  33. https://git-scm.com/
205  34. https://github.com/rockdaboot/libpsl
206  35. https://www.pcre.org
207  36. http://wget.addictivecode.org/CompilingRepoSources?action=edit&editor=text
208  37. http://wget.addictivecode.org/CompilingRepoSources
209  38. http://wget.addictivecode.org/CompilingRepoSources?action=info
210  39. http://wget.addictivecode.org/CompilingRepoSources?action=AttachFile
211  40. https://moinmo.in/
212  41. https://moinmo.in/Python
213  42. https://moinmo.in/GPL
214  43. https://validator.w3.org/check?uri=referer
215  44. http://wget.addictivecode.org/WikiLicense
216  45. https://www.python.org/
217  46. https://launchpad.net/libmetalink
218  47. https://www.gnupg.org
219  48. https://c-ares.haxx.se/
220  49. https://www.gnu.org/software/gperf/
221