How to download multiple files with wget

Last updated on August 16, 2020 by Dan Nanni

wget is a cross-platform utility for downloading files from the web. Written in portable C, wget is available on many operating systems including Linux, MacOS X, FreeBSD, and Windows. You typically use wget to retrieve an object or a web page at a particular URL (e.g., http://www.cnn.com/index.html"), and optionally with custom HTTP headers. However, what if you want to download multiple files or URLs? While you could invoke wget multiple times manually, there are several ways to download multiple files with wget in one shot.

If you know a list of URLs to fetch, you can simply supply wget with an input file that contains a list of URLs. Use -i option is for that purpose.

$ wget -i url_list.txt

If URL names have a specific numbering pattern, you can use curly braces to download all the URLs that match the pattern. For example, if you want to download Linux kernels starting from version 3.2.1 to 3.2.15, you can do the following.

$ wget https://www.kernel.org/pub/linux/kernel/v3.0/linux-3.2.{1..15}.tar.bz2

So far you specified all individual URLs when running wget, either by supplying an input file or by using numeric patterns.

If a target web server has directory indexing enabled, and all the files to download are located in the same directory, you can download all of them, by using wget's recursive retrieval option.

What do I mean by directory indexing being enabled? If directory indexing is enabled on aaa.com, going to http://aaa.com/directory will give you a listing of files in that directory (assuming there is no separate index.html in that directory). In this case, you can use the -r option of wget to download multiple files in the directory. For example:

$ wget -r -l1 -A.bz2 http://aaa.com/directory

In the above example, -r and -l1 options together enable 1-level deep recursive retrieval, and -A option specifies lists of file name suffixes to accept during recursive download (.bz2 in this case).

Support Xmodulo

This website is made possible by minimal ads and your gracious donation via PayPal or credit card

Please note that this article is published by Xmodulo.com under a Creative Commons Attribution-ShareAlike 3.0 Unported License. If you would like to use the whole or any part of this article, you need to cite this web page at Xmodulo.com as the original source.