How to download multiple files with wget

wget is a cross-platform utility for downloading files from the web. Written in portable C, wget is available on many operating systems including Linux, MacOS X, FreeBSD, and Windows. You typically use wget to retrieve an object or a web page at a particular URL (e.g., "wget http://www.cnn.com/index.html"). However, what if you want to download multiple files? While you could invoke wget multiple times manually, there are several ways to download multiple files with wget in one shot.

If you know a list of URLs to fetch, you can simply supply wget with an input file that contains a list of URLs. Use "-i" option is for that purpose.

$ wget -i url_list.txt

If URL names have a specific numbering pattern, you can use curly braces to download all the URLs that match the pattern. For example, if you want to download Linux kernels starting from version 3.2.1 to 3.2.15, you can do the following.

$ wget https://www.kernel.org/pub/linux/kernel/v3.0/linux-3.2.{1..15}.tar.bz2

So far you specified all individual URLs when running wget, either by supplying an input file or by using numeric patterns.

If a target web server has directory indexing enabled, and all the files to download are located in the same directory, you can download all of them, by using wget's recursive retrieval option.

What do I mean by directory indexing being enabled? If directory indexing is enabled on aaa.com, going to http://aaa.com/directory will give you a listing of files in that directory (assuming there is no separate index.html in that directory). In this case, you can use the "-r" option of wget to download multiple files in the directory. For example:

$ wget -r -l1 -A.bz2 http://aaa.com/directory

In the above example, "-r" and "-l1" options together enable 1-level deep recursive retrieval, and "-A" option specifies lists of file name suffixes to accept during recursive download (".bz2" in this case).

Subscribe to Xmodulo

Do you want to receive Linux FAQs, detailed tutorials and tips published at Xmodulo? Enter your email address below, and we will deliver our Linux posts straight to your email box, for free. Delivery powered by Google Feedburner.


Support Xmodulo

Did you find this tutorial helpful? Then please be generous and support Xmodulo!

The following two tabs change content below.
Dan Nanni is the founder and also a regular contributor of Xmodulo.com. He is a Linux/FOSS enthusiast who loves to get his hands dirty with his Linux box. He likes to procrastinate when he is supposed to be busy and productive. When he is otherwise free, he likes to watch movies and shop for the coolest gadgets.

2 thoughts on “How to download multiple files with wget

  1. Very helpful, indeed. But I'd link to point that wget isn't a Linux program. It's just a general C program that's natively present in many UNIX-like systems. That's not just Linux but also Apple's OS X and FreeBSD. Not forgetting to mention that it can run just fine in Windows.

    Anyway, it's a great program. It really helps me on pesky tasks that otherwise would require a lot of handwork.

Leave a comment

Your email address will not be published. Required fields are marked *