How to download multiple files with wget

The wget is a Linux utility for downloading files from the web. You typically use wget to retrieve an object or a web page at a particular URL (e.g., "wget http://www.cnn.com/index.html"). What if you want to download multiple files? While you could invoke wget multiple times manually, there are several ways to download multiple files with wget in one shot.

If you know a list of URLs to fetch, you can simply supply wget with an input file that contains a list of URLs. Use "-i" option is for that purpose.

$ wget -i url_list.txt

If URL names have a specific numbering pattern, you can use curly braces to download all the URLs that match the pattern. For example, if you want to download Linux kernels starting from version 3.2.1 to 3.2.15, you can do the following.

$ wget https://www.kernel.org/pub/linux/kernel/v3.0/linux-3.2.{1..15}.tar.bz2

So far you specified all individual URLs when running wget, either by supplying an input file or by using numeric patterns.

If a target web server has directory indexing enabled, and all the files to download are located in the same directory, you can download all of them, by using wget's recursive retrieval option.

What do I mean by directory indexing being enabled? If directory indexing is enabled on aaa.com, going to http://aaa.com/directory will give you a listing of files in that directory (assuming there is no separate index.html in that directory). In this case, you can use the "-r" option of wget to download multiple files in the directory. For example:

$ wget -r -l1 -A.bz2 http://aaa.com/directory

In the above example, "-r" and "-l1" options together enable 1-level deep recursive retrieval, and "-A" option specifies lists of file name suffixes to accept during recursive download (".bz2" in this case).

Subscribe to Xmodulo

Do you want to receive Linux FAQs, detailed tutorials and tips published at Xmodulo? Enter your email address below, and we will deliver our Linux posts straight to your email box, for free. Delivery powered by Google Feedburner.

The following two tabs change content below.
Dan Nanni is the founder and also a regular contributor of Xmodulo.com. He is a Linux/FOSS enthusiast who loves to get his hands dirty with his Linux box. He likes to procrastinate when he is supposed to be busy and productive. When he is otherwise free, he likes to watch movies and shop for the coolest gadgets.

Leave a comment

Your email address will not be published. Required fields are marked *

Current ye@r *