wget is a cross-platform utility for downloading files from the web. Written in portable C, wget is available on many operating systems including Linux, MacOS X, FreeBSD, and Windows. You typically use wget to retrieve an object or a web page at a particular URL (e.g., "wget http://www.cnn.com/index.html"). However, what if you want to download multiple files? While you could invoke wget multiple times manually, there are several ways to download multiple files with wget in one shot.
If you know a list of URLs to fetch, you can simply supply wget with an input file that contains a list of URLs. Use "-i" option is for that purpose.
If URL names have a specific numbering pattern, you can use curly braces to download all the URLs that match the pattern. For example, if you want to download Linux kernels starting from version 3.2.1 to 3.2.15, you can do the following.
So far you specified all individual URLs when running wget, either by supplying an input file or by using numeric patterns.
If a target web server has directory indexing enabled, and all the files to download are located in the same directory, you can download all of them, by using wget's recursive retrieval option.
What do I mean by directory indexing being enabled? If directory indexing is enabled on aaa.com, going to http://aaa.com/directory will give you a listing of files in that directory (assuming there is no separate index.html in that directory). In this case, you can use the "-r" option of wget to download multiple files in the directory. For example:
In the above example, "-r" and "-l1" options together enable 1-level deep recursive retrieval, and "-A" option specifies lists of file name suffixes to accept during recursive download (".bz2" in this case).
Subscribe to Xmodulo
Do you want to receive Linux FAQs, detailed tutorials and tips published at Xmodulo? Enter your email address below, and we will deliver our Linux posts straight to your email box, for free. Delivery powered by Google Feedburner.
Did you find this tutorial helpful? Then please be generous and support Xmodulo!
Latest posts by Dan Nanni (see all)
- How to set up Apt caching server on Ubuntu or Debian - February 7, 2016
- How to monitor OpenFlow messages with packet sniffer - February 2, 2016
- How to search multiple pdf documents for words on Linux - January 13, 2016