wget is a Linux command-line utility for retrieving files from the web, via HTTP, HTTPS and FTP protocols. When you are using wget to download a file at a particular HTTP url, wget sends an appropriate HTTP request to a destination web server.
To view default HTTP request header being sent by wget, you can use "-d" option.
---request begin--- GET / HTTP/1.0 User-Agent: Wget/1.12 (linux-gnu) Accept: */* Host: www.google.com Connection: Keep-Alive ---request end---
Sometimes you may want to customize the default HTTP request header used by wget. For example, you may want to customize "User-Agent" field as some sites rely on "User-Agent" string to block robots like wget to retrieve their content. You may want to add an additional "Accept-Encoding" field in order to test encoding schemes of your web server. In some other cases, you may need to set "Host" field properly to be able to access a web server running on name-based virtual hosting.
wget allows you to send an HTTP request with custom HTTP headers. To supply custom HTTP headers, use "--header" option. You can use "--header" option as many time as you want in a single run.
---request begin--- GET / HTTP/1.0 User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 Accept: */* Host: www.google.com Connection: Keep-Alive Referer: http://xmodulo.com/ Accept-Encoding: compress, gzip ---request end---
If you would like to permanently set the default HTTP request header you want to use with wget, you can use ~/.wgetrc configuration file. You can specify as many header fields as you want in ~/.wgetrc.
header = User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 header = Referer: http://xmodulo.com/ header = Accept-Encoding: compress, gzip
Once you have configured ~/.wgetrc, you no longer need to use "--header" option with wget.
curl is another command-line tool with similar functionality as wget. The curl utility also allows you to set a custom HTTP header. Refer to this guideline for detail on curl.
Subscribe to Xmodulo
Do you want to receive Linux FAQs, detailed tutorials and tips published at Xmodulo? Enter your email address below, and we will deliver our Linux posts straight to your email box, for free. Delivery powered by Google Feedburner.