wget is a Linux command-line utility for retrieving files from the web, via HTTP, HTTPS and FTP protocols. When you are using wget to download a file at a particular HTTP url, wget sends an appropriate HTTP request to a destination web server.
To view default HTTP request header being sent by wget, you can use "-d" option.
---request begin--- GET / HTTP/1.0 User-Agent: Wget/1.12 (linux-gnu) Accept: */* Host: www.google.com Connection: Keep-Alive ---request end---
Sometimes you may want to customize the default HTTP request header used by wget. For example, you may want to customize "User-Agent" field as some sites rely on "User-Agent" string to block robots like wget to retrieve their content. You may want to add an additional "Accept-Encoding" field in order to test encoding schemes of your web server. In some other cases, you may need to set "Host" field properly to be able to access a web server running on name-based virtual hosting.
wget allows you to send an HTTP request with custom HTTP headers. To supply custom HTTP headers, use "--header" option. You can use "--header" option as many time as you want in a single run.
---request begin--- GET / HTTP/1.0 User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 Accept: */* Host: www.google.com Connection: Keep-Alive Referer: http://xmodulo.com/ Accept-Encoding: compress, gzip ---request end---
If you would like to permanently set the default HTTP request header you want to use with wget, you can use ~/.wgetrc configuration file. You can specify as many header fields as you want in ~/.wgetrc.
header = User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 header = Referer: http://xmodulo.com/ header = Accept-Encoding: compress, gzip
Once you have configured ~/.wgetrc, you no longer need to use "--header" option with wget.
Subscribe to Xmodulo
You can have daily Linux tutorials and FAQs delivered to your email inbox. Simply enter your email address below to subscribe to our mailing list. You will receive hands-on guides and carefully written tutorials related to Linux, everything for free.