How to use custom HTTP headers with wget

Last updated on October 10, 2020 by Dan Nanni

wget is a Linux command-line utility for retrieving files from the web, via HTTP, HTTPS and FTP protocols. When you are using wget to download a file at a particular HTTP URL, wget sends an appropriate HTTP request to a destination web server.

To view default HTTP request header being sent by wget, you can use -d option.

$ wget -d http://www.google.com/

---request begin---
GET / HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: www.google.com
Connection: Keep-Alive

---request end---

Sometimes you may want to customize the default HTTP request header used by wget. For example, you may want to customize User-Agent field as some websites rely on User-Agent string to block robots like wget to retrieve their content. You may want to add an additional Accept-Encoding field in order to test encoding schemes of your web server. In some other cases, you may need to set Host field properly to be able to access a web server running on name-based virtual hosting.

wget allows you to send an HTTP request with custom HTTP headers. To supply custom HTTP headers, use --header option. You can use --header option as many time as you want in a single run.

$ wget -d --header="User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11" --header="Referer: http://xmodulo.com/" --header="Accept-Encoding: compress, gzip" http://www.google.com/

---request begin---
GET / HTTP/1.0
User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
Accept: */*
Host: www.google.com
Connection: Keep-Alive
Referer: http://xmodulo.com/
Accept-Encoding: compress, gzip

---request end---

If you would like to permanently set the default HTTP request header you want to use with wget, you can use ~/.wgetrc configuration file. You can specify as many header fields as you want in ~/.wgetrc.

$ vi ~/.wgetrc

header = User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
header = Referer: http://xmodulo.com/
header = Accept-Encoding: compress, gzip

Once you have configured ~/.wgetrc, you no longer need to use --header option with wget.

There are also other command-line tools that can send HTTP requests with custom HTTP headers. For example, curl provides a similar interface to set custom HTTP headers, and HTTPie is more user-friendly to perform the same task.

Support Xmodulo

This website is made possible by minimal ads and your gracious donation via PayPal or credit card

Please note that this article is published by Xmodulo.com under a Creative Commons Attribution-ShareAlike 3.0 Unported License. If you would like to use the whole or any part of this article, you need to cite this web page at Xmodulo.com as the original source.