How to use awk command in Linux

Last updated on September 29, 2020 by Dan Nanni

Text processing is at the heart of Unix. From pipes to the /proc subsystem, the "everything is a file" philosophy pervades the operating system and all of the tools built for it. Because of this, getting comfortable with text-processing is one of the most important skills for an aspiring Linux system administrator, or even any power user, and awk is one of the most powerful text-processing tools available outside general-purpose programming languages.

The simplest awk task is selecting fields from stdin; if you never learn any more about awk than this, you'll still have at your disposal an extremely useful tool.

By default, awk separates input lines by whitespace. If you'd like to select the first field from input, you just need to tell awk to print out $1:

$ echo 'one two three four' | awk '{print $1}'

one

(Yes, the curly-brace syntax is a little weird, but I promise that's about as weird as it gets in this lesson.)

Can you guess how you'd select the second, third, or fourth fields? That's right, with $2, $3, and $4, respectively.

$ echo 'one two three four' | awk '{print $3}'

three

Often when text munging, you need to create a specific format of data, and that covers more than just a single word. The good news is that awk makes it easy to print multiple fields, or even include static strings:

$ echo 'one two three four' | awk '{print $3,$1}'

three one

$ echo 'one two three four' | awk '{print "foo:",$3,"| bar:",$1}'

foo: three | bar: one

Ok, but what if your input isn't separated by whitespace? Just pass awk the -F flag with your separator:

$ echo 'one mississippi,two mississippi,three mississippi,four mississippi' | awk -F , '{print $4}'

four mississippi

Occasionally, you may find yourself working with data with a varied number of fields, and you just know you want the last one. awk prepopulates the $NF variable with the number of fields, so you can use it to grab the last element:

$ echo 'one two three four' | awk '{print $NF}'

four

You can also do simple math on $NF, in case you need the next-to-last field:

$ echo 'one two three four' | awk '{print $(NF-1)}'

three

Or even the middle field:

$ echo 'one two three four' | awk '{print $((NF/2)+1)}'

three

$ echo 'one two three four five' | awk '{print $((NF/2)+1)}'

three

While this is all very useful, you can get away with forcing sed, cut, and grep into a form to get these results, as well (albeit with a lot more work).

So, I'll leave you with one last introductory feature of awk, maintaining state across lines.

$ echo -e 'one 1ntwo 2' | awk '{print $2}'

1
2

$ echo -e 'one 1ntwo 2' | awk '{sum+=$2} END {print sum}'

(The END indicates that we should only perform the following block after we finish processing every line.)

The case where I've used this is to sum up bytes from web server request logs. Imagine we have an access log that looks like this:

$ cat requests.log

Jul 23 18:57:12 httpd[31950]: "GET /foo/bar HTTP/1.1" 200 344
Jul 23 18:57:13 httpd[31950]: "GET / HTTP/1.1" 200 9300
Jul 23 19:01:27 httpd[31950]: "GET / HTTP/1.1" 200 9300
Jul 23 19:01:55 httpd[31950]: "GET /foo/baz HTTP/1.1" 200 6401
Jul 23 19:02:31 httpd[31950]: "GET /foo/baz?page=2 HTTP/1.1" 200 6312

We know the last field is the number of bytes of the response. We've already learned how to extract them using print and $NF:

$ < requests.log awk '{print $NF}'

And so we can sum into a variable to gather the total number of bytes our webserver has served to clients during the timespan of our log:

$ < requests.log awk '{totalBytes+=$NF} END {print totalBytes}'

If you're looking for more to do with awk, I recommend you grab a copy of the original awk programming book. You may also enjoy Eric Pement's collection of awk one-liners.

Support Xmodulo

This website is made possible by minimal ads and your gracious donation via PayPal or credit card

Please note that this article is published by Xmodulo.com under a Creative Commons Attribution-ShareAlike 3.0 Unported License. If you would like to use the whole or any part of this article, you need to cite this web page at Xmodulo.com as the original source.