Awk is a powerful tool that we’ll take a look at to parse our logs. It’s much more powerful than what we’re going to be using it for in these tutorials, but I encourage you to learn more about awk on your own if you like it.
Awk separates a file into columns (known as fields) and rows – whenever you think of a file and how awk parses it, think of columns and rows.
With this knowledge, we can now take a look at awk’s -F flag which allows us to specify how awk sees columns. By default, awk uses a space as its delimiter to read through text. In the following example, we can see that if we just print then awk is going to print the entire file:
awk '{ print }' /var/log/nginx/site.access.log| head -n3
100.43.90.123 supadupa.rocks - [27/Apr/2019:00:54:32 +0000] "GET /robots.txt HTTP/1.1" 200 106 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
127.0.0.1 supadupa.rocks - [27/Apr/2019:00:59:52 +0000] "GET //?wp-cmd=ping HTTP/1.0" 200 5 "-" "curl/7.47.0"
127.0.0.1 supadupa.rocks - [27/Apr/2019:00:59:53 +0000] "GET //wp-admin/?wp-cmd=ensure HTTP/1.0" 200 305 "-" "curl/7.47.0"Now if we go to print field $1 (the IP addresses) we simply need – awk '{ print $1 }' /var/log/nginx/site.apachestyle.log
awk '{ print $1 }' /var/log/nginx/site.access.log |head -n3
100.43.90.123
127.0.0.1
127.0.0.1If I wanted to search like grep, but using awk, there’s a solution for that. With awk, you have to put your regex nested between two / in order for it to be interpreted.
Additionally, if your regex has special characters inside of it (like ?!*& or other other special characters) you will need to escape them using a \. See awk’s man page for more information.
awk -F'"' '/104\.196\.177\.220/{ print $2 }' /var/log/nginx/site.apachestyle.log |head -n3
POST /wp-cron.php?doing_wp_cron=1556326791.3251020908355712890625 HTTP/1.0
POST /wp-cron.php?doing_wp_cron=1556334044.2906210422515869140625 HTTP/1.0
POST /wp-cron.php?doing_wp_cron=1556336718.0919361114501953125000 HTTP/1.0That’s all for now. I hope you enjoyed reading this post!