Awk

Awk is a powerful language mainly for data extraction. My main use has been to extract specific records or fields from large data set files before further analyzing in Excel or other text editors. Some data sets I have worked with have been so large that Excel is unable to load the data set, which has required initial data extraction by either extracting only the required combination of records or fields of the data set.

I will be collecting various awk command examples as I run into the need to create them. Awk is a powerful language for extracting and manipulating data text files. Several examples include function similar to using grep, separating columns of data, printing only certain columns or performing aggregate functions.

Printing Columns

Awk command to print columns from a file. -F “,” defines the column separator. $1 determines the field position you want to print. $0 will print the entire row of data.

awk -F "," '{print $1}' poplation_by_zipcode

Subtracting Values

Awk command for subtracting previous value from current value to return difference.

awk -F "," '{if($2 == 24014) {print $1, $2, $3}}' poplation_by_zipcode.csv | awk 'NR==1{p=$3;next}{print $1, $2, $3, $3-p; p=$3}END{print p}'

Example Source Data:

Resulting Output of awk command:

Adding Line Number

awk '{print NR" "$0}' file.txt

Example Source Data:

$cat file.txt
First Line of file
Another line in file
This is a last line in file

Resulting Output of awk command:

awk ‘{print NR" “$0}’ file.txt
1 First Line of file
2 Another line in file
3 This is a last line in file

Calculating Sum of Values:

Awk command for adding all values in the third column together for a column total.

awk -F',' '{sum+=$3;} END{print sum;}' sample_data.csv

Example Source Data:

Resulting Output of awk command: