Linux Command sort uniq tr Tools Explanation

2021-08-17 01:45:34
OfStack

Sort Tools

The Linux sort command is used to sort the text file contents.
sort can sort the contents of text files in line units.

sort Common Options

b ignores the space character at the beginning of each line.
-c Check that files have been sorted in order.
-When sorting d, all characters except alphabet, numeric and space characters are ignored.
-When f is sorted, lowercase letters are treated as uppercase letters.
-When sorting i, all characters except ASCII characters between 040 and 176 are ignored.
-m merges several sorted files.
-M sorts the first three letters by month abbreviation.
-n is sorted by value.
-u means that it is only 1 (unique), and the output result is de-weighted.
-o < Output file > Save the sorted results to the specified file.
-r is sorted in reverse order.
-t < Separator character > Specifies the field separator character to use when sorting.
+ < Start field > - < End field > Sorts by the specified field, ranging from the starting field to the first 1 fields of the ending field.
help displays help.
version displays version information

Example of sort Tools

Default

By default, the sort tool sorts alphabetically


[root@1centos ~]# sort /etc/passwd
abrt:x:173:173::/etc/abrt:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin
avahi:x:70:70:Avahi mDNS/DNS-SD Stack:/var/run/avahi-daemon:/sbin/nologin
bin:x:1:1:bin:/bin:/sbin/nologin
chrony:x:992:987::/var/lib/chrony:/sbin/nologin
cockpit-ws:x:990:984:User for cockpit-ws:/:/sbin/nologin
colord:x:997:994:User for colord:/var/lib/colord:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
dirsrv:x:988:982:user for 389-ds-base:/usr/share/dirsrv:/sbin/nologin
dovecot:x:97:97:Dovecot IMAP server:/usr/libexec/dovecot:/sbin/nologin
dovenull:x:981:975:Dovecot's unauthorized user:/usr/libexec/dovecot:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin

Reverse sort the/etc/passwd in Column 3

Here, it is sorted by digital flashback


[root@1centos ~]# sort -t: -rk 3 /etc/passwd
nobody:x:99:99:Nobody:/:/sbin/nologin
ods:x:999:999:softhsm private keys owner:/var/lib/softhsm:/sbin/nologin
polkitd:x:998:997:User for polkitd:/:/sbin/nologin
colord:x:997:994:User for colord:/var/lib/colord:/sbin/nologin
unbound:x:996:993:Unbound DNS resolver:/etc/unbound:/sbin/nologin
gluster:x:995:992:GlusterFS daemons:/run/gluster:/sbin/nologin
libstoragemgmt:x:994:991:daemon account for libstoragemgmt:/var/run/lsm:/sbin/nologin
saslauth:x:993:76:Saslauthd user:/run/saslauthd:/sbin/nologin
chrony:x:992:987::/var/lib/chrony:/sbin/nologin
geoclue:x:991:985:User for geoclue:/var/lib/geoclue:/sbin/nologin
cockpit-ws:x:990:984:User for cockpit-ws:/:/sbin/nologin
sssd:x:989:983:User for sssd:/:/sbin/nologin
dirsrv:x:988:982:user for 389-ds-base:/usr/share/dirsrv:/sbin/nologin
setroubleshoot:x:987:981::/var/lib/setroubleshoot:/sbin/nologin
saned:x:986:980:SANE scanner daemon user:/usr/share/sane:/sbin/nologin
gnome-initial-setup:x:985:979::/run/gnome-initial-setup/:/sbin/nologin
pcp:x:984:978:Performance Co-Pilot:/var/lib/pcp:/sbin/nologin
kdcproxy:x:983:977:IPA KDC Proxy User:/:/sbin/nologin
ipaapi:x:982:976:IPA Framework User:/:/sbin/nologin
dovenull:x:981:975:Dovecot's unauthorized user:/usr/libexec/dovecot:/sbin/nologin
dovecot:x:97:97:Dovecot IMAP server:/usr/libexec/dovecot:/sbin/nologin
hsqldb:x:96:96::/var/lib/hsqldb:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologi
 ... omit...

Sort Column 3 of/etc/passwd and output to px. txt


[root@1centos ~]# sort -t: -k 3 /etc/passwd -o px.txt
[root@1centos ~]# cat px.txt 
root:x:0:0:root:/root:/bin/bash
xnftp:x:1007:1007::/home/xnftp:/sbin/nologin
vuser:x:1008:1008::/opt/vuser:/sbin/nologin
tom:x:1009:1009::/home/tom:/bin/bash
jerry:x:1010:1010::/home/jerry:/bin/bash
kongkong:x:1011:1011::/home/kongkong:/bin/bash
qemu:x:107:107:qemu user:/:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
usbmuxd:x:113:113:usbmuxd user:/:/sbin/nologin
bin:x:1:1:bin:/bin:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin

uniq Tools

The Linux uniq command is used in conjunction with the sort command to check and delete repeated columns and columns in text files.

uniq Common Options

uniq checks for recurring columns and columns in text files
-c or count displays the number of repetitions of the row next to each column.
-d or repeated displays only recurring rows and columns.
-f < Field > Or skip-fields= < Field > Ignore the field specified by comparison.
-s < Character position > Or skip-chars= < Character position > Ignore the characters specified by comparison.
-u or unique displays rows and columns only once.
-w < Character position > Or check-chars= < Character position > Specifies the characters to compare.
help displays help.
version displays version information.
[Input File] Specifies a sorted text file. If this item is not specified, the data is read from the standard;
[Output File] Specifies the file to output. If this option is not specified, the content is displayed to the standard output device (display terminal)

uniq Tool Sample

View the file the. txt


[root@1centos zhengzebiaodashi]# cat the.txt 
1the 2the 3the
1the 2the 3the
1the 2the 3the
1the 2the 3the

1the 2the 3the
2the 2the 3the
3the 2the 3the 4the 5the
4hello hi the word world
5 2 3 4 5 6 7 8

Default sort


[root@1centos zhengzebiaodashi]# uniq the.txt
1the 2the 3the

1the 2the 3the
2the 2the 3the
3the 2the 3the 4the 5the
4hello hi the word world
5 2 3 4 5 6 7 8

Delete duplicate lines and display the number of repetitions


[root@1centos zhengzebiaodashi]# uniq -c the.txt 
   4 1the 2the 3the
   2 
   1 1the 2the 3the
   1 2the 2the 3the
   1 3the 2the 3the 4the 5the
   1 4hello hi the word world
   1 5 2 3 4 5 6 7 8

Find duplicate lines in an testfile file


[root@1centos zhengzebiaodashi]# uniq -d the.txt 
1the 2the 3the

tr Tools

tr is the abbreviation of translate, which is used for translation or conversion. Specifically, it can convert or delete the input content (stdin). It is a necessary tool for linux pipeline. Here are a few common uses

tr Common Options

-c: Replaces all characters that do not belong to Character Set 1;
-d: Delete all characters belonging to Character Set 1;
-s: Represents consecutive repeated characters as a single character;
-t: Delete more characters from Character Set 1 than Character Set 2 first

The tr tool example works with echo to change the lowercase letters displayed to uppercase


[root@1centos zhengzebiaodashi]# echo "jb51" |tr 'a-z' 'A-Z'
JB51

Replace the repeated characters in the output


[root@1centos zhengzebiaodashi]# echo 'Thisssssss is cdsnnn' |tr -s 'sn'
This is cdsn

Delete dictation characters from strings


[root@1centos zhengzebiaodashi]# echo 'this is csdn' |tr -d 'th'
is is csdn

Array sorting

With these tools, you can simply ascend or descend arrays


[root@1centos ~]# sort -t: -rk 3 /etc/passwd
nobody:x:99:99:Nobody:/:/sbin/nologin
ods:x:999:999:softhsm private keys owner:/var/lib/softhsm:/sbin/nologin
polkitd:x:998:997:User for polkitd:/:/sbin/nologin
colord:x:997:994:User for colord:/var/lib/colord:/sbin/nologin
unbound:x:996:993:Unbound DNS resolver:/etc/unbound:/sbin/nologin
gluster:x:995:992:GlusterFS daemons:/run/gluster:/sbin/nologin
libstoragemgmt:x:994:991:daemon account for libstoragemgmt:/var/run/lsm:/sbin/nologin
saslauth:x:993:76:Saslauthd user:/run/saslauthd:/sbin/nologin
chrony:x:992:987::/var/lib/chrony:/sbin/nologin
geoclue:x:991:985:User for geoclue:/var/lib/geoclue:/sbin/nologin
cockpit-ws:x:990:984:User for cockpit-ws:/:/sbin/nologin
sssd:x:989:983:User for sssd:/:/sbin/nologin
dirsrv:x:988:982:user for 389-ds-base:/usr/share/dirsrv:/sbin/nologin
setroubleshoot:x:987:981::/var/lib/setroubleshoot:/sbin/nologin
saned:x:986:980:SANE scanner daemon user:/usr/share/sane:/sbin/nologin
gnome-initial-setup:x:985:979::/run/gnome-initial-setup/:/sbin/nologin
pcp:x:984:978:Performance Co-Pilot:/var/lib/pcp:/sbin/nologin
kdcproxy:x:983:977:IPA KDC Proxy User:/:/sbin/nologin
ipaapi:x:982:976:IPA Framework User:/:/sbin/nologin
dovenull:x:981:975:Dovecot's unauthorized user:/usr/libexec/dovecot:/sbin/nologin
dovecot:x:97:97:Dovecot IMAP server:/usr/libexec/dovecot:/sbin/nologin
hsqldb:x:96:96::/var/lib/hsqldb:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologi
 ... omit...

Make use of


[root@1centos ~]# sort -t: -rk 3 /etc/passwd
nobody:x:99:99:Nobody:/:/sbin/nologin
ods:x:999:999:softhsm private keys owner:/var/lib/softhsm:/sbin/nologin
polkitd:x:998:997:User for polkitd:/:/sbin/nologin
colord:x:997:994:User for colord:/var/lib/colord:/sbin/nologin
unbound:x:996:993:Unbound DNS resolver:/etc/unbound:/sbin/nologin
gluster:x:995:992:GlusterFS daemons:/run/gluster:/sbin/nologin
libstoragemgmt:x:994:991:daemon account for libstoragemgmt:/var/run/lsm:/sbin/nologin
saslauth:x:993:76:Saslauthd user:/run/saslauthd:/sbin/nologin
chrony:x:992:987::/var/lib/chrony:/sbin/nologin
geoclue:x:991:985:User for geoclue:/var/lib/geoclue:/sbin/nologin
cockpit-ws:x:990:984:User for cockpit-ws:/:/sbin/nologin
sssd:x:989:983:User for sssd:/:/sbin/nologin
dirsrv:x:988:982:user for 389-ds-base:/usr/share/dirsrv:/sbin/nologin
setroubleshoot:x:987:981::/var/lib/setroubleshoot:/sbin/nologin
saned:x:986:980:SANE scanner daemon user:/usr/share/sane:/sbin/nologin
gnome-initial-setup:x:985:979::/run/gnome-initial-setup/:/sbin/nologin
pcp:x:984:978:Performance Co-Pilot:/var/lib/pcp:/sbin/nologin
kdcproxy:x:983:977:IPA KDC Proxy User:/:/sbin/nologin
ipaapi:x:982:976:IPA Framework User:/:/sbin/nologin
dovenull:x:981:975:Dovecot's unauthorized user:/usr/libexec/dovecot:/sbin/nologin
dovecot:x:97:97:Dovecot IMAP server:/usr/libexec/dovecot:/sbin/nologin
hsqldb:x:96:96::/var/lib/hsqldb:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologi
 ... omit...