Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Linux for Power Users

Setup environment

Setup

  • Start by setting up the Virtual environment!

Introduction

Content

  • Understanding the Unix/Linux philosophy.
  • Basic use of the GUI to get started.
  • The shell - the command line interface.

What is Linux?

  • kernel

  • The kernel.

  • The distribution.

Tux, the Linux mascot:

Vendors and Distributions

Linux History

  • 1983 September 27 - GNU announced by Richard Stallman (emacs, gcc).
  • 1985 - FSF - Free Software Foundation.
  • 1991 September 17 Linus Torvalds releases v0.01 of the Linux kernel.
  • 1991-1992 - Early distributions history
  • 1994-1995 Main stream distributions appear.
  • 2008 October 22 - First Android based mobile phone released.

GNU

  • GNU - GNU is Not Unix
  • GNU/Linux

The Gnu mascot:

The GNU mascot

License - Free and Open Source

  • GPL 2
  • GPL 3
  • LGPL
  • ...
  • Apache
  • BSD

Free Software

Free as in Free speach or freedom.

  1. Run.
  2. Study (source code).
  3. Modify.
  4. Redistribute.

Free Software Definition

Unix/Linux Design Principles

  • Hierarchical filesystem with a single root /.
  • Eveything is a file.
  • Small single-purpose programs.
  • Pipe programs together to accomplish more complex tasks.
  • All configuration is stored as text. ASCII or UTF-8.

Unix vs Linux?

  • HPUX, AIX, Irix, Solaris, etc.
  • FreeBSD, NetBSD, OpenBSD
  • OSX
  • GNU/Linux

Resources and Getting Help

Linux in Israel

Installation

Exercise: Install Linux

GUI

Desktop GUIs

GUI Tasks

  • Login - Just type in the username and password provided during installation.
  • Learn about Unity
  • Launch applications
  • Install Software - The Linux appstore
  • Change password
  • Add another user
  • Switch user
  • File explorer
  • Text editor (Gedit)
  • Add language support
  • Set background
  • Install software not in the Appstore (Chrome?)
  • The terminal
  • Logout

Exercise: GUI

  • Create a new user.
  • Open Firefox.
  • Install some application from the "appstore".
  • Install Nginx and browse to http://127.0.0.1/ internally.
  • Check out the file system explorer.

Terminal

The powers of the shell

  • TAB completition

  • Command history

  • Command chaining (redirection, pipe)

  • Wildcards (Meta Characters, globbing)

  • Aliases.

  • Scripting - Programmability.

  • Flexibility.

  • Using all ten fingers.

TAB completition

$ ls /...
$ apt-get install pyth...

Command History

  • Up and Down arrows
  • history

Chaining commands: Quick Apache/Nginx log analyzer

cut -f1 -d ' ' examples/apache_access.log | sort | uniq -c | sort -k1 -n | tail -1
  • Take the Apache access log.
  • Get the first field of every line where the fields are separated by spaces. (These are the IP addresses of the visitors.)
  • Sort them.
  • Drop duplicates prefixing each value by the number of occurances.
  • Sort the new double column by the numeric value of the first field.

Remove temp files

ls -l /tmp/ | grep "Jul  3" | cut -d ' ' -f 22 | xargs -I{} ls -ld /tmp/{} | less
ls -l /tmp/ | grep "Jul  3" | cut -d ' ' -f 22 | xargs -I{} rm -rf /tmp/{}
  • List all the things in the /tmp directory
  • Filter those that were created on "Jul 3".
  • Limit the output to the filenames only.
  • Execute rm -rf on all of them.

Shells

echo

  • echo
  • -n

echo prints its parameters to the screen (standard output) follwed by a newline. -n supresses the additional newline. (Mostly interesting for scripts).

echo hello world
echo "hello world"
echo -n "hello world"

Actions on ENTER

  • Meta-character expansion.
  • Variable substitution.
  • Splitting of commands.
  • Setup redirection.
  • Command execution

Actions on ENTER - examples

  • Meta-character expansion. **echo ***
  • Variable substitution. echo "Hello $USER"
  • Splitting of commands. echo "Hello"; echo "World"
  • Setup redirection. echo "Hello"; echo "World" > /dev/null
  • Command execution

Meta-characters (Wildcards)

  • ?

  • []

  • {}

  • ? match any single character

    • match any number of characters (0 or more) (except /)

  • [abC] character class - matches a, b, or C

  • [b-e] character class - matches b, c, d, or e

  • [!b-e] character class - matches any single character except b, c, d, or e

  • name_{x,y,z} - name_x, name_y, name_z all at the same time

Can be used with 'ls' and other commands on the command line.

Information about the shell

  • $HOME
  • $0
  • $BASH_VERSION
  • $version
  • $SHELL
echo $HOME
echo $0              Current shell.
echo $BASH_VERSION
echo $version        In csh like shells such as tcsh.
echo $SHELL          The default shell of the user.

Change password using passwd

  • passwd
$ passwd
Old Password:
New Password:
Retype Password:

Getting Help

  • -h
  • --help
$ passwd -h          # get some help
$ passwd --help      # get some help

$ ls -h              # show how sizes are d
$ ls --help          # some help

$ echo -h            # just prints -h
$ echo --help        # just prints --help

Documentation: man

  • man
  • man passwd
  • man ls
  • man echo
  • man man
  • man -h

Keyboard

space  - next page
enter  - next line
b      - prev page
q      - quit
h      - help
q      - quit from the help
1G     - jump to the first line

/      - search
n      - next hit
N      - previos hit

whatis and man sections

  • whatis
$ whatis passwd
passwd (1)           - change user password
passwd (1ssl)        - compute password hashes
passwd (5)           - the password file

$ man 5 passwd
$ man 1ssl passwd

Apropos

  • apropos

man -k

$ apropos passwd
chgpasswd (8)        - update group passwords in batch mode
chpasswd (8)         - update passwords in batch mode
Crypt::SmbHash (3pm) - Perl-only implementation of lanman ...
fgetpwent_r (3)      - get passwd file entry reentrantly
getpwent_r (3)       - get passwd file entry reentrantly
gpasswd (1)          - administer /etc/group and /etc/gshadow
grub-mkpasswd-pbkdf2 (1) - generate hashed password for GRUB
htpasswd (1)         - Manage user files for basic authentication
pam_localuser (8)    - require users to be listed in /etc/passwd
passwd (1)           - change user password
passwd (1ssl)        - compute password hashes
passwd (5)           - the password file
passwd2des (3)       - RFS password encryption
SSL_CTX_set_default_passwd_cb (3ssl) - set passwd callback for encrypted PEM file handling
SSL_CTX_set_default_passwd_cb_userdata (3ssl) - set passwd callback for encrypted PEM file handling
update-passwd (8)    - safely update /etc/passwd, /etc/shadow and /etc/group

Sections of the man pages

See also man man

  1. User command (Programs)
  2. system calls
  3. Library calls
  4. Special files (devices)
  5. File formats and configuration files
  6. Games
  7. Overview, conventions, and miscellaneous
  8. System management commands

Command history

  • history
  • HISTFILE
  • !
  • Ctrl-r

Commands executed in the shell are saved in a file. By default in ~/.bash_history but the filename is controllable by the HISTFILE environment variable. HISTFILESIZE can be used to configure the number of lines saved in that file. HISTSIZE can be used to control the number if lines save in the memory during a session (before saving it to the file) The actual file is written when the shell exits. If there are multiple shells open, each will have its own history in memory and each will write it to the file, the second exiting will overwrite the file written by the first one. Use the history command to list the content of history buffer. Use ! N to run command N, or ! -N or ! name.

  • Up/Down arrows
  • history
  • !! - the last comand
  • ! N - command number N
  • Ctrl-r - reverse-i-search
  • man bash to see even more options.

which and !!

$ which adduser
$ ll `!!`

List all commands in history

List all the commands in the shell history and show them in abc order.

$ history | cut -c 8- | sort | less

Clear screen

  • clear
  • Ctrl-l

Virtualbox: Network access resources

Virtualbox: Allow access from Host to Guest

  • Virtual Box / Preferences / Network / Host only network
    • to add one (vboxnet0)
  • Select your VirtualBox / Setup / Network
  • Adapter 1 is set to NAT.
  • Click to Adapter 2
  • Set it to "Host only" and selectec the vboxnet0

Virtualbox: Set up host-only network on Ubuntu

  ifconfig -a
  sudo apt-get install ifupdown
  sudo apt-get install ssh
  edit /etc/network/interfaces  and add
       auto enp0s8
       iface enp0s8 inet static
       address 192.168.56.10
       netmask 255.255.255.0

Run sudo ifup enp0s8

Install ssh server

  • sshd
$ sudo apt-get install ssh
$ ssh localhost
$ ifconfig | grep inet

Set up ssh access

  • Enable and set up ssh access to your virual environment.
  • Set up the X server on your desktop.
  • In /etc/ssh/sshd_config make sure you have X11Forwarding yes
  • ssh -X IP

Exercise: Terminal

  • Change your password.
  • Launch Firefox.
  • Print the path to the home directory.
  • Flip through the documentation of man.

Exercise: commands

Try the following commands and try to understand that they do:

cal
cal 2000
cal 1 2000
cat /etc/passwd
curl http://google.com/
date
hostname
id
last
top       (q to quit)
uptime
who
whoami
history

Navigating the File System

File system types

  • MS-DOS (FAT32)

  • HPFS

  • NTFS

  • ...

  • ext3

  • ext4

  • ResierFS

  • ...

Linux file system structure

/
  bin/
  boot/
  dev/
  etc/
  home/
    foo/
    bar/
  lib/
  lost+found/
  media/
  mnt/
  opt/
  proc/
  root/
  sbin/
  tmp/
  usr/
  var/

Some parts might be mounted from other disk or even from other computers. This is transparent for us.

Show directory tree

tree
tree -L 1 /
tree -L 2 -d /

Current Working directory

pwd        print working directory

Directory listing - ls

  • ls
ls        "default" listing
ls /bin

ls -1     one-column listing
ls -l     long listing with file-types and rights
ls -a     all, including hidden files
ls -A     all, except . and ..

ls -l
ls -ld    the directory
ls -ltr   long, sort by time, reverse
ls -i     show inodes

File types

      • Regular files
  • d - Directories
  • l - Symbolic links
  • c - Character special files (e.g. tty: /dev/tty )
  • b - Block special files (eg. disks: /dev/sda )
  • s - Sockets
  • p - Named pipe

Hidden files

  • .

Files that the name start with a dot. By accident.

Change Directory

cd         cd to home
cd ~       cd to home
cd ~joe    cd to the home of joe
cd ..      cd to parent directory
cd -       cd back to previous directory

Display disk usage statistics

$ du
344   ./build/html/tucs/css
48    ./build/html/tucs/js
1520  ./build/html
400   ./build/linux/examples
696   ./build/linux
2288  ./build
272   ./examples/script
424   ./examples
2840  .

$ du -h      (human-readable)
172K    ./build/html/tucs/css
 24K    ./build/html/tucs/js
760K    ./build/html
200K    ./build/linux/examples
348K    ./build/linux
1.1M    ./build
136K    ./examples/script
212K    ./examples
1.4M    .

du -s

$ du -hs *          (human-readable, summarize)
4.0K    README
1.1M    build
212K    examples
 24K    intro.xml
4.0K    linux.yml
 16K    scripts.xml
4.0K    variables.xml

$ du -s * | sort -n -r
2288    build
424     examples
48      intro.xml
32      scripts.xml
8       variables.xml
8       linux.yml
8       README

Display free disk space using df

  • df
$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sda1       47931368  18992880  26480628  42% /
none                   4         0         4   0% /sys/fs/cgroup
udev             2009680         4   2009676   1% /dev
tmpfs             404732      1276    403456   1% /run
none                5120         0      5120   0% /run/lock
none             2023660       144   2023516   1% /run/shm
none              102400        28    102372   1% /run/user
/dev/sda5      413296896 371702128  20577416  95% /home

Disk layout

$ df
Filesystem     1K-blocks      Used     Available    Use%   Mounted on
/dev/sda1     40,720,960   5,282,552   35,422,024   13%    /

38 Gb

  • Partitions
  • Block size (1K)
  • Inode table and Inodes
$ df -i
Filesystem      Inodes      IUsed       IFree   IUse%   Mounted on
/dev/sda1      4,807,680   196,498   4,611,182    5%    /

Free Disk space with human readable numbers

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        46G   19G   26G  42% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            2.0G  4.0K  2.0G   1% /dev
tmpfs           396M  1.3M  394M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G  144K  2.0G   1% /run/shm
none            100M   28K  100M   1% /run/user
/dev/sda5       395G  355G   20G  95% /home

Show inodes stats: df -hi

$ df -hi
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda1        3.0M  587K  2.4M   20% /
none             495K     2  495K    1% /sys/fs/cgroup
udev             491K   533  491K    1% /dev
tmpfs            495K   555  494K    1% /run
none             495K     3  495K    1% /run/lock
none             495K     4  495K    1% /run/shm
none             495K    14  495K    1% /run/user
/dev/sda5         26M  737K   25M    3% /home

mounting external disks

  • mount

  • umount

  • NFS - Network File System

  • Samba (SMB) - Microsoft Network

  • mount, umount, /etc/fstab

Exercise: File system

  • Create a file called this is a long file with some spaces.txt
  • Rename it to not so long.txt

Exercise: Explore /proc

The /proc directory has a lot file-looking entitities that hold a lot of information about the system. Explore them and share your findings with us. For examply try to run cat /proc/meminfo.

Manipulating Files and Directories

Creating directory

  • mkdir
$ mkdir abc
$ mkdir abc/def


$ mkdir x/y/z
mkdir: cannot create directory 'x/y/z': No such file or directory

$ mkdir -p x/y/z

Create a file using 'touch'

  • touch
$ touch out.txt
$ ls -l out.txt
-rw-rw-r-- 1 vagrant vagrant 0 Jul 17 10:30 out.txt

Create file using echo

  • echo
echo "Hello World" > out.txt
echo "Hello Again" >> out.txt

Copy files and directories

  • cp
cp a.txt b.txt
cp -r x/ y/         will create y and copy all the files of x to y
cp -r x/ y/         will create y/x and copy all the files of x into y/x

cp -i               will ask for confirmation before overwriting files

cp a.txt b.txt      sets current date
cp -p a.txt c.txt   preserves date

Move files or directories

  • mv
$ mv a.txt b.txt
$ mv a.txt b/
$ mv a.txt b.txt c/
$ mv a/ b/

$ mv -i       to prompt before overwriting

Removing File or Directory

  • rmdir
  • rm
$ rm a.txt           remove file

$ rmdir d/           remove empty(!) directory
$ rm -rf d/          force remove directory tree recursively

Symbolic links

  • ln -s

  • Avialable vs enabled configuration files of Apache or Nginx.

  • Backward compatibility: Retain historical location when moving a file or directory.

  • Fake directory structure (or location of files).

$ ln -s  /tmp
$ ls -l

$ ln -s /etc/passwd
$ ls -l

$ echo "hello" > a.txt
$ ln -s a.txt b.txt
$ ls -l

Hard links

  • ln
$ echo "hello" > a.txt
$ ln a.txt b.txt
$ ls -li

Determining File types

  • file

  • By extension?

  • Using 'file'

$ file /usr/bin/python
/usr/bin/python: symbolic link to `python2.7'

$ file /usr/bin/python2.7
/usr/bin/python2.7: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.32,
BuildID[sha1]=2ca3b615bd7d6141844b6701002b80f148b80d40, stripped

$ file .viminfo
.viminfo: ASCII text

$ file a.sh
a.sh: Bourne-Again shell script, ASCII text executable

$ cp a.sh a
$ file a
a: Bourne-Again shell script, ASCII text executable

Finding files (find)

  • find
find .

find . -name \*.xml

-name NAME         wilde card for name of the thing
-iname NAME        like name but ignores case
-user  NAME        things owned by the user
-type [fdlcb]      f = file, d = directory, ...
-size [+/-]n[ck]   min or max size
-inum number       inode numbe of the thing (to find hard links)
-mtime [+/-]n      modified time in days.

Actions:

-exec command {} \;
-ok command {} \;

A little warning. If we run find . -name *.xml (without escaping the wildcard), the shell will try to expand it in the current working directory. If it is successful that result will be used as the search term.

find examples

find . -type d
find / -name .profile
find / -type f -size +1k

find / -mtime 10            modified 10 days ago
find / -mtime +10           modified more than 10 days ago
find / -mtime -10           modified less than 10 days ago

find / -mtime -2       = the things found by    -mtime 0   and  -mtime 1

exec, ok, and xargs

find . -ok echo {} \;
find . -exec echo {} \;
find . | xargs echo
find . -print0 | xargs -0 -I{} echo {}

Remove some files

find . -name "*.o" -ok rm {} \;
find . /tmp /usr/tmp -name core -exec rm {} \;

Rename files without extension to have extension jpg ?

find . -type f ! -name '*.*' -exec mv {} {}.jpg \;
find . -type f ! -name '*.*' -print0 | xargs -0 -I{} mv {} {}.jpg

Search indexed files

  • hostname
$ locate ...

$ updatedb

Comparing files using diff

Hello World
More text
Third line

Hello World
Other line

diff a.txt b.txt
2,3c2
< More text
< Third line
---
> Other line

Comparing directories using diff

diff -r a/ b/

File and Directory Name conventions

Might depend on the file system.

  • Case sensitive.
  • Can have spaces.
  • Can have Unicode characters.
  • Extensions have no special meaning to Linux, but might have special meaning to some applications.
  • Usually all lowercase, no spaces, words separated by underscore _.

Exercise: files

  • Create a directory structure under ~/ex/a with 3 directories and 5 files
  • Copy the whoke structure to ~/ex/b
  • Compare the two using 'diff'
  • Add a new file in the ~/ex/a structure.
  • Change the content of one of the files in the ~/ex/b structure.
  • Compare the two directory trees again.

Exercise: shutdown vs halt

What is the difference between shutdown and halt.

File security - File system rights management

ls -l

drwxrwxr-x  8 vagrant vagrant 4096 Mar  7 20:08 Abc
-rwxrwxr-x  1 vagrant vagrant  476 Jun  6 15:39 a.sh
lrwxrwxrwx  1 vagrant vagrant    4 Jul  3 15:39 b -> a.sh
  • 1 char for type
  • 3 chars for user (u)
  • 3 chars for group members (g)
  • 3 chars for others (o)

Type

- is file
d is directory
l is symbolic link

Rights

r - Read
w - Write
x - eXecute

chmod

  • chmod
$ touch some_file
$ ls -l some_file
-rw-rw-r-- 1 vagrant vagrant 0 Jul  7 14:58 some_file

$ chmod 0 some_file
---------- 1 vagrant vagrant 0 Jul  7 14:58 some_file

$ chmod u+rw some_file
-rw------- 1 vagrant vagrant 0 Jul  7 14:58 some_file

$ chmod a+x some_file
-rwx--x--x 1 vagrant vagrant 0 Jul  7 14:58 some_file

$ chmod o-x some_file
-rwx--x--- 1 vagrant vagrant 0 Jul  7 14:58 some_file

chmod octal

$ chmod 0753 some_file
-rwxr-x-wx 1 vagrant vagrant 0 Jul  7 14:58 some_file

4  - r
2  - w
1  - x

     rwx
7  = 111
5  = 101
3  = 011

Change ownership with chown

  • chown
$ chown foo.marketing file.txt

$ chown foo:marketing file.txt

Change group only using chgrp

  • chgrp

Change the group of a file or a directory.

$ chgrp marketing file.txt

Default file permissions: umask

  • umask

By default files would be created with permission 0666 which means rw for everyone.

umask helps having more secure defaults.

The bits that are set in umask won't be set by the applications. So if we have umask 0777 then every file will be created without any rights. If the umask is 0000 then it has no effect and files will be created with 0666.

$ umask    to show the current value
$ touch file1
$ ls -l file1

$ umask 0022
$ touch file2
$ ls -l file2

Shell Script

  • sh-bang
  • chmod +x

$ chmod u+x hello_world.sh
$ ./hello_world.sh

Perl

#!/usr/bin/env perl

print "Hello World\n";

$ perl hello_world.pl
$ chmod u+x hello_world.pl
$ ./hello_world.pl

Python

#!/usr/bin/env python

print "Hello World"

$ python hello_world.py
$ chmod u+x hello_world.py
$ ./hello_world.py

Exercise: Create Bash script

  • Create a shell script that will print "Hello Linux".
  • Try to run it.
  • Make it executable and try to run it again.

Exercise: File security

  • Create a file.
  • Change its rights.
  • Change its owner.
  • ...

Exercise: Remove write protected file

  1. Create a file that has no rights at all. Try to remove it.

  2. Create a directory and in it create a file that has no rights.

  3. Remove the write rights from the directory.

  4. Try to remove the file.

  5. Create a directory and in it create a file.

  6. Remove the eXecutable bit from the directory.

  7. Try to list the content of the directory.

Exercise: chmod

  • Create a file and set the permissions to -r--r--r--

  • Then set it to -rwx-------

  • Create a second user.

  • Create a file by your regular user and make sure the other user cannot see the content.

  • Create another file that the user can read but cannot write.

  • Create a Hello World shell script. Make sure both of you can run it but only you can write it.

  • Create a 3rd user. Make sure only the first 2 users can execute the script.

Solution: chmod

  • Create a second user sudo adduser foo
  • Create the file and make it executable by owner and by group chmod ug+x hello.sh
  • Create a new group: sudo groupadd exe
  • Add my user (vagrant) to this group: sudo usermod -a -G exe vagrant
  • Add the second user (foo) to the group: sudo usermod -a -G exe foo
  • Change the group of the file to be the new common group: sudo chgrp exe hello.sh

File content

Text editors

  • vim
  • emacs
  • gedit - notepad like editor on the desktop
  • pico - notepad like editor on the command line
  • nano -

nano - a small editor

nano file.txt

Displaying file content

  • cat - show the content of the whole file.
  • more- an old pager (not really used any more).
  • less- a new pager (h for help; q for quit).
  • head- show first 10 lines of a file.
  • head -42 - show first 42 lines of a file.
  • tail - show last 10 lines.
  • tail -42 - show last 42 lines.
  • tail -f - follow as the file growth.

cut

Extract fields or charcters from strings.

cut -c 3,2,10    files (characters)
cut -f 1,3       files (fields separated by tab)
cut -f 1,3 -d' ' files (fields separated by space)
cut -f 1,3 -d':' files (fields separated by :)

cut -b   files   (bytes)
  • -c N charcter
  • -f N field
  • -d ':' delimeter (defaults to TAB)
$ who
foo      console  Apr  8 18:57
bar      ttys000  Apr  9 07:42
moose    ttys001  Apr  8 18:57

$ who | cut -f1 -d' '
foo
bar
moose

sort

$ cat  examples/intro/names.txt
Foo 11
Bar 2
Moo 1

$ sort  examples/intro/names.txt
Bar 2
Foo 11
Moo 1

$ sort -k2 examples/intro/names.txt
Moo 1
Foo 11
Bar 2

$ sort -k2 -n examples/intro/names.txt
Moo 1
Bar 2
Foo 11

-k -select field to use for sorting -n -numerical sorting

uniq

Despite its name, uniq will not necessarly provide a unique list of values. It will collapse the adjacent equal values. If all the equal values are adjacent then it will really create a unique list. Sorting will reorder the original list so that the equal values will be next to each other.

$ cat examples/intro/words.txt 
foo
foo
bar
foo
bar

$ uniq examples/intro/words.txt 
foo
bar
foo
bar

$ sort examples/intro/words.txt | uniq
bar
foo

Finding text in a file

  • grep

  • egrep

  • ack-grep

  • grep regex FILEs

  • egrep regex FILEs

  • ack-grep

-v lines that did not match
-i ignore case
-n show line numbers
-r look into the files recursively

-w match exact words only
-l print names of the files matched
-c count of lines matched

-B n   show n lines before the match.
-A n   show n lines after the match.
-C n   show n lines both before and after the match (up to 2*n+1 lines) (context).

$ cat examples/intro/words.txt 
foo
foo
bar
foo
bar

$ grep bar examples/intro/words.txt 
bar
bar

$ grep -v bar examples/intro/words.txt 
foo
foo
foo

$ grep -n bar examples/intro/words.txt 
3:bar
5:bar

$ grep -n -v bar examples/intro/words.txt 
1:foo
2:foo
4:foo

word count with wc

  • passwd
$ wc README
     220     516    3270 README
$ wc *.xml
     750    2020   19797 intro.xml
     593    1568   14953 scripts.xml
     145     366    3338 variables.xml
    1488    3954   38088 total

$ wc -l README 
     220 README
$ wc -w README 
     516 README
$ wc -c README 
    3270 README

Counting lines, words, characters

tr - transcribe

  • tr
$ cat examples/intro/words.txt
foo
foo
bar
foo
bar


$ cat examples/intro/words.txt | tr 'f' 'x' 
xoo
xoo
bar
xoo
bar


$ cat examples/intro/words.txt | tr '[a-z]' '[A-Z]' 
FOO
FOO
BAR
FOO
BAR

sed - the stream editor

  • sed
sed 's/old/new/' < ORIGINAL_FILE > NEW_FILE
sed 's/old/new/' ORIGINAL_FILE > NEW_FILE
cat ORIGINAL_FILE | sed 's/old/new/' > NEW_FILE

s/REGEX/STRING/
s:REGEX:STRING:

s/c./(&)/     & represent the whole match

echo fool | sed -r 's/(.)/(&)/'           (f)ool
echo fool | sed -r 's/(.)\1/(&)/'         f(oo)l
echo fool | sed -r 's/(.)\1/(double \1)/'     f(double o)l

Sed tutorial

Compression and archiving

  • tar

  • gzip

  • gunzip

  • zip

  • unzip

  • bzip2

  • bunzip2

  • zcat

  • zgrep

  • xz

  • tar

  • zip unzip

  • gzip gunzip zcat zgrep

  • bzip2 bunzip2 bzcat bzgrep

  • xz unxz xzcat xzgrep

tar xzf file.tar.gz
tar czf file.tar.gz  dir/

Generate random output

  • $RANDOM

A Bash script that will generate a random number. If it is bigger that 10 then print the number.

#!/bin/bash

v=$RANDOM
if [ "$v" -gt 10 ]
then
    echo $v
fi

Generate random log

Change the script to print one of these every tenth of a second.

#!/bin/bash

while true
do 
    v=$RANDOM
    if [ "$v" -gt 16000 ]
    then
        echo $v Error
    else
        echo $v Good
    fi
    sleep 0.1
done


$ ./generate_random_log.sh

Pressing Ctrl-C will stop the execution.

Exercise: log file

  • Run the script in a way that only the lines with "Error" in them will show.
  • Then run the script and redirect the output to a file.
  • Run it in the background.
  • Display the content of the log-file as it is growing.
  • Now display only the lines with the word "Error"

Exercise: File content

  1. grep through /etc/password showing the line of your username
  2. Then show the shell of every user in the system.
  3. Count how many times each shell is used.
  4. Count how many users are in the system.
  5. Archive the content of your home directory in a tar.gz and then also ina tar.bz2 file.
  6. How do they compare in size to each other and to the real content of your home directory?
  7. Find your name in the zipped file without opening it to the disk.
  8. Open the whole directory tree in /tmp.
  9. Remove a file from that new tree and compare it to original directory tree.

Exercise: File content round 2

Use the following script to create a subdirectory called 'file_conent' and in it a file called 'data.txt'. Once we have the file, replace the numbers with their duplication so 1 will become 11.

#!/bin/bash

DIR='file_content'
FILE=$DIR/data.txt

if [ -e $DIR ]
then
   echo "$DIR already exists"
   exit
fi

echo "Creating directory $DIR"
mkdir $DIR

cat<<TEXT > $FILE
Hello World
This is the second line
1 hundred 2 hunderd 3 hundred
TEXT

Redirection and Pipes

Open Channels

  • STDIN (keyboard) 0
  • STDOUT (screen) 1
  • STDERR (screen) 2

Redirection

  • >
  • >>
  • <
  • 2>
  • 2>&1
command >    output_file   (overwrite)
command >>   output_file   (append)
command <    input_file
command 2>   error_file    (overwrite)
command 2>&1 merged_file   (both output and error)

command > output_file  2> error_file < input_file

Redirection bash

$ ls -l /bin/bash /bing/other
ls: /bing/other: No such file or directory
-r-xr-xr-x  1 root  wheel  1228240 Oct 29 13:31 /bin/bash


$ ls -l /bin/bash /bing/other > out
ls: /bing/other: No such file or directory


$ ls -l /bin/bash /bing/other 1> out
ls: /bing/other: No such file or directory


$ ls -l /bin/bash /bing/other 2> err
-r-xr-xr-x  1 root  wheel  1228240 Oct 29 13:31 /bin/bash


$ ls -l /bin/bash /bing/other > out  2> err


$ ls -l /bin/bash /bing/other 2> /dev/null
-r-xr-xr-x  1 root  wheel  1228240 Oct 29 13:31 /bin/bash

Redirection bash append

$ ls -l /bin/bash /bing/other >> out 
ls: /bing/other: No such file or directory



$ cat out 
-r-xr-xr-x  1 root  wheel  1228240 Oct 29 13:31 /bin/bash
-r-xr-xr-x  1 root  wheel  1228240 Oct 29 13:31 /bin/bash

Redirection bash merge

$ ls -l /bin/bash /bing/other >& out
(both stdout and stderr go to the same channel)

$ ls -l /bin/bash /bing/other &> out
(does the same)

Redirection tcsh

$ ls -l /bin/bash /bing/other > out
ls: /bing/other: No such file or directory


$ ls -l /bin/bash /bing/other  > & out
(combine stdout and stderr)


$ (ls -l /bin/bash /bing/other  > out) > & err
(direct stdout to 'out' and stderr to 'err')

tcsh cannot directly redirect stderr but we can use a subshell with the parentheses.

Redirect to /dev/null

  • /dev/null
find /  2> /dev/null

Pipes

  • |
$ ls -l examples/intro/
total 32
-rw-r--r--  1 gabor  staff  282 Apr 11 08:37 pipes.txt
-rw-r--r--  1 gabor  staff   15 Mar  2 15:19 redirection.txt
-rw-r--r--  1 gabor  staff  896 Apr 11 08:19 redirection_bash.txt
-rw-r--r--  1 gabor  staff  238 Apr 11 08:20 redirection_tcsh.txt

$ ls -l examples/intro/ | wc
       5      38     257

Filters

A filter is a command that can either act on a filename it sees on the command line or on the stream on STDIN.

$ filter INPUT_FILE
$ filter &lt; INPUT_FILE
$ cat INPUT_FILE | filter
  • grep
  • cut
  • uniq
  • sort
  • ...
  • sed
  • awk
  • perl

tee

Both redirects the output to a file and let's it through to the screen or to a pipe. tee -a would append to the file.

$ ls -l examples/intro/ | tee out | wc
       5      38     257

$ cat out
total 32
-rw-r--r--  1 gabor  staff  336 Apr 11 08:39 pipes.txt
-rw-r--r--  1 gabor  staff   15 Mar  2 15:19 redirection.txt
-rw-r--r--  1 gabor  staff  896 Apr 11 08:19 redirection_bash.txt
-rw-r--r--  1 gabor  staff  238 Apr 11 08:20 redirection_tcsh.txt

Exercises: redirection

  • How many files are in the current directory?
  • How many files are in the current directory tree?
  • Given the Apache log file in examples/apache_access.log
  • How many hits came from 127.0.0.1 ?
  • How many hits came from other than 127.0.0.1 ?

Exercise: HTML content

  • Find a package that can strip the html tags from an html file and install it.
  • If you found the HTML::Strip perl module, then the script bellow will do the work using that module.
  • Save the home page of IMDB, strip the html and then count how many numbers are in the file.
#!/usr/bin/env perl
use strict;
use warnings;

use Getopt::Long qw(GetOptions);
use HTML::Strip;

GetOptions('help' => \&usage) or usage();

if (@ARGV) {
	foreach my $file (@ARGV) {
		my $content;
		if (open my $fh, '<', $file) {
			local $/ = undef;
			$content = <$fh>;
		} else {
			warn "Could not open '$file'";
			next;
		}
		if ($content) {
			strip($content);
		}
	}
} else {
	my $content = join '', <STDIN>;
	strip($content);
}

sub strip {
	my ($raw_html) = @_;
	my $hs = HTML::Strip->new();
	my $clean_text = $hs->parse( $raw_html );
	$hs->eof;
	print $clean_text;
}

sub usage {
	print <<"USAGE";
Usage:
    $0 filename
    cat file | $0
USAGE
	exit;
}

Commands and aliases

which

  • which

which shows the path to the given command(s)

$ which grep
/usr/bin/grep

$ which echo
/bin/echo

$ which perl
/home/vagrant/localperl/bin/perl

$ which ls
/bin/ls

$ which ll
      (nothing printed)

type

  • type

type tells us if the command is aliased or not. an external executable.

$ type grep
grep is hashed (/usr/bin/grep)

$ type echo
echo is a shell builtin

$ type perl
perl is /home/vagrant/localperl/bin/perl

$ type ls
ls is aliased to `ls --color=auto'

$ type ll
ll is aliased to `ls -alF'

whereis

Show all the places where the given command can be found in the PATH

$ whereis perl
perl: /usr/bin/perl /etc/perl /usr/share/perl
  /home/vagrant/localperl/bin/perl /usr/share/man/man1/perl.1.gz

alias

$ alias
alias cp='cp -i'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias grep='grep --color=auto'
alias l='ls -CF'
alias la='ls -A'
alias ll='ls -alF'
alias ls='ls --color=auto'
alias mv='mv -i'
alias rm='rm -i'

Creating alias in bash

  • alias
  • unalias
alias rm='rm -i'
alias cp='cp -i'
alias ll='ls -lA'

unalias ll

Creating alias in tcsh

  • alias
alias rm 'rm -i'
alias cp 'cp -i'
alias ll 'ls -lA'

unalias ll

Suppress alias

alias rm='rm -fr'     # very dangerous!

'rm' thing
\rm  thing

Alias only works in the shell it has been defined in

Aliases only work in the same shell where they were defined. Not even in a subshell we get when we run a script. (But if we sourced the script then it runs in the same shell.)

$ alias tweet=echo
$ tweet hello
hello

#!/bin/sh
tweet "Hello"

$ ./examples/linux/tweet.sh 
./examples/linux/tweet.sh: 2: ./examples/linux/tweet.sh: tweet: not found

Exercises: commands and aliases

  • List all the aliases in your shell

  • Add an alias 'hw' that will print 'hello world' to the screen

  • Check what is the 'hw' command?

  • Remove the 'hw' alias.

  • Check that it has been removed.

  • Observe what 'ls' and 'ls -l' do and what would be the output of 'ls' if it was not already an alias.

  • Create an alias 'll' to do what 'ls -l' does.

Shell

cal - calendar

$ cal
$ cal 2015
$ cal 1 2015

bc - calculator

$ bc
3/2
1
Ctrl-d

bc -l
3/2
1.50000000000000
Ctrl-d

time commands

time find .

Will do all the work find does and at the end it will print how long did it take to run the given command.

real    0m0.757s
user    0m0.002s
sys     0m0.005s

Separating commands

cd /home; pwd

Commands are executed in sequence, from left to right.

Combining commands

  • The commnad perldoc -lm Module::Name will print the full path to the source code of a Perl Module.
  • We can take that path and open with vim using: vim perldoc -lm Module::Name.
  • Maybe better this way: vim $(perldoc -lm Module::Name).

Bash functions

Creating an alias for this might be difficult because of the parameter passing but we can create a function.

Add the following to ~/.bashrc and then type source ~/.bashrc

Then we can write epm Module::Name.

function epm(){
    vim $(perldoc -lm $*);
}

Execute shell file

  • source
  • .
  • ./
./script_name         Spawn new shell
script_name           Spawn new shell

source script_name    Run in current shell
. script_name         Run in current shell

Grouping commands

$ ls -1 build/; ls -1 examples/ | wc
html
linux
linux_v1.00.tar.gz
linux_v1.00_linux.zip
linux_v1.00_windows.zip
       1       1       6

$ (ls -1 build/; ls -1 examples/) | wc
       6       6      82

Without the parantheses, the pipe is only applied to the output of the second command. With the paranetheses, the output of both ls commands are combined and the pipe is applied to that. A subshell is spawned to execute the commands.

$ pwd
/home/foobar/training/linux

$ (cd ~; pwd)
/home/foobar

$ pwd
/home/foobar/training/linux

Quotes

  • '

  • "

  • \

  • `

  • $()

  • '' - no special meaning for any character except '

  • "" - no special meaning for any character except ", $, \ and `

  • \ - escape next character (disable special meaning of next character)

  • `` - (backtick) command substitution

  • $() - command subsitutions.

$ echo 'hello "world"'
hello "world"

$ echo "hello $0"
hello -bash
$ echo 'hello $0'
hello $0

$ echo "hello \"world\""
hello "world"

$ echo "date: `date`"
date: Fri Apr 11 10:42:21 IDT 2014

$ echo "date: $(date)"
date: Fri Apr 11 10:45:44 IDT 2014

Spell checking

  • ispell
  • aspell

hostname

  • hostname
  • /etc/hostname
$ cat /etc/hostname

$ hostname

date

  • date
$ date
Wed Jul  6 08:53:10 UTC 2016

$ date +"%d %H:%M"
06 08:55

$ TZ=EST date +"%d %H:%M"
06 03:56

First day of the month

#!/bin/bash

if [ $(date +%d) = "01" ]
then
  echo "hit"
else
  echo "nope"
fi

exit and exit code

  • exit
  • $?
  • Ctrl-D
$?     - contains the most recent exit code in the parent shell

Typing 'exit' in a login shell will log out the user. If it is typed in a terminal, that terminal will be closed. Ctrl-D has the same effect in a terminal.

locale

$ locale

/etc/default/locale

Setting the prompt

  • PS1
echo $PS1

PS1='$(hostname) $'

Secondary prompt

  • PS2
ls \
&gt;

PS2=':)'

set -o

set -o     listing the status

set -o vi
set -o emacs

Start up files (Login shells)

  • profile
  • .bash_profile
/etc/profile     - systemwide startup file for every user, only after login
/etc/profile.d/*
~.profile

~/.bash_profile  - user specific startup file, only after login
~/.bash_login

Start up file (Noninteractive shells too)

  • .bashrc
/etc/bashrc  - all users, every new shell and subshell

~/.bashrc -  user specific, every new shell and subshell

Startup files

PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin
EDITOR=/usr/bin/vim
PS1='\h:\W \u\$'

alias rm='rm -i'

logout script

  • .bash_logout
~/.bash_logout  - when exiting the shell

Exercise: Prompt

  • Set the prompt to be the current hour (or the number of seconds till the end of the day).
  • Include the current username in the prompt.
  • Include the current directory in the prompt.
  • Any other ideas?

System information

id

  • id
$ id
uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant)

who

  • who

Which user(s) are logged in to the system. Since when. From which IP address.

$ who
vagrant  pts/0        2016-07-03 11:12 (10.0.2.2)

w

  • w

Which user(s) are logged in to the system. What are they doing?

$ w
 19:16:16 up 1 day,  3:17,  2 users,  load average: 0.27, 0.23, 0.17
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
vagrant  pts/1    10.0.2.2:S.0     19:15    1.00s  0.03s  0.00s w
vagrant  pts/2    10.0.2.2:S.1     19:16    9.00s  0.06s  0.03s vim a.txt

whoami

  • whoami

Print the current user. (same as id -un)

$ whoami
vagrant

last

  • last

Show listing of last logged in users

bar       ttys002                   Wed Jul  6 16:55 - 17:12  (00:16)
foo       ttys002                   Wed Jul  6 13:11 - 13:16  (00:05)
gabor     ttys001                   Mon Jul  4 19:48   still logged in

uname

  • uname
uname
Linux

uname -a
  CPU type - Kernel Architecure - operating system - kernel version
Linux pm 3.11.0-17-generic #31-Ubuntu SMP Mon Feb 3 21:52:43 UTC 2014
  x86_64 x86_64 x86_64 GNU/Linux
  • CPU type
  • Application architecture
  • Kernel Architecture

free

  • free

Show free memory.

$ free -ht
             total       used       free     shared    buffers     cached
Mem:          3.9G       3.8G        32M        96K        97M       1.1G
-/+ buffers/cache:       2.6G       1.2G
Swap:         1.0G        61M       962M
Total:        4.9G       3.9G       995M
  • total - physical memory
  • cached - memory used to speed up operations. e.g. file read-write operations. This can be used by an application.
  • buffers - Temporary memory to help some processes. (e.g. file meta data).
  • In the first line used includes buffers + cached.
  • In the second line used only includes memory used by applications.

vmstat

  • vmstat

{% embed include file="src/examples/linux/vmstat.txt)

/proc/meminfo

  • /proc/meminfo
MemTotal:        1024832 kB
MemFree:          161916 kB
MemAvailable:     457904 kB
Buffers:           40488 kB
Cached:           359556 kB
SwapCached:            0 kB
Active:           624124 kB
Inactive:         135448 kB
Active(anon):     365820 kB
Inactive(anon):    17360 kB
Active(file):     258304 kB
Inactive(file):   118088 kB
Unevictable:        4160 kB
Mlocked:            4160 kB
HighTotal:        135112 kB
HighFree:          14720 kB
LowTotal:         889720 kB
LowFree:          147196 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                80 kB
Writeback:             0 kB
AnonPages:        363744 kB
Mapped:           108148 kB
Shmem:             20916 kB
Slab:              85756 kB
SReclaimable:      67732 kB
SUnreclaim:        18024 kB
KernelStack:        1744 kB
PageTables:         3220 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      512416 kB
Committed_AS:    1378612 kB
VmallocTotal:     122880 kB
VmallocUsed:        6544 kB
VmallocChunk:     114404 kB
AnonHugePages:    247808 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       40952 kB
DirectMap2M:      872448 kB

Exercise: System information

  • Check the free memory on your system right after booting.
  • Launch a heavy application that takes a lot of memory in one window.
  • Check the free memory again in another window.
  • Change the prompt to include the current time in seconds.
  • Make the change persistant between terminals and sessions.

System administration basics

Basic sysadmin tasks

  • reboot
  • shutdown
  • halt
  • Package Management - Installing software

sudo

  • sudo
$ ls -l /root
ls: cannot open directory /root/: Permission denied

$ sudo ls -l /root/
[sudo] password for gabor:
total 4
drwxr-xr-x 3 root root 4096 Apr 28  2012 SecretRootFile

su

  • su
$ su -

Logfiles in /var/log

  • /var/log
$ sudo ls -l /var/log

Exercise: sysadmin

  • List the files in the /root directory
  • Reboot the Linux machine from the command line.
  • Once it rebooted log in and shut it down from the command line.
  • What is the differernce between ls -l and ls -L. Create an example.

Package Management - Installing software

App-store

  • /etc/apt/sources.list

Package formats

  • RPM (Red Hat Package Management)
  • DEB (Debian, Ubuntu, etc.
  • Source code

Package managemnt tools

  • search
  • install
  • uninstall
  • upgrade
  • Which package contains a given file?
  • List dependencies

Instalation with apt-get

  • apt-get
# apt-get update
# apt-get install package-name

apt-cache

  • apt-cache
$ apt-cache search PARTIAL-PACKAGE-NAME
$ apt-cache show PACKAGE-NAME
$ apt-cache showpkg PACKAGE-NAME
$ apt-cache depends PACKAGE-NAME

aptitude

  • aptitude
# aptitude update
# aptitude safe-upgrade                   (updating fixes)
$ aptitude search PARTIAL-PACKAGE-NAME
# aptitude 

# aptitude upgrade

dpkg

  • dpkg
dpkg --list package_name - list installed packages

yum on CentOS 7.2 x64

  • yum
$ yum search STRING    - search for packages matching that string
$ yum install NAME     - install given package and its dependencies
$ yum -y install NAME  - don't wait for confirmation
$ yum list installed NAME - check if package installed 

$ yum check-update     - check which packages have newer versions
$ yum update           - update packages (install newer versions)

$ yum whoprovides  /path/to/file  - list of package(s) that installed the given file

$ yum deplist perl-Test-Perl-Critic     - list of dependencies

CentOS repoquery

$ yum -y install yum-utils

$ repoquery --requires NAME                         - list of (immediate) dependencies
$ repoquery --requires --resolve NAME               - list of (immediate) dependencies - packages
$ repoquery --requires --resolve --recursive NAME   - list of all the dependencies

yes

  • yes
# aptitude safe-upgrade
# yes | aptitude safe-upgrade


# aptitude -y safe-upgrade

Exercise: Package management

  • Install a package (e.g. a web server or an editor)

Process and Job Management

uptime

  • uptime
$ uptime
11:04:46 up 13:25,  1 user,  load average: 0.13, 0.17, 0.15

ps

  • ps
  • pstree
$ ps         (PID, TTY, TIME, CMD)
$ ps -f      full format (UID, PPID, STIME)
$ pe -e      All processes

$ pstree
$ pstree -p       Show the IDs as well

System load: top

  • top

System load: htop

  • htop

  • Tree view / List view

  • u - user

  • F - follow

Process state

  • R - Running or runnable
  • S - Sleep - waiting for event to complete
  • T - Stopped (Ctrl-Z) or traced
  • D - Uninterruptable sleep (IO)
  • Z - Zombie (defunct) terminated but not reaped

Process control

  • kill
$ kill       SIGTERM   (same as kill -15)  = please die
$ kill -9    SIGIKILL                      = die now!
$ kill -l    list available signals

$ man 7 signal

Job control

  • jobs
  • fg
  • bg
  • Ctrl-C
  • Ctrl-Z

"jobs" are processes related to the current shell.

  • Run this command: (while [ 1 ]; do (date; sleep 1); done)

  • suspend job: Ctrl-Z SIGTSTP (same as kill -20) = stop for now

  • bg [job] will move the specific (or current) job to the background.

  • fg [job] will move the specific (or current) job to the foreground.

  • command & run in the background.

  • jobs will show all the background jobs

  • %number - reference to a job number

  • %string - job whose name begins with string

  • %?string - job whose name contains string

  • %+ or %% - current job

  • %- - previous job

  • Ctrl-C SIGINT (same as kill -2) = please die

  • kill [-signal] job will send a signal to the given process.

Refuse to die

#!/usr/bin/perl
use strict;
use warnings;

print "Process ID: $$\n";

$SIG{INT} = sub { print "\nINT - I don't want to die!!!\n" };
$SIG{TERM} = sub { print "\nTERM - Don't terminate me!!!\n" };

$SIG{TSTP} = sub { print "\nTTP - No background check for me!!!\n" };

sleep 1 while 1;
  • Run the above program
  • Press Ctrl-C
  • from another window run kill PID
  • from another window run kill -15 PID
  • from another window run kill -2 PID
  • from another window run kill -9 PID
  • Run it again
  • Ctrl-Z
  • kill -20 POD

killall

  • killall

killall NAME Kill all the process by the given name

$ perl -e 'sleep 1 while 1' &amp;
$ perl -e 'sleep 1 while 1' &amp;
$ perl -e 'sleep 1 while 1' &amp;
$ ps -ef | grep perl
$ killall perl
$ ps -ef | grep perl

Zombie demonstration

In one window launch htop and filter to 'perl'. In another window run this script.

#!/usr/bin/perl
use strict;
use warnings;

print "parent $$\n";
my $pid = fork();
die "Could not fork $!\n" if not defined $pid;

if (not $pid) {
    print "in child: $$\n";
    exit;
}

print "In parent $$ child: $pid\n";
sleep 1;
system "ps -lef | grep perl";
sleep 1000;

Reaping child processes demonstration

#!/usr/bin/perl
use strict;
use warnings;

print "parent $$\n";
my $pid = fork();
die "Could not fork $!\n" if not defined $pid;

if (not $pid) {
    print "in child: $$\n";
    exit;
}

print "In parent $$ child: $pid\n";
sleep 1;
wait();
system "ps -lef | grep perl";
sleep 10;

Stopped process demonstration

  • In one window run htop and filter to 'perl'.
  • In the second window run perl -e '<STDIN>'
  • Look at the htop display to see the process appearing with a state S
  • Press Ctrl-z in the window with the process.
  • Look at the htop display to see the process appearing with a state T
  • bg will move the process to the background but the state will stay in T.
  • fg will move it to the foreground and the state will change to S.
  • ENTER will finish the process.
  • Do the same with this example: perl -e 'sleep 1 while 1'
  • In this case moving to the background will already move the process to S state.

List open files (lsof)

  • lsof

In one window run this perl script as user 'foo': perl examples/linux/open_file.pl README. In another window

$ lsof | head -1                          (to print the header)
$ sudo lsof | grep README                 (to print the line with the details)
COMMAND   PID   TID USER  FD   TYPE DEVICE SIZE/OFF   NODE NAME
perl    32242        foo   3r   REG   0,41      5950  61972 /vagrant/training/linux/README

$ sudo ls -l /proc/32242
$ sudo less /proc/32242/cmdline

Current working directory of a process

$ sudo pwdx PID

Services

  • service
$ sudo service stop
$ sudo service start
$ sudo service restart
$ sudo service reload

tail multiple log files

$ (while [ 1 ]; do (echo -n "one "; date; sleep 1); done) &gt; one.txt &amp;
$ (while [ 1 ]; do (echo -n "two "; date; sleep 1); done) &gt; two.txt &amp;

$ tail -f one.txt -f two.txt
$ multitail one.txt two.txt

Network Basics

IPv4 vs IPv6

inet addr:10.0.2.15

inet6 addr: fe80::a00:27ff:fefd:1f04/64

ifconfig

  • ifconfig
 ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:fd:1f:04
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fefd:1f04/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11894 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8887 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2523362 (2.5 MB)  TX bytes:790877 (790.8 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:10183 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10183 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2904229 (2.9 MB)  TX bytes:2904229 (2.9 MB)

lxcbr0    Link encap:Ethernet  HWaddr 86:d0:36:c7:2e:e8
          inet addr:10.0.3.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::84d0:36ff:fec7:2ee8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:570 (570.0 B)

Ping

  • ping
$ ping google.com
PING google.com (172.217.19.238) 56(84) bytes of data.
64 bytes from s22-in-f14.1e.net (172.217.19.238): icmp_seq=1 ttl=63 time=93.5 ms
64 bytes from s22-in-f14.1e.net (172.217.19.238): icmp_seq=2 ttl=63 time=200 ms
64 bytes from s22-in-f14.1e.net (172.217.19.238): icmp_seq=3 ttl=63 time=84.4 ms
64 bytes from s22-in-f14.1e.net (172.217.19.238): icmp_seq=4 ttl=63 time=95.9 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3009ms
rtt min/avg/max/mdev = 84.425/118.594/200.469/47.466 ms

Traceroute

  • traceroute
  • tracepath
$ tracepath google.com
 1?: [LOCALHOST]                                         pmtu 1500
 1:  10.0.2.2                                              0.571ms
 1:  10.0.2.2                                              0.175ms
 2:  192.168.1.1                                           3.164ms asymm 64
 3:  192.168.0.1                                           1.809ms asymm 63
 4:  bzq-25-85-30.cust.bezeqint.net                       76.260ms asymm 62
 5:  bzq-219-189-245.cablep.bezeqint.net                  22.917ms asymm 61
 6:  bzq-179-124-78.cust.bezeqint.net                     83.441ms asymm 60
 7:  bzq-179-72-241.cust.bezeqint.net                     95.980ms asymm 59
 8:  no replyre
...

/etc/hosts

Map names to IP addresses locally. Override DNS.

Telnet

  • telnet

ssh - Secure Shell

  • ssh
$ ssh hostname
$ ssh user@hostname
$ ssh -p port hostname

$ ssh hostname "command; command; ...."

Public key access

$ ssh foo@localhost
...
foo@localhost's password:

Look at ~/.ssh

ls -l ~/.ssh
    authorized_keys
    known_hosts

Generate public key

  • ssh-keygen
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/vagrant/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/vagrant/.ssh/id_rsa.
Your public key has been saved in /home/vagrant/.ssh/id_rsa.pub.
The key fingerprint is:
bd:67:29:a2:92:db:7f:70:4b:00:8b:d0:36:2e:19:5a vagrant@perl-maven
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|                 |
|                 |
|         .       |
|o E o +    . .   |
|+ o +.o. . +     |
| .  .ooo.        |
+-----------------+

.ssh/id_rsa
.ssh/id_rsa.pub

Add public key

$ scp .ssh/id_rsa.pub foo@localhost:

$ ssh foo@localhost

# if .ssh does not exist yet:

$ mkdir .ssh
$ ls -ld .ssh
drwxrwxr-x 2 foo foo 4096 Jul 17 05:51 .ssh/
$ chmod 0700 .ssh/


$ cat id_rsa.pub >> .ssh/authorized_keys
$ rm id_rsa.pub

ftp - File Transfer Protocol

  • ftp

scp - Secure Copy

  • scp
$ scp localfile user@remote:/path/to/
$ scp -r  localdir/   user@remote:/dir

rsync

  • rsync
rsync source hostname:destination

 --archive
 --verbose
 --recursive
 --progress N
 --delete-after
 --rsh="ssh -p port -l user"

curl and wget to fetch web pages

  • curl
  • wget
$ curl http://google.com/
$ wget http://google.com/
$ cat index.html

Command line browsers

  • lynx
  • w3m
  • elinks

Sending mail from the command line

  • mail
  • mail foo@bar.com < file

Keep remote session using screen

  • screen
$ ssh remote
$ screen
  ... do stuff ...
  get disconnected

$ ssh remote
$ screen -dr
  ... back where you were ...

  • screen - to start a new session.

  • screen -dr - to attach to an existing session.

  • Ctrl-a Ctrl-d - to detach from an existing session.

  • Ctrl-a Ctrl-c Create new Window

  • Ctrl-a w List Windows

  • Ctrl-a 0 Jump to Window 0

  • Ctrl-a ? Help

  • Ctrl-d Close current window (close screen)

Keep remote session using tmux

  • tmux

  • tmux - launch a new session

  • tmux ls - list sessions

  • Ctrl-b s - list sessions

  • Ctrl-b d - detach from current session.

  • tmux a - attache to running session.

  • Ctrl-b ? - list of key bindings

tmux multiple session

$tmux                    - create new session (name 0)
$ echo "session 0"
$ Ctrl-b d               - detach

$ tmux                   - create new session (name 1)
$ echo "session 1"
$ Ctrl-b d               - detach

$ tmux new -s sample     - new session (name sample)
$ echo sample
$ Ctrl-b d               - detach

$ tmux ls
0: 1 windows (created Wed Jul 20 11:51:58 2016) [101x50]
1: 1 windows (created Wed Jul 20 11:52:02 2016) [101x50]
sample: 1 windows (created Wed Jul 20 11:54:53 2016) [101x50]

$ tmux a -t 1            - attach to session 1
$ Ctrl-b d
$ tmux a -t sample       - attach to session sample

$ Ctrl-b s               - list sessions and select another one
$ Ctrl-b $               - rename current session

$ tmux kill-session -t session-name     - or just exit the window with Ctrl-d

iftop

  • iftop
$ sudo iftop
  • Right hand side: average traffic in the last 2, 10, 40 seconds
  • h - to get help
  • p - to show ports
  • n - toggle DNS host resolution
  • L - liner/logarithmic scale
  • s - show/hide source IP
  • S - show/hide source port
  • d - show/hide destination IP
  • D - show/hide destination port

Port used by (lsof - list open files)

  • lsof -i N
$ sudo lsof -i :22     list the process using port 22

Port used by example 2

  • nc
$ nc -l localhost 10000 > log.txt
$ telnet localhost 10000
$ sudo lsof -i :10000

$ sudo cat /proc/PID/cmdline

Network ports

How to identify used and free ports?

$ sudo lsof -i         list all the processes using some port

/proc/PID/exe          is a symlink to the real executable
/proc/PID/cmdline      is the content of the command line
/proc/PID/cwd          current working directory of the process

Show ports used with netstat

  • netstat
$ sudo netstat -tulpn
  • sudo - regular user won't see everything
  • -t show tcp
  • -u show udp
  • -l show only listening sockets
  • -p show both PID and the name of the program
  • -n show numberic IP and port

Routing configuration

  • route
  • netstat
$ netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG        0 0          0 eth0
10.0.2.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.0.3.0        0.0.0.0         255.255.255.0   U         0 0          0 lxcbr0

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.2.2        0.0.0.0         UG    0      0        0 eth0
10.0.2.0        *               255.255.255.0   U     0      0        0 eth0
10.0.3.0        *               255.255.255.0   U     0      0        0 lxcbr0

write

  • write
  • Ctrl-D

Send messages to other users on the system by typing

$ write username
type your message
here
Ctrl-D

Chat with users using ytalk

$ who
$ ytalk user#tty

Will split the window to 2 parts to talk to each other.

Exercise: Networking

  • Try pinging and tracerouting some other servers. Including code-maven.com
  • Try to access some of the servers in your own company using ping/traceroute/telnet/ssh
  • Download the main web page of your company and look at its source.
  • Fetch the headers the web site returns. (Read about it in the man of curl.)

Exercise: lorem ipsum

  • Download the main page of lorem ipsum
  • How many lines are there in total?
  • How many lines contain the word "lorem" ?
  • How many lines in the first half of the page have the word "lorem"?

Scheduling processes

crontab

  • crontab
$ crontab -e       (edit)
*  *  *  *  * date >> ~/out.txt

man 5 crontab

crontab -l         (list)

User specific file:

/var/spool/cron/crontabs/USERNAME

System-wide crontab files:

/etc/cron.d/
/etc/cron.daily/
/etc/cron.hourly/
/etc/cron.monthly/
/etc/crontab
/etc/cron.weekly/

Load crontab file

$ crontab ~/cron.txt

crontab examples

MAILTO=foobar@example.com

# minute                         (0-59)
#     hour                       (0-23)
#         day of month           (1-31)
#             month              (1-12)
#                 day of week    (0-7)
# m   h   dom mon dow            command

# Every minute:
#*  *  *  *  * date >> ~/out.txt

# Every hour in the 42nd minute
# 42 *  *  *  * date >> ~/out.txt

# at 1:30 and 13:30
# 30 1,13  *  *  * (cd ~/bin ; ./monitor.pl --run --email)

# Every 2 minutes
#*/2  *  *  *  * date >> ~/out.txt


# The minutes 57,58,59 every hour
# 57-59  *  *  *  * date >> ~/out.txt

# Every 3rd minute between 7-29 minutes of every hour
# 7-29/3  *  *  *  * date >> ~/out.txt

# First day of every month at 0:42 am
# 42  0  1  *  * date >> ~/out.txt

# Every Sunday at 0:42 am
# 42  0  *  *  0 date >> ~/out.txt

Schedule using at or batch

  • at

  • atq

  • atrm

  • at - Schedule a job at a specific time.

  • batch - execute when system load is low.

Boot

Boot stages

  1. BIOS - Basic Input/Output System of the hardware
  2. boot loader
  3. The Linux kernel.
  4. The init process.

BIOS

  • Hardware specific.
  • Checks hardware extensions.
  • Configurable: (e.g. the boot device)
  • F12 ???
  • Executes the code in the Master Boot Record (MBR) of the selected device.

Boot loader

  • Located in the Master Boot Record (MBR) of the HD/CD/disk on key.
  • LILO - Linux Loader (old)
  • GRUB - Grand Universal Bootloader (modern)

Grub

  • /etc/grub.conf
  • Allows the user to select which kernel to load or which options to pass
  • Single user mode!

init

  • /sbin/init
  • initrd

Multiple OS-es

  • Dual boot.
  • Host/Guest using Virtualbox or other simple virtulatization
  • virtualization - slicing of the resources.

NFS - Network Filesystem

Fileserver

  • NFS Daemon - mountd
  • nfsd
  • rquotad - quotas on mounted filesystem
  • lockd
  • statd
  • /etc/fstab
  • mount

Variables

Legal variables

  • Variable names must consist of letters, digits, and the underscore
  • The first character must be a letter or an underscore
  • Any length
  • All the existing variables in the current shell are called the "environment" and the are called "environment variables"
  • Usually predefined (or system) variables are all upper case letters.
  • Some manuals call these variables "Named parameters".

Setting and getting variables

  • $

  • Assignment using = without spaces on either side of =

  • All the variables are strings. If there are spaces in the value, the whole value mus be in quotes

  • When using a variable we must prefix it with a $ sign


name=foobar
echo $name

occupation=shell programmer
echo $occupation

echo "Hello $name"
echo $nameit
echo ${name}it

$ ./examples/script/variables.sh
foobar
./examples/script/variables.sh: line 5: programmer: command not found

Hello foobar

foobarit

There are empty lines printing the content of 'occupation' and that of the variable 'nameit'.

Backtick or back quote

  • `
$ who
gabor    console  Apr 21 10:06 
gabor    ttys000  Apr 21 10:06 
gabor    ttys001  Apr 21 10:06 

$ who | wc
       3      15      96

$ who | wc -l
       3
$ v=`who | wc -l`

$ echo $v
3

Command substitution

  • $()

Bash provides alternative syntax for command substitution. $().

Exporting variables

  • export

The following command may be used to examine and remove things from the environment. env, set, unset, export

If we set a variable using name=foobar this variable won't be visible in scripts we launch because those scripts run in a subshell and the variable is local.

If we would like to make sure the variable is visible in the subshell we have to export the variable.

$ name=foobar
$ echo $name
foobar
$ ./show_name.sh 

$ export name=FooBar
$ echo $name
FooBar
$ ./show_name.sh 
FooBar

echo $name

env

  • env

Shows all the exported variables.

set

  • set

set - shows all the variables (exported and local)

unset

unset remove variable

  • unset}
$ echo $username

$ username="foo bar"
$ echo $username
foo bar

$ unset username
$ echo $username

Predefined variables

DISPLAY
HISTFILE
HISTSIZE
HOME
LANGUAGE
MANPATH
OLDPWD
PATH
SHELL
TERM
USER

Change PATH

$ echo $PATH
...

$ export PATH=/home/foobar/bin:$PATH

PATH environment variable

  • PATH
  • set
  • export
  • which

Setting the PATH environment variable

each $PATH
export PATH=/some/new/path:$PATH
set
which command

Bash script

sh-bang

  • #!
  • echo

A bash script is just a text file with commands in it. Command like the ones we can give on the command line. It does not have to have any particular extension, but often people use the .sh extension. For example foobar.sh

In order to be able to run it we need to make it executable chmod u+x foobar.sh or to make it exacutable for all the users on the system use chmod +x foobar.sh.

To ensure the script always executes as a bash script, and not as the "current shell of the user" we need to add ad a sh-bang line.

#!/bin/bash

echo Hello
echo World


$ ./examples/script/hello_world.sh
Hello
World

echo Hello World

$ ./examples/script/hello_world_more.sh
Hello World

echo -n Hello
echo World


$ ./examples/script/hello_world_n.sh
HelloWorld

echo -n "Hello "
echo World


$ ./examples/script/hello_world_space.sh
Hello World

Comments in a Shell script

echo Hello   # this is a comment
# this is too
echo World

$ ./examples/script/comments.sh
Hello
World

cd in subshell

#!/bin/bash

pwd
cd /etc
pwd

When a shell script is executed ./foobar.sh it is executed in a subshell. This means certain changes made in the script don't apply in the original (parent) shell. For example if we change the current working directory using cd, when the script ends, we are back in the same directory where we started.

$ pwd
/home/gabor

$ ./examples/script/change_dir.sh 
/home/gabor
/etc

$ pwd
/home/gabor

Set a variable in a subshell

#!/bin/bash

echo $TERM
TERM=xterm
echo $TERM


$ echo $TERM
xterm-256color

$ ./examples/script/change_term.sh 
xterm-256color
xterm

$ echo $TERM
xterm-256color

. (dot command)

  • source
  • .

The source and the . command will execute the script in the current shell. Which means the above changes are now persistant even after the script finishes running.

$ pwd
/home/gabor
$ . examples/script/change_dir.sh 
/home/gabor
/etc
$ pwd
/etc

$ echo $TERM
xterm-256color
$ . examples/script/change_term.sh 
xterm-256color
xterm
$ echo $TERM
xterm

Input

  • Command line parameters in $1, $2, ..., $#, $*, $@, shift
  • Read from STDIN (read)

Command line parameters

  • $0
  • $1

The shell puts the space separated values from the command line in variables $0 .. $9 where $0 is the name of the current script. Can handle up to 9 parameters (excluding the script itself)

$10 would be $1 followed by a 0, but we can write ${10} in our script which will be the 10th command line parameter.

echo 0: $0
echo 1: $1
echo 2: $2


$ ./examples/script/command_line.sh Foo Bar wins
0: ./examples/script/command_line.sh
1: Foo
2: Bar

$ ./examples/script/command_line.sh "Foo Bar" wins
0: ./examples/script/command_line.sh
1: Foo Bar
2: wins

$ ./examples/script/command_line.sh Foo 
0: ./examples/script/command_line.sh
1: Foo
2:

Wildcards on the command line

If possible, wildcards are expanded before the script is executed:

$ ./examples/script/command_line.sh *
0: ./examples/script/command_line.sh
1: README
2: build

README and build are files in the current directory.

$ ./examples/script/command_line.sh *x
0: ./examples/script/command_line.sh
1: *x
2:

In this case the wildcard does not match anything and thus it is passed to the script as it is.

If you want to pass a parameter that contains a wildcard character, we need to quote the parameter or escape the wildcard. If we try this on the current script, we won't be able to really see the impact:

$ ./examples/script/command_line.sh '*'
0: ./examples/script/command_line.sh
1: README build examples intro.xml linux.yml scripts.xml variables.xml
2:
$ ./examples/script/command_line.sh \*
0: ./examples/script/command_line.sh
1: README build examples intro.xml linux.yml scripts.xml variables.xml
2:

That's because echo itself expands the wildcard when it prints *.

Command line and wildcard

A little script to show that escaping matters.

echo 0: $0
echo 1: $1
if [ "$1" = '*' ]
then
   echo 'star'
fi


$ ./examples/script/command_line_wild.sh 
0: ./examples/script/command_line_wild.sh
1:

$ ./examples/script/command_line_wild.sh *
0: ./examples/script/command_line_wild.sh
1: README

$ ./examples/script/command_line_wild.sh '*'
0: ./examples/script/command_line_wild.sh
1: README build examples intro.xml linux.yml scripts.xml variables.xml
star

$ ./examples/script/command_line_wild.sh "*"
0: ./examples/script/command_line_wild.sh
1: README build examples intro.xml linux.yml scripts.xml variables.xml
star

$ ./examples/script/command_line_wild.sh \*
0: ./examples/script/command_line_wild.sh
1: README build examples intro.xml linux.yml scripts.xml variables.xml
star

Command line $#, $*, $@

  • $#
  • $*
  • $@
echo $#
echo $@
echo $*

"$*" = "$1 $2 $3 $4 ... $n"
"$@" = "$1" "$2" "$3" "$4" ... "$n"

$ ./examples/script/command_line_params.sh foo bar
2
foo bar
foo bar

$ ./examples/script/command_line_params.sh "foo bar"
1
foo bar
foo bar

When put in double quotes "$*" and "$@" are slightly different. (See man bash)

shift

  • shift
echo $*
shift
echo $*
shift 2
echo $*



$ ./examples/script/shift.sh foo bar moo qux zorg
foo bar moo qux zorg
bar moo qux zorg
qux zorg

shift can move the content of $* to the left by N elements (defaults to 1) Mostly to avoid dealing with ${10}.

read

  • read
echo "What's your name?"
read name
echo Hello $name, how are you?

read one line from the standard input

$ ./examples/script/read.sh 
What's your name?
Foo
Hello Foo, how are you?

read}

echo "What's your name?"
read name
echo Hello $name, how are you?

read one line from the standard input

$ ./examples/script/read.sh 
What's your name?
Foo
Hello Foo, how are you?

prompte (read -p)

read -p "What's your name? " name
echo Hello $name, how are you?


$ ./examples/script/read_p.sh 
What's your name? Foo
Hello Foo, how are you?

Read several values from one line

read -p "Please enter two strings: " a b
echo You typed in $a and $b


$ ./examples/script/read_more.sh 
Please enter two strings: one two
You typed in one and two

Process ID

  • $$
  • $PPID
$$      - current process ID
$PPID   - process ID of the parent process

Background process ID $!

  • $!
$!   PID of the most recently started background process

$ perl -e 'print $$' &amp;
$ echo $!

Exit from a script

  • exit
  • $?

Calling exit will stop the running of the script and set the exit status to 0. Calling exit N with any number of N will set the exit status to that N.

exit   - on the command line to close the window (or the most recent shell)
exit   - in a script to end the script (exit code = most recent statement)
exit n - exit code n

Exit status in $?

  • $?

Every command has a so-called exit status. A number between 0-255. 0 indicates success. Other numbers indicate some failure. The exit status of the last command is always saved in $?

$ ls
README      build       examples
$ echo $?
0

$ ls /nosuch
ls: /nosuch: No such file or directory
$ echo $?
1

if

  • if
  • else
if [ $# = 2 ]
then
   echo there are 2 parameters
else
   echo The number of parameters is $#
fi

$ ./examples/script/testing_variables.sh 
The number of parameters is 0

$ ./examples/script/testing_variables.sh one
The number of parameters is 1

$ ./examples/script/testing_variables.sh one two
there are 2 parameters

$ ./examples/script/testing_variables.sh one two three
The number of parameters is 3

elif

  • elif
if [ $# = 2 ]
then
   echo there are 2 parameters
elif [ $# = 1 ]
then
   echo there is 1 parameter 
else
   echo The number of parameters is $#
fi

file tests

  • -e
  • -f
  • -d
  • -e
if [ $# != 1 ]
then
   echo "Usage $0 FILE"
   exit 1
fi

if [ ! -e $1 ]
then
    echo "$1 does NOT exist"
    exit
fi

if [ -f $1 ]
then
   echo $1 if a file
   exit
fi

if [ -d $1 ]
then
   echo $1 if a directory
   exit
fi


$ ./examples/script/file_tests.sh
Usage ./examples/script/file_tests.sh FILE

$ ./examples/script/file_tests.sh abc
abc does NOT exist

$ ./examples/script/file_tests.sh .
. if a directory

$ ./examples/script/file_tests.sh examples/script/file_tests.sh
examples/script/file_tests.sh if a file

The spaces inside the square brackets are significant.

-e    thing exists
-f    thing is a file
-d    thing is a directory
-L    thing is a symbolic link
-s    thing is not empty
...
-r    thing is readable by current user
-w    thing is writable by current user
-x    thing is executable by current user

File comparision tests

  • -nt
  • -ot
  • -ef
[ file1 -nt file1 ]   file1 is newer than file2
[ file1 -ot file2 ]   file1 is older than file2
[ file1 -ef file2 ]   file1 and file2 point to the same file (e.g. symbolic link)

Empty string

  • ""
#!/bin/bash

read -p "Please type in your name: " name
if [ "$name" = "" ]
then
    echo "Name was missing"
    exit 1
fi

$ ./examples/script/empty_string.sh 
Please type in your name: Foo

$ ./examples/script/empty_string.sh 
Please type in your name: 
Name was missing

Boolean tests

  • &&
  • ||
  • !
&&  - and
||  - or
!   - not
()  - for grouping

Boolean expressions

#!/bin/bash

[ -e "$some_file" ] && rm -f "$some_file"

[ -e "$some_file" ] || touch  "$some_file"

Testing strings - comparing their values

#!/bin/bash

s1="abc def"
s2="abc"
echo "s1=$s1"
echo "s2=$s2"

if [ "$s1"  = "$s2" ]; then echo "equal"; fi
if [ "$s1" != "$s2" ]; then echo "not equal"; fi
if [ "$s1" \< "$s2" ]; then echo "smaller in lexical order"; fi
if [ "$s1" \> "$s2" ]; then echo "larger in lexical order"; fi
if [ -z "$s1" ]; then echo "length of string is zero"; fi
if [ -n "$s1" ]; then echo "length of string is NOT zero"; fi

# variables should be enclosed in ""
# in order to avoid issues with spaces in the variables.

case

  • case
  • esac
#!/bin/bash

case "$1" in
   "foo")
        echo "it is foo";;
   [a-z][a-z][a-z])
        echo "3 letters";;
   [0-9][0-9])
        echo "2 digits";;
   42)
        echo "the answer";; 
   *)  
       echo "default";;
esac


$ ./examples/script/case.sh foo
it is foo

$ ./examples/script/case.sh abc
letters

$ ./examples/script/case.sh 123
digits

$ ./examples/script/case.sh 42
digits

$ ./examples/script/case.sh 
default

$ ./examples/script/case.sh abc123
letters
  • ;; - no subsequent cases are checked
  • ;& - execute the subsequent blocks until ;; is found or till the esac.
  • ;;& - check the subsequent condition till you find a match.

Debugging

  • -v
  • -x
bash -v script.sh   - echos commands before variable substitution
bash -x script.sh   - echos commands after varaiable substitution

while-loop

  • while
  • do
  • done
#!/bin/bash

secret="my password"
guess=""
while [ "$secret" != "$guess" ]
do
   read -p "What is your secret? " guess
done

echo "Congratulations, you passed the test"

for loop

  • for
for var [ in word... ]
do
   statements
done

#!/bin/bash

for x in 1 2 4 6
do
   echo $x
done

for file in *.xml
do
   echo $file
done

Iterate over commend line arguments

By omitting the list of values the for-loop needs to go over, it will default to the list of values supplied on the command line.

#!/bin/bash

for v
do
   echo $v
done

$ ./examples/script/for_argv.sh 
$ ./examples/script/for_argv.sh one two
one
two

break

  • break

break exits the smallest enclosing for, while, until or select loop break n exits the nth enclosing loop

continue

  • continue

Goes to the next iteration of the loop, skipping the rest of the code in the current iteration.

Testing commands

#!/bin/bash

if [ $# != 1 ]
then
    echo "Usage: $0 username"
    exit 1
fi

if grep "$1" /etc/passwd
then
    echo "Found $1 in /etc/passwd"
fi

$ ./examples/script/testing_command.sh Gate
_taskgated:*:13:13:Task Gate Daemon:/var/empty:/usr/bin/false
found Gate in /etc/passwd

We can check the exit value of a command directly as the command is executed. The problem is that in many cases we are not interested in the output of the command, just if it was successful or not. Redirection to /dev/null can solve the problem by hiding the extra output.

Testing commands while redirecting output

#!/bin/bash

if [ $# != 1 ]
then
    echo "Usage: $0 username"
    exit 1
fi

if grep "$1" /etc/passwd > /dev/null
then
    echo "Found $1 in /etc/passwd"
fi

Shell arithmetic

  • expr
$ expr 2 + 3
5


declare variable using 
declare -i variable_name
(typeset -i obsolete way?)

Opeartors
+
-
*
/
% modulo
&lt;&lt; - left shift
&gt;&gt; - right shift
&amp; | ^ ~  - bitwise and, or, not, one's complement

#!/bin/bash

declare -i a=2 
declare -i b=3
echo $a
echo $b

a=a+b
echo $a

echo '-----'

a=1
while (( a <= 10 ))
do
   echo $a
   a=$a+1
done

Arithmetic tests
&lt;=
&lt;
&gt;=
&gt;
==
!=
!
&amp;&amp;
||

Arrays

#!/bin/bash

acme[0]="Foo"
acme[1]="Bar"
acme[4]="Quix"

echo $acme
echo $acme[1]
echo ${acme[1]}
echo ${#acme}   # number of elements

echo '-----'
for i in 0 1
do
    echo ${acme[i]}
done

echo '----'
for x in ${acme[*]}
do
   echo $x
done

echo '----'
for x in ${acme[@]}
do
   echo $x
done

Create array

#!/bin/bash

declare -a week=(Mon Tue Wed Thu Fri Sat Sun)
echo $week
echo '-----'

for day in ${week[*]}
do
   echo $day
done

$ ./examples/script/array_creation.sh 
Mon
-----
Mon
Tue
Wed
Thu
Fri
Sat
Sun

Users

Overview

  • users
  • groups
  • permissions

/etc/passwd

  • /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
mysql:x:114:121:MySQL Server,,,:/nonexistent:/bin/false
gabor:x:1000:1000:Gabor Szabo:/home/gabor:/bin/bash

User properties

  • Username
  • Password
  • UID - user id
  • GID - primary group id
  • GECOS (free text details)
  • Home directory (/home/username).
  • Default shell (/bin/bash).

/etc/group

  • /etc/group
root:x:0:
daemon:x:1:
bin:x:2:
sys:x:3:
cdrom:x:24:ubuntu
sudo:x:27:ubuntu
users:x:100:
gabor:x:1000:
mysql:x:121:
  • Group name
  • ?
  • GID - Group ID
  • Additional Members

/etc/shadow

  • /etc/shadow
root:$6$6w7/wMw8$59s.cOQTW7XUhkB00DmwZewikvnnrx0/nMmAe0:16709:0:99999:7:::
daemon:*:16709:0:99999:7:::
bin:*:16709:0:99999:7:::
gabor:$6$u.GXxw4V$2DHHaY2CqRVyE3I6/wkexSGtIZ.4AP/X3j8e0:16709:0:99999:7:::
mysql:!:16709:0:99999:7:::

User manipulation

  • useradd - low level utility to add user
  • adduser - Perl script wrapping useradd
  • usermod -
  • userdel

Group manipulation

  • groupadd - low level utility to add group
  • addgroup - Perl script wrapping groupadd
  • groupmod
  • groupdel

/etc/skel

Files to be copied to the home directory when a new user is added.

ls -l1 /etc/skel/
.bash_logout
.bashrc
.profile
.Xresources

/etc/profile

  • /etc/profile

  • /etc/profile.d/

  • ~/.profile

  • /etc/profile - System-wide profile loaded when user logs in and loads the files in:

  • /etc/profile.d/*

  • ~/.profile - User specific profile

Changing password

  • passwd

  • Change user's own password.

  • Change the password of other users (privileged).

  • Lock out user.

Change shell - chsh

  • chsh
  • /etc/passwd
  • /etc/shells

Updates /etc/passwd with one of the entries from /etc/shells

$ chsh -s SHELL

More

Access modes

  • t - Sticky bit - Keep executable in memory after exit
  • s - SUID - Set process user ID on execution
  • s - SGID - Set process group ID on execution.

Process CSV file

  • perl
Tudor;Vidor;10;Hapci
Szundi;Morgo;7;Szende
Kuka;Hofeherke;100;Kiralyno
Boszorkany;Herceg;9;Meselo

perl -n -a -F';' -E '$x += $F[2]; END { say $x }' examples/linux/process_csv_file.csv

Shell Pipe ideas

sudo find /tmp/* -type d -cmin +30 -print0 | xargs -0 -I{} sudo ls -dl {} | wc -l
sudo find /tmp/* -type d -cmin +30 -print0 | xargs -0 -I{} sudo rm -rf {}

# The first commit of the README file:
git log README | grep ^commit | tail -1 | cut -d ' ' -f 2

Make a 1% sample of a large : separated accounting file
awk -F: 'rand() &lt; 0.01' accounting &gt; 1percent

List and kill processes by username

List the PIDs of the given user.

$ ps aux | awk '$1=="gabor" {print $2}'

$ ps aux | awk '$1=="gabor" {print $2}' | kill -9

Compiling software: Perl

Exercise: compile

Compile and install a new version of Python and/or Apache http server.