Bash

Created Wednesday 04 December 2013

  1. Advanced Bash-Scripting Guide:
    1. http://www.tldp.org/LDP/abs/html/
  2. Bash Reference Manual:
    1. http://www.gnu.org/software/bash/manual/bashref.html
  3. Some reference cards:
    1. http://mealldubh.files.wordpress.com/2007/02/bashreferencecard1.pdf
    2. http://tldp.org/LDP/abs/html/refcards.html
    3. http://wiki.bash-hackers.org/commands/classictest
    4. http://wiki.bash-hackers.org/syntax/ccmd/conditional_expression
    5. http://database.sarang.net/study/bash/bash.pdf

  1. Loop directories:
    1. http://stackoverflow.com/questions/15854559/bash-looping-over-directories
for path in /my/path/W*; do
[ -d "${path}" ] || continue # if not a directory, skip
dirname="$(basename "${path}")"
do_stuff
done
  1. Recursive loop with file creation in every dir (based on above):
#!/bin/bash
loopDirs()
{
for path in $1*; do
[ -d "${path}" ] || continue
touch "${path}"/.gitignore
loopDirs "${path}"/
done
}

loopDirs ./

  1. stderr and stdout to file:
    1. http://www.cyberciti.biz/faq/redirecting-stderr-to-stdout/
command > file-name 2>&1

  1. Making backup script that runs every night and implements the rotation idea:
    1. The major part of a script (a separate file, e.g. mybackcycle.sh):
#!/bin/bash
# Variables
backdirname=$(date +%d-%m-%Y_%H-%M-%S) # Backup directory name
dfOutputStr="" # Get disk usage from df -h
dfOutputNum="" # Rid of '%' sign
rawOldest="" # has ':' in the end
oldest="" # The oldest backup directory (full path)
diskRegex="/^\/dev\/sdb1/" # Path to disk for awk
maxAllowedUsage=80 # How much % your backup will use (integer only!)
pathToMounted="/media/yourUser/extdisk/" # Path to mount point
diskPath="/dev/sdb1" # Path to disk device
backupSource="/home/yourUser/" # Path to backup source
separator="----------------------------------------------------------------------------------------"

echo "New backup $backdirname" >> backups.log
echo $separator >> backups.log
# Mounting...
sudo ./mountsdb1.sh "$diskPath" "$pathToMounted" >> backups.log 2>&1
# Checking usage + getting path to the oldest backup
function checkSizeGetOldest()
{
# Getting the right usage field from 'df -h' output (e.g. 80%)
dfOutputStr=$(df -h | awk '
     '$1' {
         print $5;http://blog.twistedcode.org/2008/03/customizing-your-bash-prompt.html
     }
     ')
# Removing '%' char
dfOutputNum=$(echo $dfOutputStr | awk 'BEGIN {FS="%"}{print $1}')
# Checking if output of df -h didn't change (by '%' char)
** if $dfOutputStr == *% ; then**
# Checking if the space usage is more than allowed
    if [ $dfOutputNum -gt $2 ]; then
# Getting the path to the oldest backup
     rawOldest=$(ls -tr $3*_* | head -1)
     oldest=$(echo $rawOldest | awk 'BEGIN {FS=":"}{print $1}')
     echo "Usage is $dfOutputStr" >> backups.log
     echo "The oldest backup is $oldest" backups.log
    fi
fi
}
# Calling the just defined function :)
checkSizeGetOldest $diskRegex $maxAllowedUsage $pathToMounted
# And if there is problem with outusage then check again with loop
while [ $dfOutputNum -gt $maxAllowedUsage ]; do
echo "Removing backup $oldest" >> backups.log
# Removing the oldest backup
rm -Rf $oldest >> backups.log 2>&1
wait
# Updating the usage, and if it's OK - exiting the loop
checkSizeGetOldest $diskRegex $maxAllowedUsage $pathToMounted
done

echo "Making backup directory $pathToMounted$backdirname" >> backups.log
mkdir $pathToMounted$backdirname >> backups.log 2>&1
echo "Making a copy..." >> backups.log
cp -Rp $backupSource* $pathToMounted$backdirname >> backups.log 2>&1
wait
echo "DONE" >> backups.log
echo $separator >> backups.log

  1. The second part of the script that mounts a disk (file mountsdb1.sh):
    1. Don't want the whole script run as root, so adding to sudoers just that:
#!/bin/bash
mount -t ext4 $1 $2

  1. Now we should add the second file to sudoers so that the mounting process won't be interrupted by password requirement:
    1. sudo visudo
    2. yourUser ALL=(ALL) NOPASSWD: /home/yourUser/path/to/mountsdb1.sh
  2. Now it's time to add a cronjob like that:
    1. crontab -e
    2. 0 0 * * * /home/yourUser/path/to//mybackcycle.sh

  1. File character encoding:
    1. http://mindspill.net/computing/linux-notes/determine-and-change-file-character-encoding/
    2. http://www.softlexicon.com/2012/02/character-encoding-convert-from-utf-8-to-iso-8859-1.html
    3. file -bi filename - get encoding
    4. iconv -f ascii -t utf8 [filename] > [newfilename] - change encoding
    5. recode UTF-8 [filename] - change encoding
    6. iconv -f utf8 -t ascii [filename] - change encoding
    7. iconv --from-code=utf-8 --to-code=iso-8859-1
  2. Change EOL (end of line):
    1. http://www.cyberciti.biz/faq/howto-unix-linux-convert-dos-newlines-cr-lf-unix-text-format/
      1. dos2unix filename - to LF
      2. dos2unix -b filename - will also backup to filename.bak
      3. unix2dos [-b] filename - to CRLF
      4. tr -d '\r' < input.file > output.file - to LF from CRLF
      5. perl -pi -e 's/\r\n/\n/g' input.file - to LF from CRLF
      6. sed 's/$'"/`echo \\\r` /" input.file > output.file - to CRLF from LF
    2. Recursive
      1. http://stackoverflow.com/questions/11929461/dos2unix-command-on-entire-directory
        1. find . -type f -exec dos2unix {} \;
        2. find . -type f -print0 | xargs -0 dos2unix
      2. http://stackoverflow.com/questions/2228039/how-to-run-a-command-recursively-on-all-files-except-for-those-under-svn-direct
        1. find -exec dos2unix {} \;
  3. Removing BOM from utf-8 files:
    1. http://bkarak.wizhut.com/blog/2011/04/28/remove-that-utf-8-bom-with-one-simple-command/
      1. tail -b 4 [your-file] > [your-file]
    2. http://www.linuxask.com/questions/how-to-remove-bom-from-utf-8
      1. awk '{if(NR==1)sub(/^\xef\xbb\xbf/,"");print}' text.txt
    3. https://groups.google.com/forum/#!topic/vim_use/jsRI8Ql7l4c
      1. "Do ':set nobomb' before saving to remove a BOM." - for VIM
    4. Emacs:
      1. http://dontomp.net/2012/09/01/remove-bom-with-emacs/
      2. http://stackoverflow.com/questions/7347758/emacs-hexl-mode-utf8-bom-issue
  4. Rename multiple files:
    1. http://www.cyberciti.biz/tips/renaming-multiple-files-at-a-shell-prompt.html
      1. Rename all somename.jpg files to somename-big.jpg:
        1. rename 's/(.*)\.jpg/$1-big.jpg/g' *.jpg
  5. Images:
    1. View image metadata:
      1. http://gnutips.wordpress.com/2010/08/10/view-image-file-metadata-from-the-command-line/
        1. identify -verbose image_file.jpg
    2. View image dimensions:
      1. http://www.garron.me/en/linux/get-size-type-of-picture-linux-console.html
        1. identify -verbose image_file.jpg
    3. Resize image dimensions (follow the link for -quality, etc..):
      1. http://www.howtogeek.com/109369/how-to-quickly-resize-convert-modify-images-from-the-linux-terminal/
        1. convert image.jpg -resize 100x50 image.jpg
        2. convert image.jpg -resize 100 image.jpg - Resize by width saving aspect ratio
        3. convert image.jpg -resize x50 image.jpg - Resize by height saving aspect ratio
        4. for file in *-arrows-*; do convert $file -resize 88 resized-$file; done - Resize all with '-arrows-' in their names
    4. Resize image file size:
      1. http://superuser.com/questions/470954/how-to-compress-reduce-the-size-of-jpeg-photos-for-archiving
  6. Generate 'random' string of a fixed length, containing A-Z, a-z, 0-9, '/', '+':
    1. apg -a 1 -M SNCL -m 216 -s -E \!\|?#%]~[{}\&\\\'\>\;,-\<\*=_\(\):^.\"\`@$
    2. There may be other ways:
      1. http://www.howtogeek.com/howto/30184/10-ways-to-generate-a-random-password-from-the-command-line/
  7. Detect changes:
    1. http://superuser.com/questions/181517/how-to-execute-a-command-whenever-a-file-changes
  8. Adding to path:
    1. http://askubuntu.com/questions/322772/how-do-i-add-an-executable-to-my-search-path
      1. Create ~/bin
      2. Open ~/.bashrc
      3. export PATH="/home/$USER/bin:$PATH"
  9. Create symlink:
    1. http://stackoverflow.com/questions/1951742/how-to-symlink-a-file-in-linux
      1. ln -s /path/to/file /path/to/symlink
  10. Debugging:
    1. https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html
      1. Add set (or set -x) in the script's beginning
  11. List all aliases:
    1. http://stackoverflow.com/questions/948008/linux-command-to-list-all-available-commands-and-aliases
      1. compgen -a
  12. Set alias permanently:
    1. Local:
      1. echo 'alias whois="jwhois"' >> ~/.bash_aliases
    2. Global:
      1. sudo echo 'alias whois="jwhois"' >> /etc/bash.bashrc
    3. Linking:
      1. sudo mv /usr/bin/whois /usr/bin/whois_back
      2. sudo ln -s /usr/bin/jwhois /usr/bin/whois
  13. Recursive grep for specific files:
    1. http://www.commandlinefu.com/commands/view/2998/grep-certain-file-types-recursively
      1. grep -r --include="*.[ch]" pattern .
      2. grep -r --include="*.php" pattern .
    2. http://stackoverflow.com/questions/221921/use-grep-exclude-include-syntax-to-not-grep-through-certain-files
      1. grep -r --include=\*.{cpp,h} pattern dir
  14. Send all output to stdout and filtered output to a file
    1. http://stackoverflow.com/questions/14778917/how-can-i-use-tee-to-send-all-output-to-stdout-and-grepped-output-to-file?answertab=votes#tab-top
      1. ls -latr | awk '/^d.*/{print > "ls_test.txt"} 1'
  15. Replace string (substitute) recursively:
    1. https://coderwall.com/p/2ivpsg
      1. egrep -lRZ 'foo' . | xargs -0 -l sed -i -e 's/foo/bar/g'
    2. http://stackoverflow.com/questions/1583219/awk-sed-how-to-do-a-recursive-find-replace-of-a-string
      1. find . -type f -print0 | xargs -0 sed -i 's/subdomainA.example.com/subdomainB.example.com/g'
    3. http://www.jonasblog.com/search-and-replace-in-all-files-within-a-directory-recursively
      1. find ./ -type f -exec sed -i 's/string1/string2/g' {} \;
    4. grep -rlZ '$yo = 2' . | xargs -0 sed -i 's/\$yo = 2/\$yo = 0/'
    5. grep -rElZ '^\s*&DoSomething' . | xargs -0 sed -i 's/^\(\s*&DoSomething[^;]*\)\(;.*\)$/\1 if ($test)\2/'
    6. grep -rElZ 'if \(\$debug\);$' . | xargs -0 sed -i 's/ if (\$test)\(;\)$/\1/'
    7. grep -rElZ '^\s*&DoSome \("Some: \$thing"\)' . | xargs -0 sed -i 's/^\(\s*&DoSome ("Some: $thing")\)\(;\)/\1 if ($test)\2/'
    8. grep -rElZ '^\s*&DoSome \("Some: @thing"\)' . | xargs -0 sed -i 's/\(\s*&DoSome ("Some: @thing")\)\(;\)/\1 if ($test)\2/'
    9. perl -p -i -e 's/(?<![\.\w])(test\.co\.il)/dev$1/g' `grep -rl 'test' .`
      1. Prepare the list of files that will be edited:
        1. perl -ne 'print "$ARGV\n" if s/(?<![\.\w])(test\.co\.il)/dev$1/g;' `grep -rl 'test.co.il' .` | sort -u > test.list
  16. View line endings (eol):
    1. http://www.if-not-true-then-false.com/2011/linux-display-show-file-contents-tabs-line-breaks-non-printing-characters/
      1. cat -A file.txt
  17. Create file with time stamp in name
    1. http://www.cyberciti.biz/tips/shell-scripting-creating-reportlog-file-names-with-date-in-filename.html
    2. touch filename_`date +"%d-%m-%Y_%H-%M-%S"`
  18. Search recursively for files with a specific name and print their relative path with contents after that:
    1. find . -name "filename.ext" -exec echo {} \; -exec cat {} \;
  19. Remove multiple files but not all:
    1. http://stackoverflow.com/questions/1133698/find-name-pattern-that-matches-multiple-patterns
    2. http://stackoverflow.com/questions/8887972/find-name-xyz-o-name-abc-exec-to-execute-on-all-found-files-not-jus
    3. find . \( -name "*.html*" -o -name "*.png" \) -exec rm {} \;
  20. Get distribution version
    1. http://www.cyberciti.biz/faq/find-linux-distribution-name-version-number/
      1. cat /etc/*-release
      2. lsb_release
      3. cat /proc/version
  21. Faster grep alternatives (indexing files)
    1. http://www.techrepublic.com/article/graduating-from-grep-powerful-text-file-searching-with-isearch/
    2. http://cscope.sourceforge.net/
    3. http://www.gnu.org/software/idutils/
    4. https://code.google.com/p/seascope/
  22. find exclude directories
    1. http://stackoverflow.com/questions/4210042/exclude-directory-from-find-command
      1. find . -path ./misc -prune -o -name '*.txt' -print
  23. Clear DNS cache
    1. http://askubuntu.com/questions/414826/how-to-flush-dns-in-ubuntu-12-04
      1. sudo apt-get install nscd
      2. sudo /etc/init.d/nscd restart
      3. sudo service nscd restart
  24. Check file for BOM
    1. http://blog.ijun.org/2011/01/to-check-if-file-contains-utf-8-bom-at.html
      1. hexdump -n 3 -C file.txt
        1. ef bb bf - YES
  25. Setting prompt
    1. http://blog.twistedcode.org/2008/03/customizing-your-bash-prompt.html
      1. export PS1='@'
  26. Kill graphic window
    1. http://cboard.cprogramming.com/linux-programming/97665-what-command-exit-xfce.html
      1. xkill
  27. Run repeatedly (repeat every x seconds):
    1. http://askubuntu.com/questions/430382/repeat-a-command-every-x-interval-of-time-in-terminal
      1. watch -n 0.1 'ls -lat'
  28. Print new line after returned string
    1. printf 'a=b&c=d&e=f' | nc localhost 1234 | awk {print}
  29. Copy recursively including hidden files:
    1. http://superuser.com/questions/61611/how-to-copy-with-cp-to-include-hidden-files-and-hidden-directories-and-their-con
    2. cp -r /etc/skel/. /home/<new_user>
  30. Rsync with excluded locally:
    1. rsync -rhv --progress --exclude-from '../excluded.txt' ../fromdir .
  31. Rsync with excluded via ssh:
    1. rsync -e 'ssh' -rhv --progress --exclude-from '../excludedtxt' user@address:/from/dir/ .
  32. Recursive chmod for all *.sh files
    1. http://www.commandlinefu.com/commands/view/11936/recursive-chmod-all-.sh-files-within-the-current-directory
    2. find ./ -name "*.sh" -exec chmod +x {} \;
    3. chmod u+x **/*.sh
  33. Generate random string
    1. http://stackoverflow.com/questions/17650703/random-string-in-linux-by-system-time
    2. tr -cd '[:alnum:]' < /dev/urandom | fold -w50 | head -n1
  34. Get a list of only different files in 2 compared directories
    1. diff -qr . /path/to/other/dir | grep differ | awk '{print $2 " " $4}'
  35. Remove 2 last files from ls output
    1. http://unix.stackexchange.com/questions/134595/linux-delete-last-two-files-automatically
    2. ls -t | head -n 2 | xargs rm -f
  36. Filtering access.log for not starting with 8 or 9 and not having .jpg + formatting with Perl
    1. grep -P '^[^89]((?!\.jpg).)*$' access.log | perl -lne '@s=split(/\s+/,$_);printf("%16s\t%s\t%s\n",$s[0],$s[3],$s[6]);' > temp.txt
  37. Copy file recursively to every directory in tree
    1. http://unix.stackexchange.com/questions/94207/recursively-add-a-file-to-all-sub-directories
find . -type d -exec cp test.txt {} \;
|-- a
| |-- b
| | |-- c
| | | `-- test.txt
| | `-- test.txt
| |-- d
| | |-- e
| | | `-- test.txt
| | `-- test.txt
| `-- test.txt
|-- f
| |-- g
| | |-- e
| | | `-- test.txt
| | `-- test.txt
| `-- test.txt
`-- test.txt

  1. Look for recursively and print all matching lines after substitution (emulation before replacement)
    1. perl -ne 'print $_ if s/(?<![\.\w])(test\.co\.il)/dev$1/g;' `grep -rlE 'test' .`
  2. Delete millions of files
    1. http://www.pronego.com/helpdesk/knowledgebase.php?article=59
    2. find . -type f -print -delete
  3. Check command succession
    1. http://askubuntu.com/questions/29370/how-to-check-if-a-command-succeeded
some_command
if [ $? -eq 0 ]; then
echo OK
else
echo FAIL
fi
OR
some_command
retval=$?
do_something $retval
if [ $retval -ne 0 ]; then
echo "Return code was not zero but $retval"
fi
  1. Redirect stdout to var
    1. https://forums.opensuse.org/showthread.php/390664-How-can-I-redirect-stdout-to-a-variable
    2. var="$(fdisk -l)"
  2. grep for unique values
    1. grep --include='file_name*' -rohP '\bsome-dir/pattern.*?ext' . | cut -d '/' -f2 | sort | uniq
    2. grep --include='file_name*' -rohP 'some-dir/.*?ext' . | sort | uniq
    3. For gzipped files
      1. zgrep -Poh 'some-dir/pattern.*?ext' file_name* | cut -d '/' -f 2 | sort | uniq
  3. cat cut sed
    1. cat founds.txt | cut -d ' ' -f 4,5,7 | sed 's/\(\[\|\]\| +[0-9]\+\|\?.*\)//g'
  4. Convert dates
    1. http://stackoverflow.com/questions/6508819/convert-date-formats-in-bash
    2. date -dyesterday +%Y%m%d
    3. date -d'27 JUN 2011' +%Y%m%d
    4. date -d'last-monday' +%Y%m%d
    5. DATE = "27 jun 2011"; date -d"$DATE" +%Y%m%d
    6. read -p " Get date >> " DATE; printf " AS YYYYMMDD format >> %s" `date -d"$DATE" +%Y%m%d`
    7. echo "27 june 2011" | awk '{print "date -d\""$1FS$2FS$3"\" +%Y%m%d"}' | bash
  5. sed run command on backreference
    1. http://stackoverflow.com/questions/6355443/using-command-substitution-inside-a-sed-script-with-arguments
    2. https://www.gnu.org/software/sed/manual/html_node/The-_0022s_0022-Command.html
    3. cat founds1.txt | sed 's/^\(..\)\/\(...\)\/\(....\):\(..:..:..\)\(.*\)$/echo `date -d "\1 \2 \3 \4" +%s` \5/ep' | sort
  6. Date to timestamp:
    1. date -d "01 Apr 2014 10:30:13" +%s
  7. Sort by 4th column — all numbers + reversed
    1. cat founds11.txt | sort -k4nr
  8. grep awk perl
    1. grep -P '.*cgi\*$' flist.txt | awk '{print $8}' | perl -ne '$_ =~ s/\*$//; print $_' > flist_1.txt



Backlinks: