tiny.cc/BASH former shellScript.ref lot of work to convert to HTML as all > will need to be converted to > etc even when inside
block. see also: https://tin6150.github.io/psg/scriptRosetta.word.html#ulimit ================================================================================ limit ================================================================================ csh: limit # shows limit unlimit # lift all limits, include fd bourne: ulimit -a # show all limit ulimit -s unlimited # no bound for stack size lilmit ulimit -n 1024 # number of open file descriptor, sh need this separate from -s unlimited # somehow can't seems to go above 1024 on sh -H # hard limit -S # soft limit (default set both) CentOS 7 user limit set in /etc/security/limits.conf see linux.html open file descriptor limit need to be reset in /etc/system as set rlim_fd_max=4096 kernel default 8=1024,9+=64k, but different shell limit -n are soft limit, and lock it to 1024 to 4096, which is probably still okay /etc/system param change should not be needed in sol 9 and 10, few program can use more than 64k. check "sysdef | grep file\ desc" # current:max pair -------------------------------------------------------------------------------- mail. use /usr/ucb/Mail, exist in linux and solaris cat FILE | Mail -s "subject line" tho01@cs (linux mail does not support -s subject option, while solaris will not use the first line of the message that has 'Subject: bla' in it as the subject.) wisdom from Mohit: set absolute path at the beginning of the script. this way, no need to define commands with full path. The only other issue is, same command, such as awk, maybe in multiple places, which is not really same (linux awk is solaris nawk). Path sequence need to be assigned carefully for such commands. In a very heterogeneous platform, maybe hard and hard path prefered... ******************************************************************************** sh/bash ******************************************************************************** echo $PATH | tr ':' '\012' \012 is the octal value for linefeed, note that \015, CR, will not work in unix. --- arithmetic, numerical expressions NUM = `expr 10 + 10` NUM = `expr 10 \* 10` NUM = `expr 10 \/ 10` New in bash4 NUM = $(( 10+10 )) echo $(( 10 * 2 )) # no escape needed for * echo $(( 10 / 3 )) # no escape needed for / , Note, $(( )) only return integer portion of result echo $(( 10 % 3 )) # modulus operation, ie, give reminder X=10 Y=3 Z=$(( X / Y )) # the $VAR don't need to be used inside $(( )). Z=$(( $X/$Y )) One dimentional array in bash userlist=( foo bar baz ) echo ${userlist[0]} # get foo, first element of array is at index 0. echo ${userlist[*]} # get foo bar baz, all the elemnts are listed with * as index. echo ${userlist[@]} # get foo bar baz, all the elemnts are listed with @ as index. echo ${#userlist[@]} # get 3, the size of the array. for i in 0 1 2; do echo ${userlist[i]}; done # can use either i or $i as index variable inside the ${var[]} syntax for i in 0 1 2; do echo ${userlist[$i]}; done more examples, array slides ${userlist[1]:1:2}, etc in http://www.tldp.org/LDP/abs/html/arrays.html Integer comparision test INT1 -eq INT2 INT1 -ne INT2 INT1 -gt INT2 INT1 -lt INT2 String comparison test Str1 = Str2 # use = with test or [ ] Str1 == Str2 # use == with [[ ]] Str1 != Str2 Since they are string, may as well use "Str" so account for undefined/empty string cases. -z Str # zero length string -n Str # non-zero length string Str1 =~ regex # true if regex matched (ie found in Str1) I think =~ is very much a perl-ism , so nice it work in bash too! echo {z..a} echo {f..k} echo {08..12} # dont really need $(seq -w 08 1 12) for simple cases echo {3..-3} # space inside {} isn't tolerated. echo {03..-03} # funny output assigning regex "chunk" to variable, ref: https://www.networkworld.com/article/2693361/unix-tip-using-bash-s-regular-expressions.html read -p "Enter email: " email if [[ "$email" =~ ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}$ ]] then echo "This email address looks fine: $email" else echo "This email address is flawed: $email" fi bash parameter expansion (substring matching, though not exactly regex). https://tldp.org/LDP/abs/html/string-manipulation.html under substring extraction ${string%substring} % remove shorted substring from the end ${string%%substring} %% Deletes longest match of $substring from back of $string. stringZ=abcABC123ABCabc # || shortest # |------------| longest echo ${stringZ%b*c} # abcABC123ABCa # Strip out shortest match between 'b' and 'c', from back of $stringZ. echo ${stringZ%%b*c} # a # Strip out longest match between 'b' and 'c', from back of $stringZ. use # and ## for beginning of the string: ${string#substring} # Deletes shortest match of $substring from front of $string. ${string##substring} ## Deletes longest match of $substring from front of $string. ${string/substring/replacement} / Replace first match of $substring with $replacement. ${string//substring/replacement} // Replace all matches of $substring with $replacement. # next 3 are the same, show lenghth of the string. expr has a length sub-command! or can use the 3rd form cryptic syntax. compare to C's strlen() ${#string} expr length $string expr "$string" : '.*' There are many more, under this convoluted man page: https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion eg below is souped up "basename", matching arbitrary substring as "extension". cuz ${parameter%word} aka ${var%pattern} use bash's parameter expansion for i in *_1.fq.gz do string1="${i%_1.fq.gz}" echo ${string1} unicycler -1 ${string1}_1.fq.gz -2 ${string1}_2.fq.gz -o /Users/desktop/Fasta/${string1}.fasta --min_fasta_length 500 done -------- LD_PRELOAD_OPTS=... This environemnt will allow specific library to be "preloaded", before LD_LIBRARY_PATH modules get loaded. This way, program that has specific requirements for libs but may otherwise conflict with studd in LD_LIBRARY_PATH can be preferentially loaded first, thus used first and resolve incomptabilities issues. eg: make LD_PRELOAD_OPTS=/apps/lib other important env var (at least for RHEL): PATH MANPATH LD_LIBRARY_PATH LD_RUN_PATH INCLUDE_PATH C_INCLUDE_PATH compiling program with static lib (glibc-static needed!) ./configure LDFLAGS=-static make LDFLAGS=-all-static ------- fucntion defintions examples: Size() { ls -l $* | awk '{sum+=$5} END {print sum}' ; } # Size *.txt # not /usr/bin/size! # SGE env, print summary num of nodes, cpu cores, memory in cluster qhostTot() { qhost | awk '{h+=1; c+=$3} END {print "host="h " core=" c}' ; } qhostTot() { qhost | sed 's/G//g' | awk '/^sky/ {h+=1; c+=$3; m+=$8} END {print "host="h " core=" c " RAM=" m "G"}' ; } emulating main fn with argument passing: have fn called main() { ... } then at end of file, call it as main $* $* = all arguments inside the function, $1 = first argument use x=$1 if want to assigne the argument to a variable, don't use set x=$1 The actual calling of the fn does not need () before args. fn really become mini shell scripts w/in the script. (fn implementation is really like a kludgy after-thought) declare -F # list all defined fn by name declare # complete details of all defined functions in current shell ---- for programs that insist of having a terminal attached to it, use this in the shell script, and the whole script will seems to have a terminal attached to it (don't know what is implication of stdin). exec 0 /tmp/heredoc.txt << END_HEREDOC some long text that span many lines END_HEREDOC it can be hacked to emulate goto which doesn't exist in bash: cat >/dev/null << LABEL # sh code # that will # be skipped LABEL # code that will be exeucted bash support multiline echo, so heredocs are less critical: echo " some long text that span many lines " > /tmp/heredoc.txt for multilines comment, can use a variation of the fork bomb as: : ' blah blah blah foo bar baz ref: https://stackoverflow.com/questions/43158140/way-to-create-multiline-comments-in-bash ' shell editor set -o vi emacs mode is default ^R (ctrl+shift+r) backward history search (like vi mode ESC,?) *** keyboard shortcut for bash *** probably natura to emacs users... ^R reverse search in history, any arrow key will enter edit mode of that l ine cmd. ^W word backward erase ^U whole line erase ^A goto beginning of the line ^E goto end of the line ^L refresh screen ^S suspend scroll ^Q continue scroll ^P = up arrow, ie prev command ^B = left arrow ^J = enter ^M = enter ================================================================================ bash version 4 (and not sh) ================================================================================ $( cmd ) # `cmd` : a more readable version of backtick execution echo $(( 5/2 )) # echo `expr 5 / 2` # $(( )) will perform integer arithmatics akin to expr ================================================================================ ksh ================================================================================ set -o vi / = forward search of command history ? = backward search use vi commands to edit history, enter to run it. ================================================================================ terminal ================================================================================ stty -a display settings for terminal HP-UX seems to default "kill" to @, which cause anoying problem of mapping @ to line erase. solaris seems to use ^u for "kill". reset via key strokes of stty killor stty kill stty intr ^V^C # HP default don't take ^C for break, map to DEL which isn't delete key. stty susp ^V^Z # for suspend process ================================================================================ csh/tcsh ================================================================================ #### argggg... no fn in t/csh :( http://www.grymoire.com/Unix/CshTop10.txt #### also google for "csh considered harmful" #### why is cmaq still stuck in csh ??!! csh has a GOTO command (bash don't, but can use heredocs to emulate the fn): goto label echo this won't execute label: echo this will execute so, instead of fn, can use goto wonderful! csh may not source .login just like bash may not source .profile ? docker exec tcsh, does NOT source /etc/csh.login but DOES source /etc/csh.cshrc so put things in .cshrc (like putting things in .bashrc). eff-it :-/ ** CSH Anoyance ** csh craps: PATH is treated special in csh it seems csh keep a sync of path and PATH as space and colon delimited list only PATH has such special treatment, LD_LIBRARY_PATH does not. ##** $path is a space delimited list ##** $PATH is a colon delimited list ##** setenv expects a colon delimited list ##** set expects a space delimited list set path = ( ${path:q} /opt/openmpi/2.1.6/bin ) set path = ( $path /bin /usr/local/bin /usr/bin /usr/bin/X11 ~/bin /sbin /usr/sbin . ) ##*** don't use $PATH above, it must be lower case $path in the parenthesis. ##*** or else it will consider the colon delimited list as a unit and path won't work, ##*** even when displayed correctly!! another reason why csh is bad!! =) setenv PATH "${PATH}:${AMGEN_HOME}/bin" ### above would works, ${PATH} can be used like that. ### CSH treats : in variable name as manipulation token, ### braces {} will properly define where variable name eval ends. ### ${var:q} will apply :q var evaluation to $var ### but ${var}:bla , : is outside variable name eval and not treated special by csh... ### csh variable modifiler , only 1 can be used, cannot be combined # :q -- quote wordlist, keeping items separate; don't expand special metacharater (ie, keep things verbatim). it does NOTHING on regard whether var exist or not # :x -- quote a pattern, expand to wordlist # :e -- get var extension. these are like basename command? eg set TEST=/foo/test.txt ; echo $TEST:e get txt # :t -- get var's tail # :h -- get header. set TEST=/foo/bar/test.txt ; echo $TEST:h get /foo/bar # :ge -- get all extensions (for when there are multiple files? space delimited?) ### tcsh add :l and :u to lowercase and uppercase the FIRST letter of the var ### http://www.grymoire.com/Unix/Csh.html#uh-92 ### https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/osmanagement/c_var_file_sub.html ### https://docstore.mik.ua/orelly/unix/unixnut/ch05_03.htm ##** check path using env | grep -i path ##** and NOT echo $PATH as csh treat such evaluation specially and ##** it will produce confusing results! ### *** CONCLUSION: Best use space and the lower case version!! it is the most reliable way. ### set path ( $path /more/path ) ### cuz csh will print $path without : in them and have properly "tokenized" the entries ### /etc/csh.login do this to update path (in a loop actually) ### set path = ( ${path:q} /opt/openmpi/2.1.6/bin ) ### csh environment var qualifier :q may skip error when path was not declared or empty ?? NOPE, :q is for quote list ### LD_LIBRARY_PATH seems to be okay with the setenv LD_LIBRARY_PATH ### and colon separated list... ### csh need seed the variables before use. very tedious! ### maybe ${VAR:q} can be appended without causing error when it was never declared before? if(! ${?PATH}) then set path = ( /bin ) # ie seed PATH since it was never defined endif if(! ${?LD_LIBRARY_PATH}) then setenv LD_LIBRARY_PATH "/lib" # ie seed PATH since it was never defined endif setenv DISPLAY hostname:1 unsetenv DISPLAY # undo setenv set ... unset ... # undo set CSH variable evaluation. If a variable is not defined, trying to access it will give an error. I guess it is like C programming language after all. sh will just print blank, which I think is easier to deal with in shell script. But anyway, in CSH, to guard against such error, need to test the variable being defined before using it. eg if ($?TRACE_LOG) then setenv TRACE_LOG "$TRACE_LOG openeye_processed" else setenv TRACE_LOG "openeye_processed" endif if ($?MANPATH) then setenv MANPATH "${MANPATH}:/usr/share/man:/usr/man:/usr/local/man" else setenv MANPATH "/usr/share/man:/usr/man:/usr/local/man" endif These kind of check should be done for things like LD_LIBRARY_PATH, etc. There is probably an easier way, but :q doesn't really help. :q is used in csh.login likely to keep strange metacharacter from being interpreted. csh if statement, string comparison set machine = `hostname` if !( ( ${machine} == "firth" ) || ( ${machine} == "firth.amgen.com" ) ) then echo "no Firth here... do work" else echo "it is Firth here" endif http://www.unet.univie.ac.at/aix/aixuser/usrosdev/c_shell_cmds.htm#A279911bb :: C Shell Expressions and Operators Operator What it Means () change precedence ~ complement ! negation * / % multiply, divide, modulo + - add, subtract << > > left shift, right shift <= >= < > relational operators == != =~ !~ string comparison/pattern matching & bitwise "and" ^ bitwise "exclusive or" | bitwise "inclusive or" && logical "and" || logical "or" Things to have in .cshrc :: http://mail.hudat.com/~ken/help/unix/.cshrc #alias ins2path 'if ("$path:q" !~ *"\!$"* ) set path=( \!$ $path )' #alias add2path 'if ("$path:q" !~ *"\!$"* ) set path=( $path \!$ )' add2path ${JAVA_HOME}/bin I/O redirection cat foo >& bar # >& = stdout + stderr (sh use 2>&1 at the end) cat foo |& tee bar # | instead of >, otherwise same as above.