Skip to content

Instantly share code, notes, and snippets.

@danielkrizian
Last active October 24, 2023 18:43
Show Gist options
  • Select an option

  • Save danielkrizian/4ba62acae2e1c043647b5eed0f7e3933 to your computer and use it in GitHub Desktop.

Select an option

Save danielkrizian/4ba62acae2e1c043647b5eed0f7e3933 to your computer and use it in GitHub Desktop.
shell bash cheatsheet

special parameters

files

tar

tar -xvf archive.tar.xz
tar -zxvf archive.tar.gz

copy

cp file /dir
cp -R dir dir/                  # including src dir
cp -R dir/ dir/                 # only contents of dir, not dir itself

copy between machines

  • rsync -avzh <multiple source dirs> user@host:~/destination/dir (a)rchive: recursion; (z)compress; (h)uman readable

(don’t use scp, it doesn’t copy symlinks)

find

find **/org* -name "*.elc" -exec cp {} ~/tmp/elpa-elc/ \;    # find files matched with wildcard asterisk, then copy them
find **/org* -name "*.elc" -delete
find /dir/* -type f -mtime +14 | xargs rm                    # delete more than 14 days

grep -ir --color error *file*                 # (i)gnore case, (r)ecursive
sed -n 5,8p file                              # print range of lines (5-8)
grep -in -B 4 -A 10 error /path/file          # lines around matches

env

env $(cat .env) /some/app  # source env vars for the app, using .env file https://en.wikipedia.org/wiki/Env

text

head -n 10 file

replace text in file

find /path/to/files -type f -exec sed -i 's/oldstring/new string/g' {} \;

groups and users

id myuser

superuser (sudo)

sudo su sudo -s enter sudo mode

redirections

https://catonmat.net/bash-one-liners-explained-part-three

processes

top  # then to display SWAP column and sort it, press s p <spacebar> O p
ps aux    # snapshot of all current processes
ps -Flww -p PIDLIST
ps -p PID -o %cpu,%mem,cmd # detailed info about CPU and memory usage of that process
ps -p PID -o user=  # owner of the process
pstree -s PID; ps -p PID -o ppid= # process parent
ls -l /proc/PID/exe   # what is the executable?
ls -l /proc/PID/cwd   # current working dir of the process
strace -p PID  # trace system calls of the process
pkill -f 'q -p'   # kill all matching (p)attern in (f)ull command; example: (q) processes with (p)ort

ports (networking)

ncat -vzw 5 host.domain <port>     # check if can connect to port; (v)erbose, (w)ait 5 secs

ss -a | grep 29600 | grep <ip2>    # check if process connected from port to ip address; (ss)ockets utility; (a)ll sockets: both listening and established connections
  ESTAB      0      0          <someip>:29600          <ip2>:49887
  ESTAB      0      0          <someip>:29600          <ip2>:49888

lsof -p 97082 -i tcp -a            # list all open files - ports in this case. i.e. pid 97082 listens on 4567
  COMMAND   PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
  q       97082 katrina    4u  IPv4 2140799      0t0  TCP *:4567 (LISTEN)

netstat | grep 4567       # see established TCP connections
  tcp        0      0 localhost:50254         localhost:4567          ESTABLISHED
  tcp        0      0 localhost:4567          localhost:50254         ESTABLISHED
  unix  3      [ ]         DGRAM                    14567

netstat -anp | grep :30501 | grep ESTABLISHED | wc -l
netstat -apN | grep -i port
netstat -aB                                               # Number of bytes of data sent and received over each TCP connection
netstat -tulpn | grep PID    # network connections related to that process

ping 987654321
  PING 987654321 (12.34.56.789) 56(84) bytes of data.
  64 bytes from 12.34.56.789: icmp_seq=1 ttl=59 time=0.301 ms
  64 bytes from 12.34.56.789: icmp_seq=2 ttl=59 time=0.210 ms

nslookup 10.65.29.106
  12.34.56.789.in-addr.arpa       name = hostname1234.domain.com.

system info

  • cat /etc/issue host operating system

cpu processors info

cat /proc/cpuinfo | grep processor | wc -l lscpu lshw -class processor nproc

hostname, hosts->ip addresses,

/etc/hostname /etc/hosts /etc/resolv.conf https://ubuntuforums.org/showthread.php?t=1984993

space

dir size

du -ch /opt/conda/lib/libmkl_* | grep total

set TCP/IP buffer size

They can be set globally with the /proc/sys/net/ipv4/tcp_wmem and /proc/sys/net/ipv4/tcp_rmem files, or on individual sockets by using the SO_SNDBUF and SO_RCVBUF socket options with the setsockopt(2) call.

limit resource usage

http://www.linuxhowtos.org/Tips%20and%20Tricks/ulimit.htm ulimit -a list limits ulimit -u 30 set max number of processes for the user to 30 sysctl -a explore variables set systcl vm.swappiness=0

configure swap space

If you experience wsfull even when sufficient swap space is configured, check whether you have any soft/hard limits imposed with ulimit -v /proc/sys/vm/overcommit\_memory and /proc/sys/vm/overcommit\_ratio – these control how careful Linux is when allocating address space with respect to available physical memory plus swap.

kx recommendation:

https://k4.topicbox.com/groups/k4/Tfeb1eb551505ad07-Me8c94a6cfd4492880024f3b5/tickerplant-survival-during-out-of-memory “Thou shalt not swap!” - Mike Rosenberg.

If any swap is available, you almost certainly want vm.swappiness=1. The aim should be to never use it.

HDBs may benefit from swap being available, from the perspective of linux virt mem bean-counting wrt overcommit and file decompression - allowing linux to allocate address space that it could commit if overcommit settings required it. There should be no intention to actually utilize the swap.

Do your sys admins like cgroups? Perhaps they can restrict the set of rdb processes to a certain memory limit.

recommendations for q production servers

https://code.kx.com/q/kb/linux-production/#compression

cat /proc/sys/net/ipv4/tcp_keepalive_time = 600

NTP daemon running on the server vm.max_map_count to 1500000

configure core dump

https://linux-audit.com/understand-and-configure-core-dumps-work-on-linux/ cat /proc/sys/kernel/core_pattern core.%e sysctl -w kernel.core_pattern=/var/crash/core.%u.%e.%p

flush disk cache

sync ; sudo echo 3 | sudo tee /proc/sys/vm/drop_caches

disable Transparend Huge Pages (THP)

echo never >/sys/kernel/mm/redhat_transparent_hugepage/enabled or more permanently via grub at boot time ==transparent_hugepage=never==

curl wget md5sum

curl -fSL "path" -o nomachine.deb \
echo "${HASH} *file" | md5sum -c -
wget -P $HOME/src -O $HOME/src/script https://path/to/file

pbrun

pbrun tsm -a list -i $HOSTNAME -m inactive -f /path/to/file/that/was/deleted      # is there backup of the deleted file?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment