Combining multiple Files

Method 1

    head -n1 file1.txt > combined.txt
    for fname in *.txt
    do
        tail -n+3 $fname >> combined.txt
    done

Method 2

    awk '
        FNR==1 && NR!=1 { while (/^HEADING/) getline; }
        1 {print}
    ' file*.txt >all.txt

The first line of the awk script matches the first line of a file (FNR==1) except if it's also the first line across all files (NR==1). When these conditions are met, the expression while (/^HEADING/) getline; is executed, which causes awk to keep reading another line (skipping the current one) as long as the current one matches the regexp ^

. The second line of the awk script prints everything except for the lines that were previously skipped.

Creating Dummy file with specific size

Method 1

You can create a 10MB gzip file like this:

    head -c 10M /dev/urandom | gzip -1 >10m.gz

This uses urandom to get a high-entropy stream of bytes: since this is incompressible, the gzipped version will be about the same size as the input.

Method 2

File created using this method will be highly compressed during compression.

To create 100M size, enter:

    fallocate -l 100M file.out

The -l option specifies the length of the allocation, in bytes. Suffixes of k, m, g, t, p, e may be specified to denote KiB, MiB, GiB, etc.

Method 3

dd command syntax

The basic syntax is:

    dd if=/path/to/input of=/path/to/output [options]

OR

    dd if=/dev/zero of=/path/to/output.img [options]

OR

    dd if=/dev/zero of=YOUR-IMAGE-FILE-NAME-HERE bs=1 count=0 seek=Size-HERE

To create 1MB file (1024kb), enter:

    $ dd if=/dev/zero of=test.img bs=1024 count=0 seek=1024

You will get an empty files (also known as "sparse file") of arbitrary size using above syntax.

To create 100MB file , enter:

    $ dd if=/dev/zero of=test.img bs=1024 count=0 seek=$[1024*100]
    $ ls -lh test.img

To create 1GB, file:

    $ dd if=/dev/zero of=1g.img bs=1 count=0 seek=1G

Useful find Commands

Find files older than x days

    find /path/to/ -type f -mtime +7 -name '*.gz' -execdir rm -- '{}' \;

Explanation:

• find: the unix command for finding files/directories/links and etc.
• /path/to/: the directory to start your search in.
• -type f: only find files.
• -name '*.gz': list files that ends with .gz.
• -mtime +7: only consider the ones with modification time older than 7 days.
• -execdir ... \;: for each such result found, do the following command in ....
• rm -- '{}': remove the file; the {} part is where the find result gets substituted into from the previous part. -- means end of command parameters avoid prompting error for those files starting with hyphen.

Find files and sum the total result

    find /path/to/ -type f -mtime +7 -name '*.gz' -exec du -ch {} + | grep total$

du Commands

Folder size and sort

    du -h --max-depth=1 | sort -hr

which will give you the size of all sub-folders (level 1). The output will be sorted (largest folder on top).

While Loop - print increasing two digits number

    i=1; \
    while [ $i -le 20 ]; \
      do \
        echo $i; \
        printf -v i "%02d" $i; \
        echo $i; \
        i=$[10#$i+1]; \
      done

• i=$[10#$i+1] and i=$[${i#0}+1] can be used since this will not work i=$[$i+1]
• Octal issue - https://stackoverflow.com/questions/12821715/convert-string-into-integer-in-bash-script/12821845#12821845

Find file by date and group into folders by year and month

    #!/bin/bash
    BASE_DIR=/var/tmp/testfolder

    ## Find those files that are older than a month
    find "$BASE_DIR" -maxdepth 1 -mtime +30 -type f -name "*" | 
     while IFS= read -r file; do
        ## Get the file's modification year
        year="$(date -d "$(stat -c %y "$file")" +%Y)"
        ## Get the file's modification month
        month="$(date -d "$(stat -c %y "$file")" +%b)"

        ## Create the directories if they don't exist. The -p flag
        ## makes 'mkdir' create the parent directories as needed so
        ## you don't need to create $year explicitly.
        [[ ! -d "$BASE_DIR/$year/$month" ]] && mkdir -p "$BASE_DIR/$year/$month"; 

        ## Move the file
        mv "$file" "$BASE_DIR/$year/$month"
    done

find for a string and sed to replace

    find . -name "*.py" -exec sed -i "s/foo/bar/g" {} +

To know more on using + instead of \; - https://askubuntu.com/a/339019

Remove prefix from multiple files

Bash

for file in prefix*; do mv "$file" "${file#prefix}"; done;

The for loop iterates over all files with the prefix. The do removes from all those files iterated over the prefix.

Here is an example to remove "bla_" form the following files:

bla_1.txt
bla_2.txt
bla_3.txt
blub.txt

Command

for file in bla_*; do mv "$file" "${file#bla_}";done;

Result in file system:

1.txt
2.txt
3.txt
blub.txt

Rename multiple files

cwd=$(pwd)
echo ${cwd}
for i in `ls`
do
echo $i
cd ${cwd}/$i
x=$(echo $i | cut -d '.' -f 1)
echo $x
# add prefix
rename "s/^/${x}./" *
echo ""
sleep 5

cd ${cwd}
done

remove space in files and replace with underscore

rename 's/\s+/_/g' *