Bash Code Snippets

Collection of bash snippets which I had been keeping in gists and elsewhere. Some of these are hacky but they get the job done quickly. Updated and replaced when improved.

Curl and WGet

Create WARC of a Website (e.g. C4LJ)

Filenames and domains matter of course.

wget \
    --mirror \
    --warc-file=C4LJ_backup_05-05-2017 \
    --warc-cdx \
    --page-requisites \
    --html-extension \
    --convert-links \
    --execute robots=off \
    --directory-prefix=. \
    --span-hosts \
    --domains=journal.code4lib.org,code4lib.org \
    --user-agent=Mozilla \
    --wait=5 \
    --random-wait \
    http://journal.code4lib.org

List of LCCNs to MADSRDF

[note: this may need to be checked]

With 3 second sleep

Names:

#!/bin/bash
for lccn in nb2015017302 n89223874 n82203435 n2017074015
do
  curl http://id.loc.gov/authorities/names/$lccn.madsrdf.rdf -o $lccn.rdf
  sleep 3
done

Or alternatively subjects:

/authorities/subjects/

List of LCCNs to MARCXML

#!/bin/bash
for lccn in nb2015017302 sh2018002914 n89223874 n82203435 sh85033801
do
  curl https://lccn.loc.gov/$lccn/marcxml -o $lccn.xml
  sleep 1
done

Data Manipulation

Turn Text List Into Unique Ordered List

sort -u source_file.txt -o target_file.txt

(when using -o, the same file can be source AND target file)

Compare Arrays Write to File

for recordID in 12 16 19 20 22 29 33 35 36 37 43 44 45 47 2466 540 3437 1400 1897 2371 587
do
  if [[ " 12 16 19 20 22 29 33 35 36 37 43 44 45 47 " == *\ $recordID\ * ]]
  then
    : #pass to the next
  else
    echo $recordID >> unreconciledRecordID.txt # Echoes all the record IDs from the array not found in the second array into this file. Appending.
    echo " " >> unreconciledRecordID.txt # This is another way to space it out more. May or may not be desired.
  fi
done

Echo PDF Metadata of Each PDF Into XML File

#!/bin/bash
for each in $(ls)
  do cd "/mnt/curatend-batch/data/catholic-pamphlets/IA/"$each
  for each in $(find . -name "*.pdf")
    do pdfinfo -meta $each > "`echo $each | sed 's/.pdf/.xml/'`"
  done
done

Create MD5 Checksums in All Subdirectories

This was initially written for HathiTrust submission specs so it may require review/revision before reuse.

for dir in $(ls /path/to/directories ); do find $dir -type f | grep -v "checksum.md5" | xargs md5 -r > $dir/checksum.md5 ; done

File Manipulation

Return Lines Before and/or After With Grep

Use -A 1 to return one line after the match (as well as match), -B 1 to return 1 line before match (as well as match), and C -1 to return 1 line on either side. For example:

grep "dewey_" -A 5 dewey_lopped.txt > dewey_only_lopped.txt

gives me the five lines after the field I’m looking for so I can take only the text blocks around the actual Dewey entries in this whole index file and make a more limited file to focus on Dewey. If there were a line I needed before, I could add a -B but it’s not needed here.

Test for Invalid UTF-8 in Files

Looks for anything which doesn’t match the regex pattern and returns those matching lines. The regex will only be looking for well-encoded data.

grep -axv '.*' metadata-1.csv

Zip All Directories in a Directory

for dir in $(ls /path ); do zip -r $dir $dir ; done

Imagemagick

Transform HEIC images to PNG:

mogrify -format png *.HEIC

(may need to run again for *.heic since sometimes they come over lower case.)

Make png background white:

convert -flatten scene_1.png sm_scene_1.png

Resize a bunch of TIFFs and make them JPGs when remove the .tif that ends up in file using method (note, must be done in working directory — I be there is a thing I can do with xargs but haven’t tried):

for file in *.tif; do convert "$file" -resize '2000' "$file".jpg; done
for file in *.jpg; do mv "$file" "`echo $file | sed  's/\.tif//'`"; done

Regex Replace in Files (e.g. 1-20 spaces, add line break etc)

Sed inline (-i) with regex (-E) and backup (.bak) to replace 1-20 spaces at the beginning of a line in a file:

sed -i.bak -E s/^[[:space:]]\{1,20}//g genres_list.txt

Replace digits with nothing

sed -i.bak -E s/^[0-9]\{1,20}//g genres_list.txt`

Or replace | with linebreak

sed -i.bak -E s'/\s\|\s/\n/'g genres_list.txt

Combine multiple CSVs skipping headers

Where archives_n.csv all have the same headers, append additional files to the first file using the code below:

awk 'FNR-1' archives_2.csv archives_3.csv archives_4.csv >> archives_1.csv

To make a new version of archives_1.csv, use following:

cp archives_1.csv archives_base.csv; awk 'FNR-1' archives_2.csv archives_3.csv archives_4.csv >> archives_base.csv`

Combine Multiple PDFs with PDFTK

Or, how I combine my various scanned docs into one expense report. Very simple, just need to remember order of cat & output.

pdftk file1.pdf file2.pdf file3.pdf cat output newfile.pdf

I have not yet experimented with combining only some pages of multiple files, but it seems very doable, e.g.

pdftk A=file1.pdf B=file1.pdf cat A1-7 B1-5 A8 output combined.pdf

More documentation on interesting challenges.

Split File by Lines

Split a file by lines into files which are appended with letters. The -l flag is what triggers lines count. This can work on any kind of text-type file (and others, but won’t make sense). Example below is CSV, but see note.

split -l 500 all_tweets.csv tweets_

In the above, the file will split into things like tweets_aa.csv through tweets_az.csv, etc. onward.

Note: If there is a multi-line entry in the CSV in the wrong place, it may end up in both files. Review or be prepared to catch that error when processing.

Substitute Basic Strings in a File

To replace one string with another in all files in the directory, where the string to be found is “dc:creator#author” and the string to replace it with is “mrel:aut” run:

for f in $(ls); do sed -i -e 's/dc:creator#author/mrel:aut/g' $f; done

Note: Exercise careful consideration of what you’re finding and replacing. If you replace “placing” with “putting” you will also replace “replacing” with “reputting.”

File Management

Find Word, Character, or Space in Filename

Note, the way these scripts currently work, they’ll only change the first instance of the issue in your filename. The easiest way to check if any still exist in the directory (and note, this will also check subdirectories) is to run:

find . -name "*wordOrSpace*" | wc -l

where wordOrSpace is the word or space you’re looking for. This will give you an output count of how many files it still appears in. Normally, when I’m fixing something like a space.

Would love to see a simplified form which hits them all in one go, but it’s easier to run this quickly, run a fix, then run it again, and move on, than it is to completely rewrite it.

Substitute Word in Filename

Note: This script will not properly operate on files which have spaces in their filenames, see below. The search uses mv to replace a file with itself in the directory.

Where “Lebenen” is the misspelled word you want to change to “Lebanon” in this situation.

for badfile in $(find . -name "*Lebenen*"); do mv "$badfile" "`echo $badfile | sed 's/Lebenen/Lebanon/'`"; done

again, note, it only changes this once, so you’d run it twice if you expected to find it more than once

You can use the same format to switch all your underscores to hyphens, e.g.

for badfile in $(find . -name "*_*"); do mv "$badfile" "`echo $badfile | sed 's/_/-/'`"; done

or vice-versa.

Remove Spaces or Ampersands from Filenames

Note, the “find” method above returns filenames differently and in a way that doesn’t work well for handling spaces or ampersands. This renames ALL files using mv and substitutes the character if it finds it. Otherwise it simply replaces the file with itself.

Spaces:

for f in *; do mv "$f" "`echo $f | sed s/\ /_/`"; done

Ampersands:

for f in *; do mv "$f" "`echo $f | sed s/\&/_/`"; done

You can do it with other special characters as well.

You can change the \* into \*.jpg or any extension if you’re only targeting one kind of file. It will return errors for every filename which doesn’t contain the issue.

Very Messy Script to Move 1000 Files at a Time Into a Subdirectory

for file in $(ls -p | grep -v / | head -1000); do mv $file subdirectory/; done

(note: does tail -1000 work any faster?)

File Comparison

Compare two files with comm and sort

Nesting comm and sort to output the lines unique in file 1 and both files (see flags below).

comm -13 <(sort file_1.txt) <(sort file_2.txt)

Or without sort:

comm -13 file_1.txt file_2.txt
  • -1 suppresses lines unique to file 1
  • -2 suppresses lines unique to file 2
  • -3 suppresses lines found in both files

Additional documentation of comm.

Grep Data Diffs or Similarities Within Files

If nopenope.txt has catkeys of every record with an 019 which says “don’t edit this” and genre_keys_solo.txt has catkeys of every record which could get genre editing, then…

grep -x -f genre_keys_solo.txt nopenope.txt > nopetopus.txt

should search for genre record keys in the nopenope and output them (if they match) into nopetopus where I can check on them and use them to ensure we don’t edit any wrong files.

Flag -x is necessary to avoid inexact matches (matches whole line). Otherwise every catkey contained in another catekey will return. (x_x) Incidentally, adding a -v flag would get the differences.

Exiftool

Just getting image height and width from JPGs in a directory and outputting them with the filename

for file in *.jpg; do echo "$file"; exiftool -imagewidth -imageheight "$file" ; done

Pandoc

Used this to turn a batch of Markdown files into PDFs - had to mess a bunch with the engine.

for f in *.md; do pandoc $f --pdf-engine=xelatex \
  -V 'mainfont:DejaVuSans.ttf' \
  -V 'mainfontoptions:Extension=.ttf, UprightFont=*, BoldFont=*-Bold' \
  -o "`echo $f | sed s/md/pdf/`"; done

idloc

idloc get https://id.loc.gov/authorities/subjects/sh2002000569

Note: seems to work for both http (most appropriate) and https.

When searching, you can include limits and concept schemes:

idloc search "Semantic Web" --concept-scheme subject-headings --limit 5

to get concept schemes:

idloc concept-schemes