Collection of bash snippets which I had been keeping in gists and elsewhere. Some of these are hacky but they get the job done quickly. Updated and replaced when improved.
Curl and WGet
Create WARC of a Website (e.g. C4LJ)
Filenames and domains matter of course.
wget \
--mirror \
--warc-file=C4LJ_backup_05-05-2017 \
--warc-cdx \
--page-requisites \
--html-extension \
--convert-links \
--execute robots=off \
--directory-prefix=. \
--span-hosts \
--domains=journal.code4lib.org,code4lib.org \
--user-agent=Mozilla \
--wait=5 \
--random-wait \
http://journal.code4lib.org
List of LCCNs to MADSRDF
[note: this may need to be checked]
With 3 second sleep
Names:
#!/bin/bash
for lccn in nb2015017302 n89223874 n82203435 n2017074015
do
curl http://id.loc.gov/authorities/names/$lccn.madsrdf.rdf -o $lccn.rdf
sleep 3
done
Or alternatively subjects:
/authorities/subjects/
List of LCCNs to MARCXML
#!/bin/bash
for lccn in nb2015017302 sh2018002914 n89223874 n82203435 sh85033801
do
curl https://lccn.loc.gov/$lccn/marcxml -o $lccn.xml
sleep 1
done
Data Manipulation
Turn Text List Into Unique Ordered List
sort -u source_file.txt -o target_file.txt
(when using -o, the same file can be source AND target file)
Compare Arrays Write to File
for recordID in 12 16 19 20 22 29 33 35 36 37 43 44 45 47 2466 540 3437 1400 1897 2371 587
do
if [[ " 12 16 19 20 22 29 33 35 36 37 43 44 45 47 " == *\ $recordID\ * ]]
then
: #pass to the next
else
echo $recordID >> unreconciledRecordID.txt # Echoes all the record IDs from the array not found in the second array into this file. Appending.
echo " " >> unreconciledRecordID.txt # This is another way to space it out more. May or may not be desired.
fi
done
Echo PDF Metadata of Each PDF Into XML File
#!/bin/bash
for each in $(ls)
do cd "/mnt/curatend-batch/data/catholic-pamphlets/IA/"$each
for each in $(find . -name "*.pdf")
do pdfinfo -meta $each > "`echo $each | sed 's/.pdf/.xml/'`"
done
done
Create MD5 Checksums in All Subdirectories
This was initially written for HathiTrust submission specs so it may require review/revision before reuse.
for dir in $(ls /path/to/directories ); do find $dir -type f | grep -v "checksum.md5" | xargs md5 -r > $dir/checksum.md5 ; done
File Manipulation
Return Lines Before and/or After With Grep
Use -A 1 to return one line after the match (as well as match), -B 1 to return 1 line before match (as well as match), and C -1 to return 1 line on either side. For example:
grep "dewey_" -A 5 dewey_lopped.txt > dewey_only_lopped.txt
gives me the five lines after the field I’m looking for so I can take only the text blocks around the actual Dewey entries in this whole index file and make a more limited file to focus on Dewey. If there were a line I needed before, I could add a -B but it’s not needed here.
Test for Invalid UTF-8 in Files
Looks for anything which doesn’t match the regex pattern and returns those matching lines. The regex will only be looking for well-encoded data.
grep -axv '.*' metadata-1.csv
Zip All Directories in a Directory
for dir in $(ls /path ); do zip -r $dir $dir ; done
Imagemagick
Transform HEIC images to PNG:
mogrify -format png *.HEIC
(may need to run again for *.heic since sometimes they come over lower case.)
Make png background white:
convert -flatten scene_1.png sm_scene_1.png
Resize a bunch of TIFFs and make them JPGs when remove the .tif that ends up in file using method (note, must be done in working directory — I be there is a thing I can do with xargs but haven’t tried):
for file in *.tif; do convert "$file" -resize '2000' "$file".jpg; done
for file in *.jpg; do mv "$file" "`echo $file | sed 's/\.tif//'`"; done
Regex Replace in Files (e.g. 1-20 spaces, add line break etc)
Sed inline (-i) with regex (-E) and backup (.bak) to replace 1-20 spaces at the beginning of a line in a file:
sed -i.bak -E s/^[[:space:]]\{1,20}//g genres_list.txt
Replace digits with nothing
sed -i.bak -E s/^[0-9]\{1,20}//g genres_list.txt`
Or replace | with linebreak
sed -i.bak -E s'/\s\|\s/\n/'g genres_list.txt
Combine multiple CSVs skipping headers
Where archives_n.csv all have the same headers, append additional files to the first file using the code below:
awk 'FNR-1' archives_2.csv archives_3.csv archives_4.csv >> archives_1.csv
To make a new version of archives_1.csv, use following:
cp archives_1.csv archives_base.csv; awk 'FNR-1' archives_2.csv archives_3.csv archives_4.csv >> archives_base.csv`
Combine Multiple PDFs with PDFTK
Or, how I combine my various scanned docs into one expense report. Very simple, just need to remember order of cat & output.
pdftk file1.pdf file2.pdf file3.pdf cat output newfile.pdf
I have not yet experimented with combining only some pages of multiple files, but it seems very doable, e.g.
pdftk A=file1.pdf B=file1.pdf cat A1-7 B1-5 A8 output combined.pdf
More documentation on interesting challenges.
Split File by Lines
Split a file by lines into files which are appended with letters. The -l flag is what triggers lines count. This can work on any kind of text-type file (and others, but won’t make sense). Example below is CSV, but see note.
split -l 500 all_tweets.csv tweets_
In the above, the file will split into things like tweets_aa.csv through tweets_az.csv, etc. onward.
Note: If there is a multi-line entry in the CSV in the wrong place, it may end up in both files. Review or be prepared to catch that error when processing.
Substitute Basic Strings in a File
To replace one string with another in all files in the directory, where the string to be found is “dc:creator#author” and the string to replace it with is “mrel:aut” run:
for f in $(ls); do sed -i -e 's/dc:creator#author/mrel:aut/g' $f; done
Note: Exercise careful consideration of what you’re finding and replacing. If you replace “placing” with “putting” you will also replace “replacing” with “reputting.”
File Management
Find Word, Character, or Space in Filename
Note, the way these scripts currently work, they’ll only change the first instance of the issue in your filename. The easiest way to check if any still exist in the directory (and note, this will also check subdirectories) is to run:
find . -name "*wordOrSpace*" | wc -l
where wordOrSpace is the word or space you’re looking for. This will give you an output count of how many files it still appears in. Normally, when I’m fixing something like a space.
Would love to see a simplified form which hits them all in one go, but it’s easier to run this quickly, run a fix, then run it again, and move on, than it is to completely rewrite it.
Substitute Word in Filename
Note: This script will not properly operate on files which have spaces in their filenames, see below. The search uses mv to replace a file with itself in the directory.
Where “Lebenen” is the misspelled word you want to change to “Lebanon” in this situation.
for badfile in $(find . -name "*Lebenen*"); do mv "$badfile" "`echo $badfile | sed 's/Lebenen/Lebanon/'`"; done
again, note, it only changes this once, so you’d run it twice if you expected to find it more than once
You can use the same format to switch all your underscores to hyphens, e.g.
for badfile in $(find . -name "*_*"); do mv "$badfile" "`echo $badfile | sed 's/_/-/'`"; done
or vice-versa.
Remove Spaces or Ampersands from Filenames
Note, the “find” method above returns filenames differently and in a way that doesn’t work well for handling spaces or ampersands. This renames ALL files using mv and substitutes the character if it finds it. Otherwise it simply replaces the file with itself.
Spaces:
for f in *; do mv "$f" "`echo $f | sed s/\ /_/`"; done
Ampersands:
for f in *; do mv "$f" "`echo $f | sed s/\&/_/`"; done
You can do it with other special characters as well.
You can change the \* into \*.jpg or any extension if you’re only targeting one kind of file. It will return errors for every filename which doesn’t contain the issue.
Very Messy Script to Move 1000 Files at a Time Into a Subdirectory
for file in $(ls -p | grep -v / | head -1000); do mv $file subdirectory/; done
(note: does tail -1000 work any faster?)
File Comparison
Compare two files with comm and sort
Nesting comm and sort to output the lines unique in file 1 and both files (see flags below).
comm -13 <(sort file_1.txt) <(sort file_2.txt)
Or without sort:
comm -13 file_1.txt file_2.txt
-1suppresses lines unique to file 1-2suppresses lines unique to file 2-3suppresses lines found in both files
Additional documentation of comm.
Grep Data Diffs or Similarities Within Files
If nopenope.txt has catkeys of every record with an 019 which says “don’t edit this” and genre_keys_solo.txt has catkeys of every record which could get genre editing, then…
grep -x -f genre_keys_solo.txt nopenope.txt > nopetopus.txt
should search for genre record keys in the nopenope and output them (if they match) into nopetopus where I can check on them and use them to ensure we don’t edit any wrong files.
Flag -x is necessary to avoid inexact matches (matches whole line). Otherwise every catkey contained in another catekey will return. (x_x) Incidentally, adding a -v flag would get the differences.
Exiftool
Just getting image height and width from JPGs in a directory and outputting them with the filename
for file in *.jpg; do echo "$file"; exiftool -imagewidth -imageheight "$file" ; done
Pandoc
Used this to turn a batch of Markdown files into PDFs - had to mess a bunch with the engine.
for f in *.md; do pandoc $f --pdf-engine=xelatex \
-V 'mainfont:DejaVuSans.ttf' \
-V 'mainfontoptions:Extension=.ttf, UprightFont=*, BoldFont=*-Bold' \
-o "`echo $f | sed s/md/pdf/`"; done
idloc
idloc get https://id.loc.gov/authorities/subjects/sh2002000569
Note: seems to work for both http (most appropriate) and https.
When searching, you can include limits and concept schemes:
idloc search "Semantic Web" --concept-scheme subject-headings --limit 5
to get concept schemes:
idloc concept-schemes




