Python Scripts and Snippets

Working With JSON

Write JSON Content to File

Defines a function that writes JSON content (obtained via json.load) to a file (filename passed as newJson).

def write_JSON_file(content,newJson):
  with open(newJson, 'w') as outfile:
    json.dump(content, outfile, sort_keys=True, indent=4)

Read JSON Content from File

Opening a JSON file and loading it into a JSON object:

with open(fileName, "r") as myJSON:
      recorddata = json.loads(myJSON.read())

Then use recorddata.

Test Whether Key in JSON

indexeddata = json.loads(myJSON.read())
if 'hathitrust_struct' in indexeddata:
  ## do stuff with indexeddata['hathitrust_struct']
else:
  ## it's not in ETAS

Appending to Lists and Dictionaries

Samples of how to append to lists and dictionary. Append to list:

newList = []
newList.append(value)

Append to dict:

newDict = {}
newDict['key'] = 'value'

Rebuild List as Unique Values

When a list contains multiple non-unique values, this rebuilds a list of just the unique values in the order in which they first appeared. In the case below, the list comes from a JSON record where the list is in “Sponsors.”

unique=[]
  for funder in record["Sponsors"]:
      if funder not in unique:
          unique.append(funder)

One would then iterate through unique instead of record["Sponsors"] as one might’ve originally done.

Working With CSV & Pandas

CSV: DictReader

Creates a useful dictionary with header names as keys. Read, newline, and encoding aren’t really required but also can be useful in specifying things.

with open('database.csv','r',newline='',encoding='utf-8') as csvfile:
  reader = csv.DictReader(csvfile)
  for row in reader:

CSV: DictWriter

When writing rows, you can reference other CSVs you’re iterating through, JSON objects, strings, etc.

with open('arsenic_updates_books.csv', 'w', newline='') as csvfile:
    fieldnames = ['catkey', 'winterthur_title','catalog_title','catalog_date','flag']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    ### stuff, you'd end up with this being a loop or something and indented
    writer.writerow({'catkey':record['id'], 'winterthur_title':original['title'],'catalog_title':record['title_245ab_tsim'][0], 'catalog_date':pub_date,'flag':"true"})

Pandas: Get Pandas Dataframe Header

Use below to print to console when working in terminal or set variables, etc.

list(df)

Pandas: Extract CSV where Rows Only Match One Value

import csv
import pandas as pd

df = pd.DataFrame(pd.read_csv("Parcels.csv",low_memory=False))

psu = df[df.University == 'Pennsylvania State University']

psu.to_csv('PSU_parcels.csv',index=False) #use this so as not to export the dataframe's index column

Pandas: Extract CSV With Only Certain Columns

From when I redacted my survey data to only extract non-identifying Qualtrics columns.

import pandas as pd
f=pd.read_csv("Where_Are_We_Now_Qualtrics_Export_Publication.csv")
keep_col = ['Q7','Q7_5_TEXT','Q8','Q10','Q10_9_TEXT','Q11','Q12','Q12_6_TEXT','Q13','Q14','Q15','Q16','Q16_6_TEXT','Q17','Q18','Q21','Q22','Q22_6_TEXT','Q23','Q23_6_TEXT','Q24','Q24_4_TEXT','Q25','Q26','Q26_3_TEXT','Q27','Q27_5_TEXT','Q28','Q29','Q30','Q31']
new_f = f[keep_col]
new_f.to_csv("Where_Are_We_Now_Redacted_Data.csv", index=False)

Pandas: Merge CSVs on Shared Key

see Merge documentation for more


import pandas

first = pandas.read_csv('dataset1.csv')
second = pandas.read_csv('dataset2.csv')

merged = pandas.merge(first, second, on='location_string')
merged.to_csv('output.csv')

XML to CSV

from lxml import etree
import csv

# A useful function to test how many nodes match a single xpath. e.g. if you have 3 dc:creator fields at 0, 1, 2, it will return 3. This is super helpful if you need to set up a condition for processing 1 element vs. multiples.

def lenTest(path,tree):
  return len(tree.xpath(path))

# example

elemZero = tree.xpath(path)[0]
elemZeroValue = etree.tostring(elemZero, encoding='UTF-8', method='text')

# titleXPath is declared as a variable so you can switch to EAD 2002 if you want
# throws on a .strip() just in case your data is imperfect.

titleXPath = "/ead/control/filedesc/titlestmt/titleproper"
titleString = etree.tostring(tree.xpath(titleXPath)[0], encoding='UTF-8', method='text').strip()

Common Tasks

Glob files to make a list

Super simple:

import glob

files = glob.glob("just_names/*.json")  

Compare String Similaritiy - Fuzzywuzzy Fuzz Ratio

from fuzzywuzzy import fuzz

fuzz.ratio("First","Furst") # 80

See fuzzywuzzy docs to see additional things like partial ratio, token sort ratio, token set ratio, and process.

Simple Regex Match

In this example, the first match is testing for a number and hyphen, e.g. the last two char of “Last, First, 1960-” so you grab the last 5 to have the whole year and hyphen for later matching. Otherwise, if it’s the end of a closed date, e.g. “Person, Older, 1840-1922” then it would grab the 1922. You could add another “else” for when it doesn’t match either or test for letters.

import re

if re.match(r"\d-",match[-2:]) != None:
  end_matcher = match[-5:]
elif re.match(r"\d{2}",match[-2:]) != None:
  end_matcher = match[-4:]

If there isn’t a match, re.match returns “None.” If there is a match, it returns something like:

<re.Match object; span=(0, 2), match='0-'>

(that match is for the 1960- example)

idloc

Some simple stuff copied from the idloc README for Python (I contributed the change from “name” to “title”). Much more useful for accessing and using the LC files.

import idloc

concept = idloc.get('http://id.loc.gov/authorities/subjects/sh2002000569')
for result in idloc.search('Semantic Web'):
    print(result['title'], result['uri']
for result in idloc.search('Semantic Web', concept_schemes=['subject-headings', 'name-authority']):
    print(result['title'], result['uri'])

get mapping of concept schemes:

idloc.CONCEPT_SCHEMES

ArchivesSpace and ArchivesSnake Reference

Python scripting focused primarily on metadata managements and updates, including ArchivesSpace scripting and other.

ASpace: Update Top Containers

Basics: Post the repositories/N/top_containers/batch/location with an array of IDs as the params and a location_uri variable. Requires admin rights. Example:

response = client.post('repositories/3/top_containers/batch/location', params={'ids': [40571, 40572], 'location_uri': '/locations/3722'} ).json()

Can be batched: The API expects to add a single location to every top container which is in that location (IDs array). Simplest way to create data in a CSV is pairing a location_uri with a pipe-separated list of all the top_container IDs.

location_uri,id
/locations/6423,6631
/locations/4025,24592|23842|23232

Key functions from the script that’s in my ASpaceASnake repo:


with open(csvName, newline='') as data:
    reader = csv.DictReader(data)
    for row in reader:
        id_group = []
        for id in row['id'].split("|"):
            id_group.append(int(id))  # ASpace is really particular that it get a list of integers and no dang strings.
        location = row['location_uri']
        response = client.post('repositories/' + repo_num + '/top_containers/batch/location', params={'ids': id_group , 'location_uri': location } ).json()

ASpace: Update Authority_IDs If Don’t Exist

This assumes that when you downloaded your subjects, you didn’t prefix them with anything, so subject 9035 is 9035.json, etc. It will write out with a filestem of “new-” but you can also edit that to be something like “new/” + jsonFile, which would put it in a directory called new.

def write_URI(subject_id,uri):
    jsonFile = subject_id + '.json'
    subject = json.load(open(jsonFile))
    newJson = 'new-' + jsonFile # to put in a directory called 'new', change the '-' to a '/'. And ensure the directory exists.
    if subject.has_key('authority_id'):
        if subject['authority_id'] == '':
            subject['authority_id'] = uri
            write_subject_JSON(subject,newJson)
        elif subject['authority_id'] == ' ':
            subject['authority_id'] = uri
            write_subject_JSON(subject,newJson)
        else:
    else:
        subject['authority_id'] = uri
		write_subject_JSON(subject,newJson)

Miscellaneous Tasks

Script to Submit URL List to Wayback Machine


import requests, datetime, time
from datetime import date

def backupURLS(urls):
    today = date.today()
    results = str(today) + '-results.txt'
    base_url = 'http://web.archive.org'
    for url in urls:
        r = requests.get(base_url + '/save/' + url)
        with open(results,'a') as resultsfile:
            if r.status_code == requests.codes.ok:
                resultsfile.write('Success: archived url' + url + '\n')
            else:
                resultsfile.write('Error in response: ' + str(r.status_code) + 'URL: ' + url + '\n')
        time.sleep(0.5)

# REPLACE THESE WITH YOUR REAL URLS AS A LIST.
urls = ['http://example.org', 'http://example2.org']

backupURLS(urls)