Working With JSON
Write JSON Content to File
Defines a function that writes JSON content (obtained via json.load) to a file (filename passed as newJson).
def write_JSON_file(content,newJson):
with open(newJson, 'w') as outfile:
json.dump(content, outfile, sort_keys=True, indent=4)
Read JSON Content from File
Opening a JSON file and loading it into a JSON object:
with open(fileName, "r") as myJSON:
recorddata = json.loads(myJSON.read())
Then use recorddata.
Test Whether Key in JSON
indexeddata = json.loads(myJSON.read())
if 'hathitrust_struct' in indexeddata:
## do stuff with indexeddata['hathitrust_struct']
else:
## it's not in ETAS
Appending to Lists and Dictionaries
Samples of how to append to lists and dictionary. Append to list:
newList = []
newList.append(value)
Append to dict:
newDict = {}
newDict['key'] = 'value'
Rebuild List as Unique Values
When a list contains multiple non-unique values, this rebuilds a list of just the unique values in the order in which they first appeared. In the case below, the list comes from a JSON record where the list is in “Sponsors.”
unique=[]
for funder in record["Sponsors"]:
if funder not in unique:
unique.append(funder)
One would then iterate through unique instead of record["Sponsors"] as one might’ve originally done.
Working With CSV & Pandas
CSV: DictReader
Creates a useful dictionary with header names as keys. Read, newline, and encoding aren’t really required but also can be useful in specifying things.
with open('database.csv','r',newline='',encoding='utf-8') as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
CSV: DictWriter
When writing rows, you can reference other CSVs you’re iterating through, JSON objects, strings, etc.
with open('arsenic_updates_books.csv', 'w', newline='') as csvfile:
fieldnames = ['catkey', 'winterthur_title','catalog_title','catalog_date','flag']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
### stuff, you'd end up with this being a loop or something and indented
writer.writerow({'catkey':record['id'], 'winterthur_title':original['title'],'catalog_title':record['title_245ab_tsim'][0], 'catalog_date':pub_date,'flag':"true"})
Pandas: Get Pandas Dataframe Header
Use below to print to console when working in terminal or set variables, etc.
list(df)
Pandas: Extract CSV where Rows Only Match One Value
import csv
import pandas as pd
df = pd.DataFrame(pd.read_csv("Parcels.csv",low_memory=False))
psu = df[df.University == 'Pennsylvania State University']
psu.to_csv('PSU_parcels.csv',index=False) #use this so as not to export the dataframe's index column
Pandas: Extract CSV With Only Certain Columns
From when I redacted my survey data to only extract non-identifying Qualtrics columns.
import pandas as pd
f=pd.read_csv("Where_Are_We_Now_Qualtrics_Export_Publication.csv")
keep_col = ['Q7','Q7_5_TEXT','Q8','Q10','Q10_9_TEXT','Q11','Q12','Q12_6_TEXT','Q13','Q14','Q15','Q16','Q16_6_TEXT','Q17','Q18','Q21','Q22','Q22_6_TEXT','Q23','Q23_6_TEXT','Q24','Q24_4_TEXT','Q25','Q26','Q26_3_TEXT','Q27','Q27_5_TEXT','Q28','Q29','Q30','Q31']
new_f = f[keep_col]
new_f.to_csv("Where_Are_We_Now_Redacted_Data.csv", index=False)
Pandas: Merge CSVs on Shared Key
see Merge documentation for more
import pandas
first = pandas.read_csv('dataset1.csv')
second = pandas.read_csv('dataset2.csv')
merged = pandas.merge(first, second, on='location_string')
merged.to_csv('output.csv')
XML to CSV
from lxml import etree
import csv
# A useful function to test how many nodes match a single xpath. e.g. if you have 3 dc:creator fields at 0, 1, 2, it will return 3. This is super helpful if you need to set up a condition for processing 1 element vs. multiples.
def lenTest(path,tree):
return len(tree.xpath(path))
# example
elemZero = tree.xpath(path)[0]
elemZeroValue = etree.tostring(elemZero, encoding='UTF-8', method='text')
# titleXPath is declared as a variable so you can switch to EAD 2002 if you want
# throws on a .strip() just in case your data is imperfect.
titleXPath = "/ead/control/filedesc/titlestmt/titleproper"
titleString = etree.tostring(tree.xpath(titleXPath)[0], encoding='UTF-8', method='text').strip()
Common Tasks
Glob files to make a list
Super simple:
import glob
files = glob.glob("just_names/*.json")
Compare String Similaritiy - Fuzzywuzzy Fuzz Ratio
from fuzzywuzzy import fuzz
fuzz.ratio("First","Furst") # 80
See fuzzywuzzy docs to see additional things like partial ratio, token sort ratio, token set ratio, and process.
Simple Regex Match
In this example, the first match is testing for a number and hyphen, e.g. the last two char of “Last, First, 1960-” so you grab the last 5 to have the whole year and hyphen for later matching. Otherwise, if it’s the end of a closed date, e.g. “Person, Older, 1840-1922” then it would grab the 1922. You could add another “else” for when it doesn’t match either or test for letters.
import re
if re.match(r"\d-",match[-2:]) != None:
end_matcher = match[-5:]
elif re.match(r"\d{2}",match[-2:]) != None:
end_matcher = match[-4:]
If there isn’t a match, re.match returns “None.” If there is a match, it returns something like:
<re.Match object; span=(0, 2), match='0-'>
(that match is for the 1960- example)
idloc
Some simple stuff copied from the idloc README for Python (I contributed the change from “name” to “title”). Much more useful for accessing and using the LC files.
import idloc
concept = idloc.get('http://id.loc.gov/authorities/subjects/sh2002000569')
for result in idloc.search('Semantic Web'):
print(result['title'], result['uri']
for result in idloc.search('Semantic Web', concept_schemes=['subject-headings', 'name-authority']):
print(result['title'], result['uri'])
get mapping of concept schemes:
idloc.CONCEPT_SCHEMES
ArchivesSpace and ArchivesSnake Reference
Python scripting focused primarily on metadata managements and updates, including ArchivesSpace scripting and other.
ASpace: Update Top Containers
Basics: Post the repositories/N/top_containers/batch/location with an array of IDs as the params and a location_uri variable. Requires admin rights. Example:
response = client.post('repositories/3/top_containers/batch/location', params={'ids': [40571, 40572], 'location_uri': '/locations/3722'} ).json()
Can be batched: The API expects to add a single location to every top container which is in that location (IDs array). Simplest way to create data in a CSV is pairing a location_uri with a pipe-separated list of all the top_container IDs.
location_uri,id
/locations/6423,6631
/locations/4025,24592|23842|23232
Key functions from the script that’s in my ASpaceASnake repo:
with open(csvName, newline='') as data:
reader = csv.DictReader(data)
for row in reader:
id_group = []
for id in row['id'].split("|"):
id_group.append(int(id)) # ASpace is really particular that it get a list of integers and no dang strings.
location = row['location_uri']
response = client.post('repositories/' + repo_num + '/top_containers/batch/location', params={'ids': id_group , 'location_uri': location } ).json()
ASpace: Update Authority_IDs If Don’t Exist
This assumes that when you downloaded your subjects, you didn’t prefix them with anything, so subject 9035 is 9035.json, etc. It will write out with a filestem of “new-” but you can also edit that to be something like “new/” + jsonFile, which would put it in a directory called new.
def write_URI(subject_id,uri):
jsonFile = subject_id + '.json'
subject = json.load(open(jsonFile))
newJson = 'new-' + jsonFile # to put in a directory called 'new', change the '-' to a '/'. And ensure the directory exists.
if subject.has_key('authority_id'):
if subject['authority_id'] == '':
subject['authority_id'] = uri
write_subject_JSON(subject,newJson)
elif subject['authority_id'] == ' ':
subject['authority_id'] = uri
write_subject_JSON(subject,newJson)
else:
else:
subject['authority_id'] = uri
write_subject_JSON(subject,newJson)
Miscellaneous Tasks
Script to Submit URL List to Wayback Machine
import requests, datetime, time
from datetime import date
def backupURLS(urls):
today = date.today()
results = str(today) + '-results.txt'
base_url = 'http://web.archive.org'
for url in urls:
r = requests.get(base_url + '/save/' + url)
with open(results,'a') as resultsfile:
if r.status_code == requests.codes.ok:
resultsfile.write('Success: archived url' + url + '\n')
else:
resultsfile.write('Error in response: ' + str(r.status_code) + 'URL: ' + url + '\n')
time.sleep(0.5)
# REPLACE THESE WITH YOUR REAL URLS AS A LIST.
urls = ['http://example.org', 'http://example2.org']
backupURLS(urls)




