Spring Check-In

In February, when we realized this year's Ruby and Rails EOLs had significant implications for our shop, I tilted a lot of effort towards getting blacklight 7.x "in order" for our migration.

I am almost seeing the daylight of "and now we can migrate our remaining apps on to Ruby 3/Rails 6+/Blacklight 7+", but this effort, combined with the drumbeat of project development work and team management... it has been a really intense spring.

Along the way I've become a maintainer of code4lib/ruby-oai, where there are now a tiny handful of inherited PRs (3, with unclear interest) and tickets (6, one of which is Ruby 3 compatibility and a major rev) - after hacking through 12 unattended PRs running back to 2015 or so. This also took me on a brief detour into configuring shared-cache SQLite and ActiveRecord, though the tests that initiated it turned out to be unnecessary. 

We've cut a release of Blacklight's OAI provider making use of that ruby-oai release (now down to a single ticket for cutting an additional release at difference ruby/linter requirements).

We're plowing through a bunch of Blacklight tickets - there's a pretty dramatic dent in the open PRs, and I think the tickets that apply to 7.x with consensus support are all merged.

That's leaving the release of a Blacklight 7-friendly refactor of the blacklight_range_limit plugin, which is nearly done.

I'm fairly certain that the "rampage of feckless merge commits" (thanks code4lib#blacklight) around all this has annoyed my co-committers to no end; I can verify that more than one direct message has essentially said "I cannot remember why I opened that PR". I think, though, that cleaning it all out makes the project(s) more approachable in addition to being required work for our team.

Trey Pendragon has made a stalwart effort towards getting the Blacklight committers on a regular meeting schedule again, and I'm slowly trying to re-articulate myself to some form of the Samvera conversation. Somehow in all this I've also been volunteering on a DLF planning committee and recruiting, but fingers crossed that link will be dead soon and I'll be able to rejuvenate our efforts towards the other vacancy on our team.

I am hopeful that all this also indicates a mutual, if unvoiced, recognition and rejection of the malaise of the last two (or five!) years. On the other hand: At the end of last summer, my boss of 15 years announced his retirement at year's end. I intended to apply for that job, but it's gone unposted. I have to admit the possibility, then, that a part of all this is a sublimation of the anxiety and frustration of that situation - and with a backdrop of pandemics, wars, and dismaying politics, it's very easy to slip into thinking that work is the air around you.

I've maintained some resolve towards weaning myself off Twitter as the way to stay on top of both professional developments and distant friendships - cleaned up the feed, and just not posted. Hilariously, thanks to the aforementioned recruiting, I've had a taste of the LinkedIn experience and... it is not for me. I've experimented with mastodon, but I want to make an effort towards the blog - and frankly, towards slower interactions. The summer will tell how successful that is.

ActiveRecord, SQLite, URI Database name tokens

Blogging an answer I posted at Stack Overflow:

I ran into an issue recently with a library I was testing - the in-memory SQLite database wasn't shared between threads in the testing process. This can be accommodated in SQLite with a shared cache, and this should be usable in ActiveRecord by configuring the database connection with a URI filename... but it wasn't working.

If your SQLite build did not set the SQLITE_USE_URI flag to true, then after SQLite v3.38.2 it will default to false. As the folks at S/O observe, you will see files created with the name of the URI token.

You can work around this by passing the appropriate bit switches to SQLite via the flags parameter on your ActiveRecord connection.

In particular, you will want:

• SQLite3::Constants::Open::READWRITE (0x02)
• SQLite3::Constants::Open::CREATE (0x04)
• SQLite3::Constants::Open::URI (0x40)

... which is to say:

ActiveRecord::Base.establish_connection {

adapter: "sqlite3",

database: "file::memory:?cache=shared",

flags: 70 # SQLite3::Constants::Open::READWRITE | CREATE | URI

}

 

Featured Searches in a Blacklight/Solr Web Application

We've been working on a project to allow a Solr-backed application (an institutional repository built on Blacklight) to display configured "search features" - something akin to the breakout information boxes in a search engine - when a search is strongly affiliated with an organizational partner/journal/etc.

Rather than trying to predict actual searches that should be associated with a feature, we decided to leverage the facets in a given result set - the Search Feature is associated with a faceted field and one or more values. When a search is executed and there are facet values matching results in excess of a 'component threshold' (for example, 16% of the result set), we query a database for matched Features, and compare the aggregated tallies for the Feature to a display threshold (pretty high - 80% or more in testing).  This might be obvious to Blacklight veterans, but sorting the facets in the result set by count rather than by value is what makes this all possible.

Working against the analyzed result set rather than a predetermined set of queries permits the display of a Feature to be more emergent (for example, catching common acronyms for an academic unit or journal), but still accommodates a 'stable' link to a Feature that redirects to a search using the associated facet values as a filter query.

The data model for a Feature is pretty simple in its initial iteration - a slug identifier, a category (which maps to a faceted field), a description, links to a logo and external web site, and the associated facet values. We use the Feature data in two contexts - in the search results (with a compact display that can be expanded to show the description), and in "explore" pages presenting all the features for some categories. The application in question already has some authorization-restricted pages, so we were able to stand up a simple CRUD user interface for the features allowing us to delegate content management.

The stable links for features take advantage of duck-typing and Blacklight's deep-hash configuration to allow establishment of a filtered search context without precluding further filtering on the facet associated with a Feature's category: We define a query facet, but configure it not to display. Rather than an explicit query hash (which would be used to write out user-selectable values in a displayed facet), we have a "lazy" query proxy that implements the bracket method and builds named filters based on the configured facet values for a Feature, retrieved by the slug.

We anticipate ongoing work in scoring the Features - the limited data to begin with means very simple rules like "the top feature from each category" are sufficient to get us started - and in content management - making the descriptions Markdown seems likely in the foreseeable future. I'm interested to see how this develops in use, particularly in the context of some important counterpart efforts: We're also developing a reusable search "widget" to surface content associations in the university's centralized web content management system, and we are leveraging OJS's SWORD plugins to deposit articles from hosted journals immediately on publication (the hosted journals are all Features). Together these efforts suggest an intriguing capacity for our institutional repository to function as a partner platform.

Our IR is developed in a public source repository, so if you're interested in tracking this effort as a Blacklight developer you can find us on Github: https://github.com/cul/ac-academiccommons
 

Follow up on pulling Internet Archive ebooks data for reuse

Following up on a recent post - and making belated good on a promise to a colleague, sorry Alex!

1. I touched on pagination in that post, but didn't mention sorting! IA's search api won't have predictable response order without specifying some kind of sort.
2. I put together a Python script that I think embodies what I wanted to document in that post.

I usually work in Ruby (or, you know - bash) these days, but I'm trying to knock the rust off with Python, for a few reasons:

1. It's the preferred language of my code-friendly colleagues, and I prefer to be able to just share an annotated script for tasks of mutual interest.
2. I've been playing around a little with rendering 3D models, and my harrowing experience with OpenCV and Ruby motivates me to just use the dang Python bindings for, say, Blender. Could you imagine, writing some FFI rig for Blender? No, let's just get on with it, thanks.
3. I got rusty! You can't get rusty. I realize I've just summoned a Java project into my future.

Numpy Surprise: "Non-string object detected for the array ordering. Please pass in 'C', 'F', 'A', or 'K' instead"

You've got to hand it to the numpy contributors, it's a pretty forthright error message.
If you are working with some old python and encounter:

ValueError: Non-string object detected for the array ordering. Please pass in 'C', 'F', 'A', or 'K' instead

... then this might be helpful: Numpy's ndarray order is an enum in the C API, and prior to v1.4.0 (!!!) python clients passed the corresponding value constants as arguments to methods like flatten (for example) directly.

In v1.4.0 there's an argument parser introduced that maps the strings from the error messages to the enum values. It's pretty straightforward to do the mapping if you know what changed and look up the enum, but for convenience's sake:

pre-1.4.0 argument	Order Enum	post-v1.4.0 argument
-1	NPY_ANYORDER	"A"
0	NPY_CORDER	"C"
1	NPY_FORTRANORDER	"F"
2	NPY_KEEPORDER	"K"

I'm sure no one else out there is looking at decade-old python, but just in case.

Rough and Ready Guide to Pulling Columbia University Libraries eBooks from IA for Reuse

As the basis for the examples here, I am referring to Columbia University Libraries' Muslim World Manuscripts upload, browsable at https://archive.org/details/muslim-world-manuscripts; but you might also identify a collection in the "Collection" facet at https://archive.org/details/ColumbiaUniversityLibraries.

There is a search UI for archive.org, and a pretty nice python client, too; but I will shortcut here to the resulting search URL for a collection identified above in the URL segment after /details/ (which produces a 'q' parameter value of "collection:muslim-world-manuscripts"):

https://archive.org/advancedsearch.php?q=collection%3Amuslim-world-manuscripts&sort%5B%5D=&sort%5B%5D=&sort%5B%5D=&rows=50&page=1&output=json&callback=

(this example removes the default JSONP callback value for clarity as json)

This response is paginated (in this example 50 docs per page from the 'rows' parameter, pages numbered from 1 in the URL by the 'page' parameter).

Parse returned JSON; referred to from here as parsed_json.

The total number of rows is available at parsed_json['response']['numFound'] - use this to determine how many pages of content there are, or to try to fetch them in one page (if it's a modest number).

Iterate over docs at parsed_json['response']['docs'] -

Docs will generally have a link back to CLIO under the key 'stripped_tags'; if you can match the pattern /http:\/\/clio.columbia.edu\/catalog\/([0-9]+)/ then appending '.marcxml' will allow you to download more detailed metadata from CLIO.

If stripped_tags does not provide this information, many (but not all) CUL docs have an identifier format that indicates a catalog id, e.g. ldpd_14230809_000 - the middle part of the id, delineated by underscores ('_'), is a record identifier usable in the CLIO url patterns above in place of the grouped match (the last segment before the added '.marcxml').

Absent that, item details are available at archive.org as detailed below. There's also some metadata included in the docs from IA, but outside of 'date' it is often aggregated in an array/list under the tag 'description'. Some of the list members may have a field-like prefix (eg, "Shelfmark: "), but the data from CLIO (if available) will be more certain.

each doc will have a value under the key 'identifier' which can be use for downloading content from IA:

metadata details: "https://archive.org/metadata/#{identifier}" (see also the metadata API docs)
thumbnail: "https://archive.org/services/img/#{identifier}"
poster: "https://archive.org/download/#{identifier}/page/cover_medium.jpg"
details: "https://archive.org/details/#{identifier}"
embeddable viewer (iframe): "https://archive.org/stream/#{identifier}"?ui=full&showNavbar=false"
download pdf: "https://archive.org/download/#{identifier}/#{identifier}.pdf"

A Job on a Team I know Very Well

This post will be elaborated ASAP, but I'm excited to start describing a job on the team I manage:

https://opportunities.columbia.edu/en-us/job/506308/developer-for-digital-research-and-scholarship

The principal portfolio is in tooling to support DH projects; the predecessor, Marii Nyröp, is the developer behind the Wax static site generator: https://minicomp.github.io/wax/

It's a junior position with the most forgiving experience prerequisites we could manage, but I like to think our team has a track record of mentorship and professional development.

The incumbent would join a team with a lot of experience and an appetite for learning and improving.  We're in the midst of our first big push towards React/GraphQL over Rails and Solr. We use (and have imminent plans to elaborate our implementation of) IIIF.  There's a Software Carpentries program with certification opportunities. More soon!

The Island and the Archipelago

This post does not begin with an orthogonal observation about NYC as archipelago, but: Close Rikers.

On 22 August 2019 I sat in on a meeting of the Archipelago Advisory Board at METRO.

METRO developers describe Archipelago as:

... an evolving Open Source Digital Objects Repository / DAM Server Architecture based on the popular CMS Drupal8/9 and ... a mix of deeply integrated custom-coded Drupal8 modules (made with care by us) and a curated and well-configured Drupal8 instance, running under a discrete and and well-planned set of service containers. All of this driven by a clear and concise but thoughtfully planned technical roadmap.

Archipelago was dreamt as a multi-tenant, distributed, capable system (as its name suggests!) and can live isolated or in flocks of similar deployments, sharing storage, services, or -- even better -- just the discovery layer. Learn more about the different Software Services used by Archipelago.

Archipelago's primary focus is to serve the GLAM community by providing a flexible, consistent, and unified way of describing, storing, linking, exposing metadata and media assets. We respect identities and existing workflows.

All of this operates under a different concept than the one we all have become used to in recent times.

I think this is a compelling project, but I want to push on that last sentence a bit.

Archipelago might be summarized as collapsing several parts of repository application system in the early 2010s mode - certainly the management application and the repository itself, possibly the reader/researcher-facing publications - into a Drupal management application. Part of this is accomplished by recognizing that at least some of the nuts-and-bolts work of the repository have been subsumed into services - for example, S3-bucket as storage API or a DOI source as an identification API.

Archipelago eschews a system-determined schema for objects in favor of json (I think json-ld, actually) as an object storage format, and leveraging the object interpretation of the stored json to expose the objects to Twig templates.

In the subsumption of the repository into a management and publication tool, Archipelago tracks with Princeton's Figgy (and, I might add, the work our team does at Columbia on Hyacinth, although we still publish to a Fedora installation). It is also not an alien trajectory to the one Stanford's SDR has been on - or Duke's Digital Repository, which was also recently redesigned more completely around Drupal.

A talk I gave at Open Repositories in- 2016? I forget when. The Dublin one, where the video disappeared. Anyway, in a talk about the future of Fedora Commons and APIs, I briefly digressed into the virtues of CDL's curation microservices (to the surprise, I think, of the CDL delegation) - but I think it's clear that even if Merritt didn't per se change the way we all go about this work, the footprints (feetprint?) of S3 and Datacite across digital libraries suggests that the disarticulation of the repository into a process of services has happened - a trend that continues in Archipelago, which disarticulates a storage service, a description service, and an index service/API (Solr in particular, but the particulars are not necessary or even especially interesting to me this morning).

Listening to the METRO folks (that is, the inimitable Diego Pino) discuss Archipelago's templating system, I found myself reconsidering along these lines the Fedora Commons 3 Content Model Architecture. No, seriously!

The CMA was an elaboration of Sandy Payette and Karl Lagoze's Flexible and Extensible Digital Object Repository Architecture (that's right, F E D O R A) disseminators into a quasi-SOAP, aspirationally object-oriented set of behaviors specified as best they could be in other repository objects, and linked with RDF assertions between the content-bearing objects and their linked type-defining objects. In a frictionless world, this is an excellent model.

Unfortunately, the Fedora 3 CMA was deployed in the frictional world of J2EE. The practical constraints of Fedora-side implementation meant that the syntax was arcane, fragile, and expensive to run (as the linked services, hidden behind REST-fully accessed object "property" URLs, made calls back to the repository to get the information they needed from the object for which they were building a response while the client waited and so on). Like the Handle architecture (that's right, I said it) things weren't necessarily this way - but the social, organizational and platform considerations of the day determined them.

Not very long after, the Hydra project (staffed by Fedora Commons committers, growing out of the Blacklight project at Virginia, and aiming to manage and index Fedora content) would begin developing what might be understood as an overlay approach to disseminators in Rails apps. While the mixins and gems of the resulting Rails framework might themselves not seem to track towards Twig templates, moving the environment of development into a platform (see also Islandora on Drupal and Emory University's analogous work on Django) that has more front-end concerns and a diversity of dynamically evaluated template options strikes me as a necessary conceptual step towards them. Or, if not necessary, supported by being less horrifying than storing JSP and recompiling them to operate against your object exposed as JAXB somehow. A chill runs up my spine.

This is all to say that I see the Archipelago project as being on a vector of repository work (and as noted above, not alone on it) that intersects previous work. It's not the only vector that does - we remain in a holding pattern about the distinct repository at my place of work, and I think there's service preservation/sustainability arguments that can be made for it, to say nothing of performance considerations - but I think it makes interesting observations about where to locate and value the labor of running, managing, and publishing digital collections. Its design approach also makes a claim about what the optimal balance of abstraction and community of practice is. I'm interested to see where it goes from here.

Drifting

In a post reflecting on the software development practice in the Hydra/Samvera community, Jonathan Rochkind begins a late pivot towards a more general complaint by framing Samvera:

And finally, a particularly touchy evaluation of all for the hydra/samvera project; but the hydra project is 5-7 years old, long enough to evaluate some basic premises. I’m talking about the twin closely related requirements which have been more or less assumed by the community for most of the project’s history:

1) That the stack has to be based on fedora/fcrepo, and

2) that the stack has to be based on native RDF/linked data, or even coupled to RDF/linked data at all.

I believe these were uncontroversial assumptions rather than entirely conscious decisions, but I think it’s time to look back and wonder how well they’ve served us, and I’m not sure it’s well.

This 5 sentence history of Hydra/Samvera is a fabrication. The Hydra project began in 2008 as attempt to combine a Blacklight discovery layer and a Fedora 3 repository, debatably with the goal of improving the notion of services/disseminators in the Fedora 3 CMA by making them contained applications. The Fedora Commons project was one of its original partners. It's strange to characterize that backend as an assumption rather than the motivating use case when the core library from the project's onset is ActiveFedora (published February 2009).

I'm more sympathetic to interrogating the relationship of Samvera to linked data, but casting that decision as an assumption- rather than the conscious development goals of Hydra/Samvera partners who were trying focus their descriptions less on XML serializations and more on the description as data- is patronizing. I can agree that we should "look back and wonder how well they’ve served us", but it's always been the time to do that (as far as I can tell, the ActiveFedora:RDF package was introduced in 2013 as a reaction to frustration managing object descriptions as files). If I were an employee at Penn State, whose work prompted the accommodation of RDF as ActiveRecord-style properties rather than as serialized files, I'd be insulted to see my work characterized as the product of not "entirely conscious decisions" or "uncontroversial assumption[s]".

At more than one meeting now (in the interest of disclosure: I gave a talk on related topics at OR2016, participated in two panels touching on the issue at Hydra Connect 2016, and have been involved for some years with the Fedora Commons community and project), there's been open discussion of what the relationship of Samvera to Fedora ought to be going forward. It's clearly a question motivating the work on Valkyrie at Princeton. Over the years there's been more than one alternate backend written to mimic the Fedora APIs. There's analogous conversations in the world of Blacklight when a potential installer wants to use a different noSql store than Solr.

The critical question underlying those efforts and conversations is to what degree the software products of these various projects should be shareable, whether the surface of interaction is within a platform, across a shared index, across API abstractions, or at the achievement of consensus around use case and functional requirements. Rochkind takes a different tack, suggesting that a hard pivot away from abstraction should be the baseline and arguing that we need to justify any commitment beyond Rails and Paperclip. This strikes me as reductive and dismisses of my own experience: that generalizing description in a database moves pretty quickly towards re-inventing RDF in tables, and that storing blobs of serialized description leads to re-inventions of Fedora without the mediating APIs. If our response to the problems motivating Rochkind's post were to advocate interacting directly with the backing databases and file systems of Fedora, it might work - it might be faster! - but we would certainly not be proposing it as a minimalist path towards more sustainable software approaches.

We can scrutinize our approaches to the problem of managing assets and description in shareable ways without a fabulous and dismissive historical framing of the project and the use cases of its participating institutions. But we should also be cognizant of what *some kind* of abstraction yields: An Avalon or a Charon can function as a common tool to originate content subsequently repurposed for independent, locally developed publication platforms; I still think we'll inch towards shared practice, and thus shared content, with Islandora. Integrative projects like this require some kind of interface- the question is where to locate it.

Just a bunch of ESTC Library Names

Following Meaghan Brown on trying to match STC and ESTC library names, I threw together a quickie ruby script that parses all the library names from the ESTC library name browse list, then follows the "Next Page" links while they are present and grabs the next set.

Fetching a DEEP record, ESTC in hand, Part 2

Having been recommended the Database of Early English Plays (DEEP) as a source for descriptive metadata, and mapped our ESTC citations to a STC/Wing number, we can look at programmatically querying DEEP.

DEEP as currently released (the 2007 project) is a PHP search app in which a single resource (search.php) presents the query interface and the results, switching modes according to the HTTP request semantics and form fields. This first is important: As far as I can tell, DEEP requires the form fields to have been POSTed as multipart data, issuing a GET or POST to a DEEP URL composed with query parameters will only return the search page.

The DEEP Search Interface

Looking under the hood of the DEEP search interface requires more than just viewing the source: Some javascript manipulates the form based on user input (in large measure to steer queries on fields with a controlled list of values, like author), but you can get a picture of the effective source in a browser that supports DOM inspection. For example, in Chrome, you can control- or right-click in the search interface, and select 'inspect element'. If we've selected 'STC / Wing Number' as the the search type, we'll see something like this in the inspected source:

That tells us quite a bit about how search.php works, but for our purposes we are concerned about only 2 of the fields:

1. terms[0][type], which we want to be 'stc_or_wing'
2. terms[0][val], which we want to be the STC number we're searching for

The other fields pertain to adding a second query, how many results are returned, and how the results are sorted. We're assuming the simplest case (unique match between STC number and description), so the other fields are not relevant (and importantly, not required by the PHP script).

Once we've sorted out these basics, it's not very difficult to execute a DEEP query outside the browser. Here, for example, is some BASH calling cURL, executable from the terminal window or in a shell script:

BASE_URL="http://deep.sas.upenn.edu/search.php" 

FORM="" 

FORM="$FORM --form terms[0][type]=stc_or_wing"

FORM="$FORM --form terms[0][val]=$1"

curl $FORM $BASE_URL

... where $1 is replaced with the STC/Wing number we are searching for. In the case of cURL, you might also use the --data parameter; this would require concatenating them into a single value separated by ampersands (&). If you were using Python and the requests library, something like:

payload = {}

payload['terms[0][type]'] = 'value1'

payload[ 'terms[0][val]'] = 'some STC number'

r = requests.post("http://deep.sas.upenn.edu/search.php", data=payload)

... should work, too.

Parsing the DEEP Results

First, a note of thanks: The creators of DEEP encode its search results in XHTML, a variant of HTML that further requires documents to be valid XML. Although not necessary to produce HTML that's valid XML, it's a sign that the creators care about the documents being parse-able with less effort.

As you will be able to see from the output in your terminal, DEEP presents its search results in a TABLE element with the id 'searchresults'. This table has a row (TR) of column labels (id = 'headerrow'), and then presents the search results as pairs of rows, the first with a class 'record', and the second (containing the details of the description) with no class immediately following. If we were parsing this content with (for example) an XPath utility, we would iterate over:

//table[@id='searchresults']/tr[@class='record']

... and refer also to the next sibling of the TR element in our node handling.

The record rows contain the author (./td[@class='authorname']) and title (./td[@class='playname']). The description is a little more difficult to parse, since the nested div elements in that row present data adjacent to a span[@class='label'] whose content indicates the type of data (e.g. "Greg #"), and are followed by a text node containing the data.

The Other DEEP Interface

One of the fields in that row of description is labelled 'DEEP #', and it is tempting to think that this number might be used to refer directly to a document via the unadvertised single-record view at URLs of the form:

http://deep.sas.upenn.edu/viewrecord.php?deep_id={deep_record_number}

Unfortunately, the DEEP citation number is not (yet) the linkable record number, which appears to be a surrogate key from the backing database. However, if the STC number you've searched for has "contained" descriptions, those descriptions are linked in the div labelled 'Collection contains:'. Each of these anchor elements (span[@class='label' and text()='Collection contains:']/../a) has a javascript URI calling a function in its href attribute, and parsing the number argument to that function will provide you with the id necessary to fetch the contained descriptions with the viewrecord.php script. Conversely, from those descriptions you can mine the viewrecord.php id of the original collection: Follow the same pattern as before, but look instead for the label 'In Collection:'.

Quick Observations on CAP and Graduate Student Loans

First, please take a look at Scott Weingart's piece on journalism, charts/visualizations, data, and viral texts.

Several articles (eg Washington Post: "These 20 schools are responsible for a fifth of all graduate school debt" July 9; Yahoo: "20 schools account for $6.6 billion of U.S. government grad student loans", July 10) were in circulation this week, apparently re-written from Elizabeth Baylor's piece for the Center for American Progress in the Chronical of Higher Education (paywalled, "As Graduate-Student Debt Booms, Just a Few Colleges Are Largely Responsible", July 8). The source data is available online: It's from the Title IV Program Volume Reports, Loan Volume -> Direct Loan Program -> AY 2013-2014 Q4 (second sheet is award year summary, look at columns T and AD).

These data don't break the awards down by program; in this context, the downstream claims from WaPo:

What’s striking about the Center’s findings is that a majority of the debt taken to attend the 20 schools on its list is not for law or medical degrees that promise hefty paydays. Most graduate students at those schools are seeking master’s degrees in journalism, fine arts or government, according to CAP.

... look a little fishy. The CAP claim is more nuanced (emphases mine):

But it appears that a majority of debt taken on to attend those institutions is not for costly law or medical degrees, but for nonterminal degrees. Among the 20 institutions responsible for the most graduate debt, 81 percent of graduate degrees conferred in the most recent year were master’s degrees. ... As at other universities on the list, most graduate students at those institutions earn master’s degrees, in disciplines like journalism, fine arts, government, and the sciences.

This makes no claim about the relationship of Master's degrees overall to that student debt (though it implies something), nor about proportion of Master's programs to debt. It is worth noting, in these STEM reform times, that WaPo drops "the sciences" from its enumeration of implicitly-blamed programs.

The question raised for me, considering how suspicious the implications about degree programs and debt are, is what percentage of the debt carried by graduate students (especially at the private non-profits) is actually in the more expensive professional schools and "executive" Master's programs.

Fetching a DEEP record, ESTC in hand, Part 1

Pretext

I recently began working with a Early Modern English corpus in which the bibliographic metadata was sparse, but all the items were identified with an ESTC number. I was recommended the Database of Early English Playbooks (DEEP) as a source of description, but ran into problems almost immediately...

Wait: Let me pause here and be clear: These are valuable online resources doing important work in support of scholarship. When I say "problems", I mean "inconveniences for someone trying to teach a script or program to process". As valuable as they are, these resources are a little long in the tooth- the DEEP web interface I describe is from 2007, the ESTC interface of unclear vintage. Applying the linkable data, mashable API aesthetics of the mid-00's to them is not fair. And yet, here I am with work to do. So:

Problems!

1. DEEP is searchable by STC/Wing numbers, but not ESTC citation numbers
2. DEEP is navigated through pop-ups and what not, and requires POSTed form data to search.

Getting the STC/Wing Numbers

To get the STC/Wing numbers I went to the source the presumptive online source of record: The ESTC online at http://estc.bl.uk/. This interface supports permalinks to identifiers based on the ESTC citation number- promising! These links are of the form http://estc.bl.uk/{CITATION}. Unfortunately, these links do not resolve to the items themselves, but redirect to a search page with a single result. Ignoring the session tracking bits of the URLs (which appear to be removable), the search redirects are like so:

1. http://estc.bl.uk/{CITATION}
2. http://estc.bl.uk/F/?func=find-b&local_base=BLL06&request={CITATION}&find_code=ESTID

The resulting page communicates the context of a server-managed search, and so the page presumed to map the permalink has to be parsed out of the response markup. The search result set is nested in tables, but the useful data is down in the result rows, whose data cells have the class "td1". Within those rows are some summary metadata (title, author) and a link to the full-record page we have wanted. W can distinguish these URLs because, in addition to a set_number parameter, they have a set_entry parameter. So, presuming we have only XPATH:

//td[@class=td1]//a[contains(@href,'&set_entry')]/@href

... which should (and we will cross our fingers that there's just the one entry in the result set) get a url of the form:

http://estc.bl.uk/F/?func=full-set-set&set_number=NNNNNN&set_entry=000001&format=999

Result! But where is the STC/Wing number? The table in the full record page's HTML is structured for presentation to humans: Readable, but hard to parse. You could iterate over the rows of the table with ID=estcdata, looking for second rows starting with Wing or STC and trying to parse IDs out of them. A more accurate approach is available if you notice that the full record is available in multiple formats, one of which is MARC. It's linked at the top of the full record page, but you can also get there by changing the format parameter in the URL from 999 to 001. In the MARC format, you can more precisely look for rows in the estcdata table whose first cell's data is '5104'...

Wait, 5104? Yes, this is a combination of a MARC 510 (Citation/Reference) subfield 4 (Location).

... whose first cell's data is '5104', and parse the subfields out. This will be easiest to do in some kind of scripting language, but the subfield delimiter is a pipe '|', the id is a character (in this case 'a' or 'c'), and there's a whitespace for legibility here. We are interested in the value of the c subfield if the a subfield starts with Wing or STC (and we should be case-insensitive to be safe). Whew.

Ok! With a Wing/STC identifier in hand, we can teach a computer to look up a DEEP entry, which at this point is another post.

Link dump

http://www.ibm.com/developerworks/webservices/library/ws-restwsdl/
http://www.keith-chapman.org/2008/09/restfull-mashup-with-wsdl-20-wso2.html

Jackrabbit, RMI, etc.

JCR Remote Repo

Should a variation on Server implement javax.jcr.Repository? Or a modularized wrapper? Will need access to either JAAS or internal machinery for authN, as well.

Mulgara, allocateDirect, swap space

On sequences of large Mulgara queries, Java was crashing for lack of swap space. Culprit appears to be the lack of reuse of ByteBuffers, all of which are direct byte buffers (and thus outside heap space).

First crack was using pojo byte buffers (fixed swap issue). Second was/is making some of the read only and one-at-a-time classes reuse their buffers by adding a Block recycling method.

edit: Reusing buffers in the find method cuts the live memory for my test "large" searches by a tick over 40%, according to HPROF. Response time for the servlet is decreased as well, but the proportion varies from 40% to 25% according (I suspect) to how big a slice the IO accounts for.

Describing a tile

Trying to use the Djatoka jpeg2000 image viewer to display the image tiles / regions served up by an installation of the now-defunct eRez image server underscored the value of a good web API.

eRez Tile API

Parm Name	Type	Function
src	string	path to the ptif src, relative to the eRez image root
width	integer	width of the resulting image tile (will stretch to fit)
height	integer	height of the resulting image tile (will stretch to fit)
top	float	the position of the top edge of the tile relative to the entire scaled image, expressed as a decimal fraction
left	float	see "top"
bottom	float	see "top"
right	float	see "top"
scale	float	the ratio of the dimensions of an entire image composed of tiles in the requested size to the dimensions of the original image, expressed as a decimal fraction.
tmp	string constant	the value is "ajax-viewer", unquoted

OpenURL getRegion API

Parm Name	Type	Function
svc.level	integer	a scaling indicator, as specified here
svc.region	integer or float list	the top edge position, left edge position, region height and width. Concatenated as a comma-delimited value.
svc.scale	integer or float list	scaling factor as either a single value, or a targeted width and height. If the latter, a value of zero for one of the dimensions indicates the original proportions should be maintained.

Translating OpenURL Level to eRez Scale

After calculating the maximum levels, any given level converts to scale as:

scale = 1 / 2(maxLevels - requestedLevel)


you lying, non-ascii bastards

grep -l $'[\x80-\xff]' * > nonascii.txt

brain dump

What about a collection of micro-apps that extract linked data from epidoc, a la SNERT/OC? One for date info normalization, one to spit back pleiades, etc.

Date ranges, Ontology, etc.

Digital Humanities 2007
Time Period Directory Standard, 2006
A naïve ontology for concepts of time and space for searching and learning, 2007

Pharaonic navigation has an advantage here in that (ignoring protests of some historians, I'm sure) there are commonly defined, named periods with determinate endpoints. It would be possible to suggest some vocabulary to incorporate them. But that wouldn't provide many linkages to currently-known partners, and it wouldn't accurately describe most of the collection. So, what other approach would be more appropriate and inclusive?