File Under: Reasons not to encode Data in File Names

Consider: Two adjoining papyrus fragments that connect on the horizontal. On one side, a relatively long text that spans the two fragments (text A). From the perspective of this text, there are two additional texts (in two different hands) in the right "margin" of the rightmost fragment, one (text B) above the other (text C). Finally, there is some writing on the other side (text D) of the joined fragments.

You have a cataloging scheme that separately identifies each text on the physical artifact. You have an image file naming scheme that effectively represents the relationship of a catalog record with an image file: {catalog id}.{front/back}.{tile coordinate if applicable}.{resolution}.{format}.

From the perspective of the text A, images of the two fragments might be named A.front.left.high.tif, A.front.right.high.tif, etc. But wait! Isn't A.front.left.high.tif also D.back.left.high.tif? And isn't A.front.right.high.tif also B.front.high.tif?

Mapping note

Mapping structured citations: If there is a successful sided match, add the unsided cite to the matched list to preclude false recto/verso matches.

Concordia, naming, sparse relationships

http://www.atlantides.org/trac/concordia/wiki/ConcordiaThesaurus
Numbers server already resolves (ultimately) to RDF
low-hanging fruit: adding some known metadata properties to the leaves
high-hanging fruit: disentangling the many sparse relationships
**
What if rather than having any organizational center for the index, we reorganized things around a more abstracted graph relating Objects (inventory numbers), Texts (citations), and CatalogEntries (metadata records, these might be Editions)

Say we have two more relationships: METADATA ore:describes TEXT/OBJECT, and METADATA ore:similarTo METADATA

Number Server lets you drill down through identifier hierarchies as aggregates, OR lets you see a graph centered on a particular URI.

More at some indeterminate point in the future.

Encoding javascript utf16 characters for urls

Dug this up from the bowels of the ddbdp webapp code. I wrote it a long time ago, but didn't end up needing it. Seems like it might be useful down the line...

---

var firstByteMark = [ 0x00,0x00,0xC0,0xE0,0xF0,0xF8,0xFC ];
var byteMask = 0xBF;
var byteMark = 0x80;

function UTF16toUTF8Bytes(u16){
    var bytes = new Array();
    if (u16 < 128){
        bytes.length = 1;
    } else if (u16 < 2048){
        bytes.length = 2;
    } else { // presuming max js charCode of 65535
        bytes.length = 3;
    }
    switch (bytes.length){
        case 3:
            bytes[2] = ((u16 | byteMark) & byteMask);
            u16 >>= 6;
        case 2:
            bytes[1] = ((u16 | byteMark) & byteMask);
            u16 >>= 6;
        case 1:
            bytes[0] = (u16 | firstByteMark[bytes.length]);
    }
    return bytes;
}
function encode(input){
    var output = new Array();
    var inputArray = input.split(/\s+/);
    for(var i=0;i<inputArray.length;i++){
        var term = '';
        for(var j=0;j<inputArray[i].length;j++){
            var u16 = inputArray[i].charCodeAt(j);
            if (u16 < 128){
                term += inputArray[i].charAt(j);
                continue;
            }
            var utf8bytes = UTF16toUTF8Bytes(u16);
            for(var k=0;k<utf8bytes.length;k++){
                if(utf8bytes[k] < 16){
                    term += "%0";
                } else {
                   term += "%";
                }
                term += utf8bytes[k].toString(16);
            }
        }
        output[i] = term;
    }
    return output;
}   

Lexington Prep

• wiki outline


• fix the tests to operate against more properly contained data


• continued refactoring/cleanup


• get the project components into the new repository - in progress: http://idp.atlantides.org/trac/idp/browser

Lucene Customization Performance

Hardware: Athlon 64 X2 Dual 3800; 2GB RAM
Data: Lexicon of 273365 terms

Coarse Response Time (50 iterations on a single substring query)

Query Type	Rough System Timing (System.current…)
indexOf query	8078
Wildcard query	11078
Bigrams Span query	2657

HPROF Results (Single terms and phrases)

Metric	Wildcard (custom)	Bigram Spans (custom)	%difference
Cpu (total)	34233203	12130249	-64.5%
Heap(total)	76345454	92707096	+21.4%
	Wildcard (custom)	Bigrams (custom)	%difference
Cpu (total)	34233203	6229156	-81.8%
Heap(total)	76345454	87350103	+14%

to-do

* new interface templates
* update standalone servlet with metadata filters
* document logging
* check OAI refreshes
* ZA XSLT for PN
* APIS images (eRez) for qualifying hgv views
* fix highlighting of HGV pub in metadata search results
* fix image/trans float-to-top in ddb
** correct hasImage index
** propagate sort flags on paging links
** filter "keine" from bibl.illustration