[Glebs] - SUB Canonical Importer

[piconti]

This issue describes the tasks and steps which compose the creation of a new SUB importer in the text_preparation.importer module.
The idea is to delegate this task to @maslionok as it is quite self contained.

General description of our needs

We would need to implement the classes which import SUB's OCR/OLR data into our canonical format on S3.
This is done by the text_preparation.importer module in two main parts:

• Classes and detect functions - in the classes.py and detect.py scripts of the importers.sub module
  • classes.py define the Canonical Issue and Canonical Page classes for the SUB case, taking into account all the specificities of this exact version of mets/alto OCR/OLR
  • detect.py define the functions used to identify all the issues to ingest inside the file system, using "IssueDir" objects, which are named tuples holding all the main information identifying an issue.
  • Both of these scripts are very similar from importer to importer, and the code from other METS/ALTO based importers can (and should!) completely be used as guide and example, especially the code of BL, BNF, BNF-EN.
• Orchestators generic_importer.py and core.py.
  • These are the main orchestrators of the imports. they are not to be modified too much as they work in the same way for all importers. Reading them can allow to understand better how everything works (eg. how issues are serialized first, the lazy behavior of page objects, error logging etc). It might be that there are errors in these scripts, but it would be best to come to me about them as this impacts all canonical imports.

Much more precise information can be found in the documentation for the project, but don't hesitate to reach out and ask if there is anything which is unclear

Here are the json schemas the pages and issues follow.

Current Situation and File Structure

The current situation is the following: we have received from the SUB the full contents of the "Hamburger Echo" newspaper title, spanning from 1887 to 1933, and in METS/ALTO format.

The structure of the files is the following:

Hamburger_Echo/                          # Root directory (newspaper title)
├── 1919/                                # Year directory
│   ├── 02/                              # Month directory (February)
│   │   ├── 19/                          # Day directory (19th)
│   │   │   ├── Morgenausgabe/           # Morning edition
│   │   │   │   ├── 00000001.tif         # Page image (facsimile)
│   │   │   │   ├── 00000001.xml         # Page OCR (ALTO or PAGE XML)
│   │   │   │   ├── 00000002.tif
│   │   │   │   ├── 00000002.xml
│   │   │   │   ├── [...]                # More page pairs (.tif/.xml)
│   │   │   │   └── PPN1754726119_19190219MO.xml   # METS file (Morgenausgabe, “MO”)
│   │   │   │                                      # Format: PPN[titleID]_[YYYYMMDD][edition_code].xml
│   │   │   │                                      # Here: PPN1754726119 = newspaper ID
│   │   │   │                                      #        19190219 = date in YYYMMDD format
│   │   │   │                                      #        MO = morning edition (Morgenausgabe)
│   │   │   │
│   │   │   ├── Abendausgabe/            # Evening edition (same day)
│   │   │   │   ├── 00000001.tif
│   │   │   │   ├── 00000001.xml
│   │   │   │   ├── [...]                # More pages
│   │   │   │   └── PPN1754726119_19190219AB.xml   # METS file (Abendausgabe, “AB”)
│   │   │   │
│   │   │   └── [other editions or none for this day...]
│   │   │
│   │   ├── 20/                          # Next day (example of multiple evening editions)
│   │   │   ├── A1-Abendausgabe/
│   │   │   │   ├── 00000001.tif
│   │   │   │   ├── 00000001.xml
│   │   │   │   ├── [...]
│   │   │   │   └── PPN1754726119_19190220A1.xml   # METS file (first evening edition)
│   │   │   │
│   │   │   ├── A2-Abendausgabe/
│   │   │   │   ├── 00000001.tif
│   │   │   │   ├── 00000001.xml
│   │   │   │   ├── [...]
│   │   │   │   └── PPN1754726119_19190220A2.xml   # METS file (second evening edition)
│   │   │
│   │   ├── 21/                          # Example of a single daily edition
│   │   │   └── Ausgabe/
│   │   │       ├── 00000001.tif
│   │   │       ├── 00000001.xml
│   │   │       ├── [...]
│   │   │       └── PPN1754726119_19190221.xml     # METS file (single daily edition)
│   │   │
│   │   └── [other days...]
│   │
│   └── [other months...]
│
└── [other years...]

Each level represents:

• Title → Hamburger_Echo (newspaper)
• Year → e.g. 1919
• Month → e.g. 02
• Day → e.g. 19, 20, 21
• Edition → one of:
  • Ausgabe → single daily edition (it seems)
  • Morgenausgabe → morning edition (MO)
  • Abendausgabe → evening edition (AB)
  • A1-Abendausgabe, A2-Abendausgabe → multiple evening editions (A1, A2) - probably can also exist for morning
    -> it can be that there the morning or evening edition is the only one of one day. From what I have seen, whenever we have "Ausgabe" it's always the only edition of the day, but this may not apply to the entirety of the data.

This structure can be directly and easily used to construct the Impresso IDs of each issue (and page):
[media-alias]-[YYYY]-[MM]-[DD]-[edition letter(s)] ([media-alias]-[YYYY]-[MM]-[DD]-[edition letter(s)]-p[page number filled to 4 digits] for pages).
--> Note that the editions are simply a letter assigned, 'a' for the first, 'b' for the second etc.

Here we will assign the alias "hamb_echo" to the Hamburger Echo - and decide later what to assign to other titles

More precise tasks and questions left to discussion

I have already added some skeletons for the classes and scripts to implement in the feature/sub-importer branch.

• defining the SubNewspaperIssue object
• defining the SubNewspaperPage object
• defining the functions in the detect.py script
• modularization of any helper function in a helpers.py script

[e-maud]

Link to METS/ALTO short presentation: https://docs.google.com/presentation/d/1hwIoj3w_aycxVGGLcmyE64J9i9cF8Y-78oTAqGBhToU/edit?slide=id.g39ddef1486f_0_0#slide=id.g39ddef1486f_0_0

[piconti]

I have added 3 example issues to this repo, and I'm currently uploading more samples to S3's 00-original-samples bucket.
for each of the 3 sample issues, I'll upload the corresponding entire month of data.

[piconti]

Following our discussion, adding additional comments and description about small details and creation of the i (items) list in the issues.

The issue's schema contains the list i of content items which are part of the issue.
In SUB's case, we have no OLR, so we usually only have what we call "page" content-items, images and tables.
Page content-items are the equivalent to a usual article (they contain the text regions and their corresponding coordinates), but for an entire page. Hence, they are composed of all the text regions (TextBlock elements in the alto files) of their corresponding page.

The content-items should be constructed in the _parse_mets() function, before the self.issue_data object is filled it.
Here is an example of the _find_content_items function in the BCUL importer (work-in-progress branch).

As one can see in the schema of the issue, each content item is composed of two main elements:

• The metadata
  • contains all the "general" information necessary to uniquely identify the CI: its ID, language, type, the pages it appears on, its title etc
  • Quite straightforward I don't doubt you'll manage, just note that to assign the IDs:
    • all content-items that correspond to all the text within a page are of type "page" and have the same ID as the page, just replacing the p by i (eg. the content item from page title-year-month-day-a-p0001 will be title-year-month-day-a-i0001)
    • additionally, pages can contain Illustration objects or TextBocks of type "Table", which correspond respectively to content-items of type "image" and "table". They should be uniquely numbered, starting after the last CI corresponding to a page.
• The legacy (The part that seemed confusing when we discussed)
  • Its goal is to keep track of all the METS/ALTO components which make up a specific content-item (in particular so that the page's _parse function knows which ALTO textblock to consider for which CI). But now In the case where we have no OLR all the TextBlock elements (of actual text excluding tables) from the page.
  • Now we are also trying to track relevant info/metadata about the institution's naming/IDs approach and original source filenames.
  • The fields of the legacy are the following:
    • id - the id of the "article" section in the OLR. In our case here it could be a "composite" id for page CIs (eg: [PPN]-[page_num], where the PPN is the mets file name, and page_num the page's alto filename), and the Illustration/TextBlock ID in the alto file for image and table CIs.
  • parts - the list of all elements which compose the CI. For page CIs it will be all the TextBlock elements which are not of type "table", and for images and table it will be the corresponding element.
  • They are composed of the Ids of the elements, their role (usually "body" or "text" or not defined if we don't know), the file's ID (page file name) and number
  • source - ideally the source mets filename (the ones for ALTO files will already be in parts.
  • You can add other fields here, based on the needs. For instance, a field ppn (issue's internal ID for SUB) and/or title_ppn (but the title's so without the date) would be useful, especially if the id actually contains the element's ID.

Realizing now that for images and tables, the element's ID will always be in the parts and as there will only one part, it's not necessary to store them in the overal legacy.id field. Then you could use the composite ID I mentioned for all CIs, which would systematically contains the title's PPN, the issue's PNN (with the date) and the page filename. This would be enough for my needs in data return so it might be the best approach, but based on the cases you encounter in the data it might be complex to put in place (eg, if you have multiple ALTO elements for a table or an image).
Anyways, always welcome to ask questions!

[maslionok]

Thank you very much for this clear description and additional information. It really made things clearer! I have pushed the updated version of sub importer with the changes that I believe cover all/most of the things we have talked about! I have also re-run the debug notebook and also pushed it so it is easier to inspect the outputs.

I have adapted _find_content_items method from bcul as you have suggested. If you have time to skim through the updated code and leave me some comments that would be great!