StabiHacks

Various utilities to deal with metadata and content provided by the Berlin State Library/Staatsbibliothek zu Berlin

The scripts work together as illustrated below:

SBBget

• a Python script that is capable of downloading digitized media, the associated metadata, and its fulltext from Berlin State Library's digitized collections.
• it also extracts images that have been detected by the OCR and stores them in the desired file format, e.g., JPEG
• the extracted illustrations can also be stored as .tar files to facilitate distribution
• its logic is based on the more or less unique PPN identifier used at the Berlin State Library.
• some PPN lists are shipped for demonstration purposes. more can be obtained at the Berlin State Library or the creator of the script.
• the script will create various folders below its current working directory, e.g.,
  • downloads (fulltexts, original digitizations etc.) are stored at: sbbget_downloads/download_temp/
  • extracted images are stored at: sbbget_downloads/extracted_images/
  • METS/MODS files are stored at: sbbget_downloads/download_temp//__metsmods/

Setup

• clone the repository: git clone https://github.com/elektrobohemian/StabiHacks.git

• navigate to the sbbget/ subdirectory

• setup a Python environment: python3 -m venv ./venv

• activate the environment: source ./venv/bin/activate.fish (skip .fish if you are not using the wonderful fish shell)

• install the needed packages: pip install -r requirements.txt

• OPTIONAL: the script can be configured by modifying the accompanying YAML configuration file, the shipped YAML comes with reasonable settings and the objective to get as much data as possible only removing the master TIFF images to save storage space

First Run

• the script comes pre-configured and tries to download an existing book from the Berlin State Library if run directly (no parameters needed), simply run it by calling: python sbbget.py

  • the downloaded book has 341 scanned pages, thus, the download and processing will take some time (some minutes) as you will obtain a digital copy of a German children's book - stricly for boys because that seemed to matter in ancient times ;-)

  

  • if everything went well you should see an output similar to the one below:

  Number of documents to be processed: 1
  100%|████████| 1/1 [04:42<00:00, 282.02s/it]
  
  SUMMARY
      Downloads (fulltexts, original digitizations etc.) were, e.g., stored at: sbbget_downloads/download_temp/PPN745182844
  
      Extracted images were, e.g., stored at: sbbget_downloads/extracted_images/PPN745182844
      
      METS/MODS files were, e.g., stored at: sbbget_downloads/download_temp/PPN745182844/__metsmods/
  
  Started at:     2026-05-06 16:34:35.363607
  Ended at:       2026-05-06 16:39:17.388846
  Done.
  

• the script has been tested with Python 3.9, 3.11 and 3.14 but should run with other versions as well

Sample Data

• the script comes with some sample collection that are described here

OAI-Analyzer

• a Python script that downloads METS/MODS files and DC metadata via OAI-PMH and analyzes them, e.g., to save ALTO XML URLs for certain records or to save metadata such as language codes or authorships
• the results of the analyses are saved locally for further processing in various formats, e.g. Excel and CSV

Fulltext Analysis

• a Python script that retrieves all fulltexts from a SBBget created download directory and converts all files to raw text files
• additionally, the script runs a NER on all created raw text files and saves the results, the NER is based on flair
• for best (i.e. fast) results you should use a GPU but the script will also run on the CPU
• alternatively the script can operate on the result file created by OAI-Analyzer and download ALTO files directly, from this perspective it serves as a Stabi fulltext corpus builder

First Run

• the script is based on NLTK which needs additional installation steps, i.e.:
  • install NLTK in your Python environment
  • when running the script, Python will ask you to install additional NLTK packages, the easiest way is to open a Python interpreter and run to launch NLTK's graphical installer or to download the needed data via:

  import nltk
  nltk.download('punkt')
  

  • usually, NLTK will inform you about missing data if you forgot this step
  • further information can be found an online book that also gives an introduction into natural language processing
• the script comes pre-configured and can be launched after SBBget has run (no additional parameters are needed, the script looks for the SBBget download folder at the standard location)
• the script has been tested with Python 3.9 and 3.11 but should run with other versions as well

Future Improvements

• ner_analysis is based on the results from fulltext_analysis and creates graph data etc.
• this script is still under development

Pica Plus

• a Python script that parses files in the Pica+ format as provided by the GBV

• the script lets you choose interesting fields (as stored in the fieldsOfInterest list) and will output the contained data

• records will be separated by a NEW_RECORD string on command line or by an empty line in the text format

• output can be saved in text format, separated by the language of the record

• standard fields are:

  • title
  • author (+ optional GND ID)
  • country of publication (only the first entry in a specific extension of the DIN ISO 3166 format)
  • publisher and place of publication

• documentation of the Pica Plus format is only available in German here:

  • general overview
  • list of fields