External IIIF Source Inventories

timeatlas.SourceInventory normalizes CSV/JSON metadata for images already hosted by an external IIIF server. It uses the same camelCase source configuration as the ingest assistant. It performs no network requests: the supplied metadata and service capabilities are validated locally, not verified against a server.

It also exposes normalize_source_manifests for already downloaded IIIF Presentation 2/3 manifests, using the same inventory and generation pipeline. plan_source_image_urls and normalize_source_image_urls support image URL lists enriched from host-downloaded Image API information responses.

Configuration

An inventory has one row per canvas, a stable image identifier, a complete image request URL, positive width/height, and the MIME type of the delivered image. Optional columns provide the image service root and label. Do not supply a info.json URL as the service root. For example:

config = {
    "enabled": True,
    "inputType": "inventory",
    "columns": {
        "imageKey": "filename", "imageUrl": "url",
        "width": "width", "height": "height", "mediaType": "media_type",
    },
    "imageService": {"version": "3", "profile": "level2"},
    "grouping": {"mode": "single", "label": "Volume I"},
    "ordering": {"mode": "natural", "field": "filename"},
    "recordLink": {
        "recordKeyField": "record_id", "recordImageField": "page_number",
        "imageJoinField": "filename", "normalization": "page-number",
        "missing": "error",
    },
}

grouping.mode also accepts field with field naming a column, or per-image. grouping.labelField maps a document label column; conflicting labels inside one manifest are rejected. ordering.mode also accepts inventory (input order) or numeric (a numeric ordering column).

recordLink.normalization accepts exact or page-number. The latter matches numeric page values such as 174.0 to image filenames such as /174.jpg and explicit page labels such as pg. 174. Image request and service URLs can also be matched by a numeric image-identifier basename; URL decoding applies only to the join key and never changes the stored external URLs. Page numbers repeated in different volumes are ambiguous: use a compound key and an exact join instead. missing must explicitly be error or allow-unlinked. Invalid or ambiguous links always fail.

Generation

Pass actual generated Historical Record UUIDs, not independent UUID guesses:

from timeatlas import (
    UUIDManager, normalize_source_inventory, link_source_records,
    build_source_collection, write_source_collection, write_source_inventory,
)

inventory = normalize_source_inventory("images.csv", config)
# HRs have already been produced using the source record IDs.
hr_ids_by_record_key = {row["record_id"]: hr.id for row, hr in produced_records}
links, report = link_source_records(
    inventory, record_rows, config, hr_ids_by_record_key,
)
uuid_manager = UUIDManager("https://example.org/my-dataset")
collection, counts = build_source_collection(
    inventory, uuid_manager=uuid_manager,
    collection_key="my-dataset", collection_label="My source collection",
    links=links,
)
write_source_collection(collection, uuid_manager=uuid_manager, output_dir=dataset_dir)
write_source_inventory(inventory, "/job/source-inventory.csv")
dataset.sources = [collection.id]

Catch SourceInventoryError and save its report when joins fail. Store the inventory and link report outside import output. IIIF files are written under the dataset’s iiif/collections and iiif/manifests directories, named using their UUIDs.

The normalized schema contains manifest_key, manifest_label, canvas_key, canvas_label, canvas_position, image_url, image_service_url, image_service_type, image_service_profile, width, height, media_type, image_join_key and source_row. Manifest and canvas keys produce deterministic IDs independently of CSV order. Multiple HR links create annotations without duplicating canvases. Blank pages without annotations are retained.

A JSON array in a record’s image-reference field links that record to multiple canvases. CSV cell strings are not implicitly split. Different source record keys may map to the same HR UUID when the production model deliberately merges records, but each source record key itself must remain unique.

Downloaded manifest input

The application downloads configured manifestUrls on the host, saves source snapshots and checksums, then passes local JSON paths or already loaded manifest objects to the offline parser:

from timeatlas.SourceInventory import normalize_source_manifests

config = {
    "enabled": True,
    "inputType": "manifest-url",
    "manifestUrls": ["https://example.org/iiif/book/manifest"],
    "recordLink": {
        "recordKeyField": "record_id", "recordImageField": "page_number",
        "imageJoinField": "canvas_label", "normalization": "page-number",
        "missing": "error",
    },
}
inventory = normalize_source_manifests(
    ["/job/source-manifests/manifest-001.json"], config,
)

Use the same link_source_records and build_source_collection calls as for CSV input. Manifest grouping, image-service metadata and canvas order come from each supplied manifest; inventory-mode settings do not override them. Join fields are canvas_id, canvas_label, image_url and image_service_url. A missing source canvas label receives a display fallback but never a fabricated join key.

The normalized rows additionally contain original manifest_id/canvas_id, manifest_position, complete manifest_labels/canvas_labels language maps, and optional source_manifest_metadata rights/requiredStatement fields. The builder preserves these label maps and rights statements. Existing source Historical Record annotations are excluded; only explicitly joined current dataset HRs are annotated. New UUIDs depend on stable original manifest/canvas identifiers and the destination dataset namespace.

Supported input is Presentation 3 embedded canvases/AnnotationPages or Presentation 2 with one embedded sequence, one complete image painting per canvas, matching image/canvas dimensions, required image width/height/MIME metadata and one advertised ImageService2/3. Version 2 string/language labels and attribution are translated to Presentation 3 language maps and requiredStatement. Service profiles are normalized to their advertised compliance level; the downloaded source snapshot retains any extra service capability details. For a complete Image API request, a recognized output format in the painting URL takes precedence over the declared image MIME type: a TIFF service identifier requesting default.jpg produces image/jpeg. Other URLs retain their declared MIME type. Source snapshots are unchanged.

For matching image-request/service pairs with a file-style identifier below /iiif/2/ or /iiif/3/, duplicate identifier separators (literal /, encoded %2F or a mixture) collapse to one encoded separator. For example, venice%2F%2Fcatastici%2F001.tif becomes venice%2Fcatastici%2F001.tif consistently in the image and service URL. This compatibility rule treats those identifiers as filesystem-style paths; opaque identifiers and arbitrary URL prefixes are never rewritten. Valid single separators, other escapes, host/scheme spelling, queries and fragments remain unchanged. Original canvas objects remain available in the source snapshots for authored record joins; normalized inventory URL fields contain the canonical values. This rule does not alter the CSV-inventory or image-URL-list adapters.

Optional structures (navigation ranges/tables of contents) are accepted but not copied. They can repeat, omit or reorder canvas references independently of the page sequence. All pages follow the manifest’s top-level items order (or its single Presentation 2 sequence), once each, including pages absent from the ranges. Each supplied manifest stays a separate volume; the collection follows the supplied manifest order. Rebuilding the original navigation hierarchy is outside this adapter’s scope.

Unsupported input fails explicitly: Collections, multiple sequences, referenced AnnotationPages, Choice/multiple painting bodies, region or composite layouts, differing image/canvas dimensions and missing required metadata. No remote image enrichment or source annotation copying occurs in this adapter.

Image URL lists and information snapshots

Image URL input requires only a UTF-8 text file containing one complete IIIF image request URL per nonempty line. The planner performs no network access and preserves the original URL spelling, escaping and order:

from timeatlas.SourceInventory import (
    plan_source_image_urls, normalize_source_image_urls,
)

plan = plan_source_image_urls("images.txt")
# Host code downloads each unique plan row's info_url. Each snapshot is:
snapshots = [{
    "serviceUrl": plan[0]["image_service_url"],
    "infoUrl": plan[0]["info_url"],
    "finalUrl": plan[0]["info_url"],  # actual final URL after redirects
    "path": "/job/source-image-info/info-0001.json",
    "sha256": "...SHA-256 of the downloaded file bytes...",
}]
config = {
    "enabled": True, "inputType": "image-urls",
    "grouping": {"mode": "single"},
    "ordering": {"mode": "natural", "field": "image_filename"},
    "recordLink": {
        "recordKeyField": "record_id", "recordImageField": "page_number",
        "imageJoinField": "image_filename", "normalization": "page-number",
        "missing": "error",
    },
}
inventory = normalize_source_image_urls("images.txt", snapshots, config)

The snapshot list contains one entry per unique requested info URL. Each plan row contains image_url, image_service_url, info_url, source_position and request_format. An already loaded info object can replace path/sha256 for offline use. The normalizer verifies snapshot checksums when supplied and requires the declared service identity to match the requested service or the final downloaded metadata URL. Percent-encoding equivalence is accepted during identity comparison; duplicate slashes and unrelated identifiers are not fixed. The original image URL remains unchanged in generated painting bodies.

Dimensions and ImageService2/3 compliance levels come from the metadata; MIME type comes from the actual requested format, not the source file extension or a list of available formats. Requested formats must be supported by the advertised compliance level or additional format declarations. Requests are limited to full region, default quality, rotation 0 and Image API 2 full/max or Image API 3 max size. If maxWidth/maxHeight/maxArea reduce the available full size, normalization fails instead of guessing scaled dimensions. Metadata and image pixel retrieval are separate: the helpers never download or copy image pixels.

Grouping, ordering, labels and HR joins use these derived columns:

• image_url: exact supplied request URL, also the stable canvas key.

• image_service_url: validated service ID advertised by info.json.

• image_identifier: decoded last service-URI component.

• image_filename: basename of the decoded service path.

• image_directory: parent of the decoded service path, including its prefix.

• source_position: original 1-based input position.

Natural ordering defaults to image_filename; numeric ordering defaults to source_position. columns.label optionally names one derived field; otherwise labels use page numbers. The default HR join field is image_filename. Existing inventory-mode image column mappings and manual service settings are ignored. Text input is limited to 10,000 unique URLs and 2 MiB. Duplicate URLs, missing snapshots, unsupported requests and unavailable metadata produce explicit errors.

External URL handling

Page supports explicit image_url, image_service_url, image_service_type, image_service_profile and optional thumbnail_url. Explicit URLs are preserved without URL-decoding or re-encoding. Without an explicit thumbnail, the original image request serves as the thumbnail; the library does not invent a resize request that a remote server may not support. Existing object_ref paths and a shared url_prefix keep their behavior.

Image API 2 services serialize using @id/@type and a version 2 profile URI. Image API 3 uses id/type and its short profile name, as described in the IIIF Presentation API service specification.

API

Normalize external IIIF image inventories and join them to Historical Records.

The configuration is the ingest assistant’s ModelingSourceConfig (camelCase). This module is deliberately offline: it neither downloads nor publishes images.

exception timeatlas.SourceInventory.SourceInventoryError

Invalid source metadata or links; report contains actionable details.

__init__(message, report=None)

Parameters:

• message (str)

• report (dict | None)

timeatlas.SourceInventory.read_source_rows(rows)

Read UTF-8 CSV, a JSON array (or {"rows": [...]}), or row mappings.

Parameters:

rows (str | Path | Iterable[Mapping])

Return type:

list[dict]

timeatlas.SourceInventory.image_service_from_request(image_url)

Remove a full IIIF request’s region/size/rotation/quality.format suffix.

Encoded identifiers and query strings are retained. This function requires an Image API request URL; a bare JPEG URL cannot identify an IIIF service.

Parameters:

image_url (str)

Return type:

str

timeatlas.SourceInventory.normalize_source_inventory(rows, config)

Validate mapped CSV/JSON data and return one canonical row per canvas.

Required column mappings: imageKey, imageUrl, width, height, mediaType. Grouping is single, field, or per-image. Ordering is inventory, natural, or numeric. Image API version/profile must be explicitly configured. IDs are derived later from manifest_key/canvas_key, never row positions.

Parameters:

config (Mapping)

Return type:

list[dict]

timeatlas.SourceInventory.link_source_records(inventory, records, config, hr_ids_by_record_key)

Join source records to actual generated HR UUIDs, without guessing links.

hr_ids_by_record_key maps stable source record keys to the exact UUIDs used by the production script. Values may be one UUID or a list of UUIDs. A JSON array in the record image field links one record to several canvases; CSV strings are never split implicitly. Missing source references are fatal unless missing=’allow-unlinked’. Unannotated source images are allowed: blank book pages are legitimate.

Parameters:

• inventory (list[dict])

• config (Mapping)

• hr_ids_by_record_key (Mapping)

Return type:

tuple[list[dict], dict]

timeatlas.SourceInventory.build_source_collection(inventory, *, uuid_manager, collection_key, collection_label, links=(), allow_unlinked=False)

Build stable Collection/Document/Page/Annotation objects from normalized rows.

Parameters:

• inventory (list[dict])

• uuid_manager (UUIDManager)

• collection_key (str)

• links (Iterable[Mapping])

• allow_unlinked (bool)

Return type:

tuple[Collection, dict]

timeatlas.SourceInventory.write_source_collection(collection, *, uuid_manager, output_dir)

Write IIIF under output_dir/iiif/{collections,manifests}/<uuid>.json.

output_dir is the dataset directory, e.g. /output/rde/datasets/<slug>. This does not create metadata sidecars inside the import output directory.

Parameters:

• collection (Collection)

• uuid_manager (UUIDManager)

• output_dir (str | Path)

Return type:

list[Path]

timeatlas.SourceInventory.write_source_inventory(inventory, path)

Save the canonical inventory CSV, normally under /job for review.

Parameters:

• inventory (list[dict])

• path (str | Path)

Return type:

Path

timeatlas.SourceInventory.normalize_source_manifests(manifests, config)

Normalize local Presentation 2/3 manifests; download URLs on the host first.

See timeatlas.SourceManifests for the supported manifest profile.

Parameters:

config (Mapping)

Return type:

list[dict]

timeatlas.SourceInventory.plan_source_image_urls(urls_or_txt_path)

Derive info.json requests from a URL list, without fetching them.

Return type:

list[dict]

timeatlas.SourceInventory.normalize_source_image_urls(urls_or_txt_path, info_snapshots, config)

Normalize a URL list plus host-fetched Image API metadata snapshots.

Parameters:

config (Mapping)

Return type:

list[dict]