Building Humanities Tools with LLMs

Building tools for humanities researchers means handling messy inputs, like OCR-scanned archival documents, and turning them into structured data. In this tutorial I will walk through a small Python utility that ingests raw historical text, cleans transcription errors, extracts named entities, and produces a structured JSON analysis. I shipped this on Oxlo.ai because the flat per-request pricing keeps costs predictable even when I feed it multi-page documents with long context windows.

What you'll need

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client and system prompt

I start by importing the SDK and pointing it at Oxlo.ai. Then I define a system prompt that constrains the model to act as a careful archival assistant, not a chatbot.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

SYSTEM_PROMPT = """You are an archival analysis assistant. Your job is to process raw historical document text.
- Correct obvious OCR errors without rewriting the original voice.
- Extract entities (people, places, organizations, dates).
- Output only valid JSON. Do not add markdown code fences or commentary outside the JSON object.
- If a date is ambiguous, note the ambiguity. Do not invent facts not present in the text."""

Step 2: Ingest and sanitize raw OCR text

Archival OCR usually contains hard line breaks, hyphenation artifacts, and garbled characters. I strip those out so the model receives flowing paragraphs.

import re

def load_and_clean(filepath: str) -> str:
    with open(filepath, "r", encoding="utf-8") as f:
        raw = f.read()

    # Remove soft line breaks and hyphenation
    text = re.sub(r"(\w)-\n(\w)", r"\1\2", raw)
    text = re.sub(r"\n+", " ", text)
    # Normalize whitespace
    text = re.sub(r"\s+", " ", text).strip()
    return text

Step 3: Extract entities and dates

Now I send the cleaned text to Llama 3.3 70B with a focused user prompt that requests a JSON object. I keep the temperature low to reduce hallucination.

import json

def extract_entities(text: str) -> dict:
    user_message = (
        "Extract entities from the following historical document. "
        "Return a JSON object with keys: people, places, organizations, dates, ocr_corrections. "
        "Each person should have fields: name, role_if_mentioned. "
        "Each date should have fields: raw_text, normalized_iso_8601 (or null if uncertain).\n\n"
        f"Document:\n{text}"
    )

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
        max_tokens=2048,
    )

    content = response.choices[0].message.content
    return json.loads(content)

Step 4: Generate historical context and summary

With the raw entities in hand, I run a second pass to get a one-paragraph summary, a guessed document type, and a list of suggested research themes.

def generate_analysis(text: str, entities: dict) -> dict:
    user_message = (
        "Provide a structured analysis of the historical document. "
        "Return a JSON object with keys: summary (one paragraph), document_type, "
        "themes (list of 3 to 5 themes), and confidence (high, medium, low).\n\n"
        f"Document text:\n{text[:3000]}\n\n"
        f"Extracted entities:\n{json.dumps(entities, indent=2)}"
    )

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.2,
        max_tokens=2048,
    )

    content = response.choices[0].message.content
    return json.loads(content)

Step 5: Bundle into a reusable CLI tool

I wrap the pipeline in a small class so I can point it at any text file and get a single JSON report. This is the shape I actually deploy for my research group.

import json
import re
import sys
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

SYSTEM_PROMPT = """You are an archival analysis assistant. Your job is to process raw historical document text.
- Correct obvious OCR errors without rewriting the original voice.
- Extract entities (people, places, organizations, dates).
- Output only valid JSON. Do not add markdown code fences or commentary outside the JSON object.
- If a date is ambiguous, note the ambiguity. Do not invent facts not present in the text."""

def load_and_clean(filepath: str) -> str:
    with open(filepath, "r", encoding="utf-8") as f:
        raw = f.read()
    text = re.sub(r"(\w)-\n(\w)", r"\1\2", raw)
    text = re.sub(r"\n+", " ", text)
    text = re.sub(r"\s+", " ", text).strip()
    return text

def extract_entities(text: str) -> dict:
    user_message = (
        "Extract entities from the following historical document. "
        "Return a JSON object with keys: people, places, organizations, dates, ocr_corrections. "
        "Each person should have fields: name, role_if_mentioned. "
        "Each date should have fields: raw_text, normalized_iso_8601 (or null if uncertain).\n\n"
        f"Document:\n{text}"
    )
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
        max_tokens=2048,
    )
    return json.loads(response.choices[0].message.content)

def generate_analysis(text: str, entities: dict) -> dict:
    user_message = (
        "Provide a structured analysis of the historical document. "
        "Return a JSON object with keys: summary (one paragraph), document_type, "
        "themes (list of 3 to 5 themes), and confidence (high, medium, low).\n\n"
        f"Document text:\n{text[:3000]}\n\n"
        f"Extracted entities:\n{json.dumps(entities, indent=2)}"
    )
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.2,
        max_tokens=2048,
    )
    return json.loads(response.choices[0].message.content)

class ArchiveAnalyzer:
    def process(self, filepath: str) -> dict:
        text = load_and_clean(filepath)
        entities = extract_entities(text)
        analysis = generate_analysis(text, entities)
        return {
            "source_file": filepath,
            "cleaned_text_preview": text[:500],
            "entities": entities,
            "analysis": analysis,
        }

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python archive_analyzer.py ")
        sys.exit(1)

    analyzer = ArchiveAnalyzer()
    result = analyzer.process(sys.argv[1])

    with open("report.json", "w", encoding="utf-8") as out:
        json.dump(result, out, indent=2, ensure_ascii=False)

    print("Report written to report.json")

Run it

I created a sample file named sample_letter.txt with a fabricated 19th-century correspondence snippet containing intentional OCR noise. Running the tool produces a structured report.

$ python archive_analyzer.py sample_letter.txt
Report written to report.json

The resulting report.json looks like this:

{
  "source_file": "sample_letter.txt",
  "cleaned_text_preview": "Boston, March 14th, 1847. Dear Sir, I write to inform you that the shipment of textiles...",
  "entities": {
    "people": [
      {"name": "Jonathan Hale", "role_if_mentioned": "correspondent"},
      {"name": "Mr. Aldridge", "role_if_mentioned": "recipient"}
    ],
    "places": [
      {"name": "Boston", "role_if_mentioned": "origin"},
      {"name": "Liverpool", "role_if_mentioned": "destination"}
    ],
    "organizations": [
      {"name": "Hale & Sons Trading Co."}
    ],
    "dates": [
      {"raw_text": "March 14th, 1847", "normalized_iso_8601": "1847-03-14"}
    ],
    "ocr_corrections": [
      {"original": "14tli", "corrected": "14th"}
    ]
  },
  "analysis": {
    "summary": "A business letter from Jonathan Hale in Boston to Mr. Aldridge regarding textile shipments and anticipated arrival in Liverpool.",
    "document_type": "business correspondence",
    "themes": [
      "19th-century maritime trade",
      "textile industry",
      "transatlantic commerce",
      "business communication"
    ],
    "confidence": "high"
  }
}

Wrap-up and next steps

This pipeline turns unstructured archival text into a structured database entry in two API calls. Because Oxlo.ai charges a flat rate per request, I can feed it ten-page documents without watching token costs spike.

Two concrete next steps: first, add Oxlo.ai's embedding endpoint to index every report into a vector store so researchers can semantically search across thousands of documents. Second, wire in the vision models such as Kimi K2.6 to process scanned page images directly, skipping the OCR step entirely.