Skip to content

About

prototypes for AI in regulatory compliance

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

AI in Regulatory Compliance

A prototype system for extracting structured information from regulatory compliance documents using Large Language Models (LLMs) and vector databases.

Overview

This project provides a multi-phase approach to process regulatory PDFs, convert them into searchable vector embeddings, and extract structured requirements using LLM-based information extraction.

Project Structure

ai_in_reg/
├── prototype.py           # Main prototype: PDF ingestion, chunking, and embedding
├── extract_minimalist.py  # Requirement extraction using LLM and Pydantic schemas
├── requirements.txt       # Python dependencies
├── data/                  # Directory for PDF documents
├── chroma_db/            # Vector database storage (gitignored)
└── un_r152_db/           # Specific vector database for UN R152 regulation (gitignored)

Features

Phase 1: PDF Ingestion and Splitting (prototype.py)

  • PDF Loading: Uses PyMuPDFLoader from LangChain for layout-preserving text extraction
  • Intelligent Chunking: RecursiveCharacterTextSplitter with configurable chunk size (1000 chars) and overlap (200 chars)
  • Metadata Preservation: Maintains page numbers and source file information for each chunk
  • Batch Processing: Processes all PDFs in a folder automatically

Key Functions:

  • load_pdf_with_metadata() - Loads PDF with page metadata
  • split_documents() - Splits documents into chunks with header awareness
  • process_pdf_folder() - Processes entire folder of PDFs

Phase 2: Embeddings and Vector Storage (prototype.py)

  • Embedding Generation: Uses all-MiniLM-L6-v2 sentence transformer model
  • Vector Database: Stores embeddings in ChromaDB with persistence
  • Smart Caching: Automatically detects and reuses existing embeddings
  • Verification: Includes tools to verify storage and test similarity search

Key Functions:

  • load_embedding_model() - Loads sentence transformer model
  • create_embeddings() - Generates embeddings for all chunks
  • store_in_chroma() - Stores in ChromaDB with metadata
  • check_existing_embeddings() - Checks if embeddings already exist

Phase 3: Requirement Extraction (extract_minimalist.py)

  • Vector Search: Searches existing ChromaDB using semantic similarity
  • LLM Extraction: Uses GPT-4.1 Nano (with fallback to GPT-4o-mini) for structured extraction
  • Pydantic Schema: Validates extracted requirements against defined schema
  • Token Tracking: Monitors and reports token usage per extraction

Extracted Fields:

  • type: Either 'Must-have' or 'Should'
  • requirement: Core technical statement
  • metric: Numerical value (optional)
  • unit: Unit of measure (optional)

Installation

  1. Clone the repository:

    git clone <repository-url>
    cd ai_in_reg
  2. Create and activate virtual environment:

    python -m venv .venv
    source .venv/bin/activate  # On Windows: .venv\Scripts\activate
  3. Install dependencies:

    pip install -r requirements.txt
  4. Set up environment variables: Create a .env file in the project root:

    OPENAI_API_KEY=your-api-key-here
    

    Important: Never commit the .env file. It's already in .gitignore.

Usage

Phase 1 & 2: Building the Vector Database

  1. Place PDF files in the data/ folder

  2. Run the prototype:

    python prototype.py

    This will:

    • Load all PDFs from data/ folder
    • Split them into chunks (1000 chars, 200 overlap)
    • Generate embeddings using all-MiniLM-L6-v2
    • Store in ChromaDB at ./chroma_db/
  3. To regenerate embeddings:

    python prototype.py --regenerate-embeddings

Phase 3: Extracting Requirements

  1. Ensure you have a ChromaDB ready (created in Phase 1 & 2, or manually created)

  2. Run the extraction script:

    python extract_minimalist.py

    This will:

    • Load ChromaDB from ./un_r152_db/ (or configure your path)
    • Search for chunks using predefined search terms
    • Extract structured requirements using GPT-4.1 Nano
    • Display results with token usage statistics
  3. Customize search terms in extract_minimalist.py:

    SEARCH_TERMS = [
        "Braking performance",
        "Warning signals",
        "System requirements",
        "Safety standards"
    ]

Configuration

prototype.py Configuration

PDF_FOLDER = "data"
CHUNK_SIZE = 1000          # Characters per chunk
CHUNK_OVERLAP = 200        # Overlap between chunks
COLLECTION_NAME = "regulatory_documents"
CHROMA_PERSIST_DIR = "./chroma_db"

extract_minimalist.py Configuration

CHROMA_DB_PATH = "./un_r152_db"
EMBEDDING_MODEL = "intfloat/e5-small"
LLM_MODEL = "gpt-4.1-nano"  # Falls back to gpt-4o-mini if unavailable
TEMPERATURE = 0
K_CHUNKS = 2                # Number of chunks to retrieve per search

Dependencies

  • langchain: Core LLM framework
  • langchain-community: Community integrations (PDF loaders, vector stores)
  • langchain-openai: OpenAI integration
  • langchain-huggingface: HuggingFace embeddings
  • sentence-transformers: For embedding generation
  • chromadb: Vector database
  • pymupdf: PDF processing
  • pydantic: Schema validation
  • python-dotenv: Environment variable management

See requirements.txt for version specifications.

Architecture

┌─────────────┐
│   PDFs      │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Phase 1:    │  PyMuPDFLoader → Text Extraction
│ Ingestion   │  RecursiveCharacterTextSplitter → Chunking
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Phase 2:    │  SentenceTransformer → Embeddings
│ Embeddings  │  ChromaDB → Vector Storage
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Phase 3:    │  Similarity Search → Relevant Chunks
│ Extraction  │  GPT-4.1 Nano → Structured Extraction
└─────────────┘

Examples

Example Output from extract_minimalist.py:

================================================================================
CATEGORY 1: BRAKING PERFORMANCE
================================================================================

Extracted 2 requirement(s):

  Requirement 1:
    Type: Must-have
    Requirement: The braking system must achieve a minimum deceleration of 5.0 m/s²
    Metric: 5.0
    Unit: m/s²

  Requirement 2:
    Type: Should
    Requirement: The system should provide warning signals before braking
    Metric: None
    Unit: None

  Tokens used: 342

================================================================================
SUMMARY
================================================================================
Total requirements extracted: 8
Total tokens used: 1245
Categories processed: 4

Future Enhancements

  • More flexible schema definitions
  • Batch processing for multiple documents
  • Evaluation metrics for extraction quality
  • Web interface for querying and visualization

About

prototypes for AI in regulatory compliance

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages