A prototype system for extracting structured information from regulatory compliance documents using Large Language Models (LLMs) and vector databases.
This project provides a multi-phase approach to process regulatory PDFs, convert them into searchable vector embeddings, and extract structured requirements using LLM-based information extraction.
ai_in_reg/
├── prototype.py # Main prototype: PDF ingestion, chunking, and embedding
├── extract_minimalist.py # Requirement extraction using LLM and Pydantic schemas
├── requirements.txt # Python dependencies
├── data/ # Directory for PDF documents
├── chroma_db/ # Vector database storage (gitignored)
└── un_r152_db/ # Specific vector database for UN R152 regulation (gitignored)
- PDF Loading: Uses PyMuPDFLoader from LangChain for layout-preserving text extraction
- Intelligent Chunking: RecursiveCharacterTextSplitter with configurable chunk size (1000 chars) and overlap (200 chars)
- Metadata Preservation: Maintains page numbers and source file information for each chunk
- Batch Processing: Processes all PDFs in a folder automatically
Key Functions:
load_pdf_with_metadata()- Loads PDF with page metadatasplit_documents()- Splits documents into chunks with header awarenessprocess_pdf_folder()- Processes entire folder of PDFs
- Embedding Generation: Uses
all-MiniLM-L6-v2sentence transformer model - Vector Database: Stores embeddings in ChromaDB with persistence
- Smart Caching: Automatically detects and reuses existing embeddings
- Verification: Includes tools to verify storage and test similarity search
Key Functions:
load_embedding_model()- Loads sentence transformer modelcreate_embeddings()- Generates embeddings for all chunksstore_in_chroma()- Stores in ChromaDB with metadatacheck_existing_embeddings()- Checks if embeddings already exist
- Vector Search: Searches existing ChromaDB using semantic similarity
- LLM Extraction: Uses GPT-4.1 Nano (with fallback to GPT-4o-mini) for structured extraction
- Pydantic Schema: Validates extracted requirements against defined schema
- Token Tracking: Monitors and reports token usage per extraction
Extracted Fields:
type: Either 'Must-have' or 'Should'requirement: Core technical statementmetric: Numerical value (optional)unit: Unit of measure (optional)
-
Clone the repository:
git clone <repository-url> cd ai_in_reg
-
Create and activate virtual environment:
python -m venv .venv source .venv/bin/activate # On Windows: .venv\Scripts\activate
-
Install dependencies:
pip install -r requirements.txt
-
Set up environment variables: Create a
.envfile in the project root:OPENAI_API_KEY=your-api-key-hereImportant: Never commit the
.envfile. It's already in.gitignore.
-
Place PDF files in the
data/folder -
Run the prototype:
python prototype.py
This will:
- Load all PDFs from
data/folder - Split them into chunks (1000 chars, 200 overlap)
- Generate embeddings using
all-MiniLM-L6-v2 - Store in ChromaDB at
./chroma_db/
- Load all PDFs from
-
To regenerate embeddings:
python prototype.py --regenerate-embeddings
-
Ensure you have a ChromaDB ready (created in Phase 1 & 2, or manually created)
-
Run the extraction script:
python extract_minimalist.py
This will:
- Load ChromaDB from
./un_r152_db/(or configure your path) - Search for chunks using predefined search terms
- Extract structured requirements using GPT-4.1 Nano
- Display results with token usage statistics
- Load ChromaDB from
-
Customize search terms in
extract_minimalist.py:SEARCH_TERMS = [ "Braking performance", "Warning signals", "System requirements", "Safety standards" ]
PDF_FOLDER = "data"
CHUNK_SIZE = 1000 # Characters per chunk
CHUNK_OVERLAP = 200 # Overlap between chunks
COLLECTION_NAME = "regulatory_documents"
CHROMA_PERSIST_DIR = "./chroma_db"CHROMA_DB_PATH = "./un_r152_db"
EMBEDDING_MODEL = "intfloat/e5-small"
LLM_MODEL = "gpt-4.1-nano" # Falls back to gpt-4o-mini if unavailable
TEMPERATURE = 0
K_CHUNKS = 2 # Number of chunks to retrieve per search- langchain: Core LLM framework
- langchain-community: Community integrations (PDF loaders, vector stores)
- langchain-openai: OpenAI integration
- langchain-huggingface: HuggingFace embeddings
- sentence-transformers: For embedding generation
- chromadb: Vector database
- pymupdf: PDF processing
- pydantic: Schema validation
- python-dotenv: Environment variable management
See requirements.txt for version specifications.
┌─────────────┐
│ PDFs │
└──────┬──────┘
│
▼
┌─────────────┐
│ Phase 1: │ PyMuPDFLoader → Text Extraction
│ Ingestion │ RecursiveCharacterTextSplitter → Chunking
└──────┬──────┘
│
▼
┌─────────────┐
│ Phase 2: │ SentenceTransformer → Embeddings
│ Embeddings │ ChromaDB → Vector Storage
└──────┬──────┘
│
▼
┌─────────────┐
│ Phase 3: │ Similarity Search → Relevant Chunks
│ Extraction │ GPT-4.1 Nano → Structured Extraction
└─────────────┘
================================================================================
CATEGORY 1: BRAKING PERFORMANCE
================================================================================
Extracted 2 requirement(s):
Requirement 1:
Type: Must-have
Requirement: The braking system must achieve a minimum deceleration of 5.0 m/s²
Metric: 5.0
Unit: m/s²
Requirement 2:
Type: Should
Requirement: The system should provide warning signals before braking
Metric: None
Unit: None
Tokens used: 342
================================================================================
SUMMARY
================================================================================
Total requirements extracted: 8
Total tokens used: 1245
Categories processed: 4
- More flexible schema definitions
- Batch processing for multiple documents
- Evaluation metrics for extraction quality
- Web interface for querying and visualization