Benchmarking Document AI on Pharmaceutical Manufacturing Records: Why DocVQA Isn’t Enough

Executive Summary

We evaluated Litewave's end-to-end data pipeline and retrieval framework across a combined benchmark comprising the standard DocVQA validation set and a specialized set of complex enterprise pharmaceutical documents, including manufacturing batch records, Master Batch Records (MBRs), life science documentation, and chromatography reports. Across 7,322 total evaluation queries, our system achieved 6,891 correct extractions, overall accuracy 94.11%.

Standard public benchmarks like DocVQA provide a valuable public baseline for general visual document QA, but they do not capture the operational complexity of pharmaceutical manufacturing paperwork. By combining parsed structured records from our Data Extraction Engine (DEE) with a multi-stage review pipeline, Litewave enables reliable, low-latency document processing and cross-document retrieval without requiring constant visual re-processing or raw VLM access during querying.

Why We Built a Combined Benchmark

Public document AI benchmarks such as DocVQA are constructed primarily from standard business documents: letters, forms, memos, and single-page reports. While useful for general document understanding, they fail to represent the reality of pharmaceutical manufacturing, which relies on:

• •Multi-page executed Master Batch Records (MBRs)
• •Dense handwritten entries and operator sign-offs
• •Complex chromatography reports and Certificates of Analysis (CoAs)
• •Cross-document queries where an answer depends on correlating data across dozens of pages

A document AI system can achieve high scores on DocVQA yet fail in production when attempting to read a handwritten critical control parameter on page 39 of a batch record.

To address this gap, we evaluated our system on both standard DocVQA and a dedicated pharma evaluation set built from de-identified executed batch records and lab documents from the manufacturing floor. DocVQA provides a public, reproducible benchmark, while our pharma set measures what matters in production: handwriting precision, cross-document context, and zero-tolerance accuracy for regulatory compliance.

Dataset Composition & Statistics

To maintain full transparency, we report the statistics of our evaluation corpus separated by source:

Note: While the initial pharma benchmark was constructed using a limited set of 36 representative documents, the dataset is continuously expanding to encompass a broader array of document structures and manufacturing workflows

Question types evaluated: Factual extraction, spatial table understanding, cross-document correlation, and handwritten field parsing.

Evaluation Results & Benchmark Performance

DocVQA Benchmark

• •DocVQA Dataset Accuracy: Across 5,349 queries on the standard DocVQA validation split, our extraction pipeline achieved an overall accuracy of 93.81% establishing a robust baseline for general visual document understanding.

Pharmaceutical Manufacturing Benchmark

• •Pharma Dataset Accuracy: Across 1,973 queries on complex batch records and CoAs, our specialized data extraction engine and multi-stage review pipeline achieved an overall accuracy of 94.93%, directly addressing domain-specific challenges like dense handwriting and cross-document correlation.

Combined Benchmark (DocVQA + Pharma)

• •Combined Dataset Accuracy: On all 7,322 queries across both DocVQA and Pharmaceutical Dataset, our specialized data extraction engine and multi-stage review pipeline achieved an overall accuracy of 94.11%.

Note: DocVQA uses ANLS (Levenshtein similarity with a 0.5 threshold) to score text extraction. We used a hybrid approach: standard ANLS for text, exact match for numbers/dates.

Architectural Enhancements to Reach Final Accuracy

• •Single-Pass Document Extraction: Executes optimized OCR and layout analysis in a single pass, converting visual content into structured JSON schemas. This eliminates repetitive visual reprocessing, significantly reducing compute overhead and operational latency.
• •Targeted Multi-Stage Review Pipeline: Integrates a specialized layout extraction layer above standard OCR to identify metadata and structural relationships. This architecture drives robust cross-document reasoning while seamlessly incorporating human-in-the-loop review for high-fidelity verification.

Error Classification & Transparency

In regulatory environments like GMP batch release, understanding error modes is as critical as overall accuracy. Out of 7,322 total evaluation queries, 431 resulted in misses due to extraction failures:

Note: There were 13 dataset ground truth issues due to inaccuracies in the ground truth labels of the DocVQA evaluation dataset (including six question-answer pairs containing incorrect answers).

Quality Assurance & Compliance Safeguards

• •Silent Errors: In highly regulated GMP environments, an undetected misextraction poses a far greater risk than an unextracted field. Out of 431 extraction failures, only 54 were incorrectly flagged as high-confidence extractions, translating to a remarkably low unflagged error rate of less than 1% across all 7,322 evaluated queries.
• •Full Data Lineage: Every extracted data point maintains an immutable, traceable link to its origin document and page number.

Technical Advantages over Standalone VLMs

While Vision-Language Models (VLMs) demonstrate strong single-page visual comprehension, they present distinct operational challenges in enterprise pharmaceutical settings:

• •Handwritten Data Extraction: Specialized vision engines and detection heads adapts to operator-specific handwriting styles, outperforming general VLMs in extracting dense, cursive, or irregular handwritten entries in cGMP batch production records.
• •Cross-Document Context: A unified database schema preserves relationships across multi-page dossiers and multi-document workflows, overcoming the single-page visual isolation of standard VLMs.
• •Scalable Low-Latency Retrieval: By decoupling single-pass document ingestion from rapid structured retrieval, the platform provides immediate query responses at a fraction of the compute cost required to re-analyze high-resolution images.

References

1. 1.DocVQA: A Dataset for VQA on Document Images.
2. 2.Evaluated Dataset: Single Page DocVQA (Task 1) from the ICDAR 2021 Competition on Document Visual Question Answering. Accessible at: https://rrc.cvc.uab.es/?ch=17