Repository navigation
Conversation
The Markdown path refuses image-only pages with NeedsOcr because text is the whole output. DocLang's picture+location model can represent them faithfully, so add a PDF-specific pipeline that exports scanned pages instead of erroring: positioned runs become text/heading/footnote elements with formatting, links, fields, and normalized locations, and image XObjects become picture elements backed by extracted raster parts (JPEG/JP2 passthrough, decoded samples to PNG, CCITT wrapped in TIFF), all packed into a spec-0.9 OPC archive with the source PDF alongside. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Move the DocLang renderer under src/render/doclang — it is an output format, not an input format. Detected lists emit as list/ldiv; tables come from the structure tree's Table elements (tagged) and from drawn-rule grid analysis (ruled), emitted as OTSL-token table blocks. DoclangHints gains pages_with_tables to scope ruled-table detection. Review fixes: only cell-assigned items are consumed, tagged cells on other pages keep rectangular placeholders without bogus locations, malformed operands no longer corrupt the CTM, paths paint under the CTM active at the stroke, boundary clustering cannot chain-merge, decoded pixels are capped before allocation, the archive budget covers ZIP overhead, page content and Form XObjects decode once per page, and the document parses at most twice. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 14 files
Reply to a comment to ask cubic a question or push back. It learns from your replies.
Click Fix with cubic on a comment, or reply @cubic-dev-ai fix this, and cubic writes the fix.
View guided diff | Re-trigger cubic
Decode PDF streams (page content, Form XObjects, images, palettes, masks) through a bounded decoder — Flate/LZW/ASCII85 with lopdf's PNG predictors — instead of unbounded decompressed_content. Ruled tables cap rules at append and reject oversized grids before the quadratic cell matrix; rejected tagged fragments return their items to normal flow. Close-and-stroke ops (s/b/b*) now paint their implied closing segment. Images: bound the decode by declared dimensions before decompressing, apply /Decode remaps to samples, indices, and masks, map ICCBased N=4 to CMYK, reject Separation/DeviceN/Pattern, and fall back to XObject /Height when CCITT /Rows is absent. Emission: esc drops XML-1.0-invalid control chars, the field separator no longer sits between elements, furniture merges only with furniture, fused (iv)Text markers parse, pictures sharing an XObject name keep distinct srcs, explicit /Rotate 0 overrides a rotated ancestor, archive entry limits count OPC plumbing + source, and non-BOM titles decode as PDFDocEncoding. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
All reported issues were addressed across 12 files (changes from recent commits).
Requires human review: Auto-approval blocked because this review re-detected 1 unresolved P0–P2 issue already reported by Cubic.
Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack
View guided diff | Re-trigger cubic
Absent /DecodeParms now means predictor 1 instead of decode failure, /DecodeParms arrays are honored per filter (indirect entries resolved, nulls default), and an empty /Filter array returns the raw content. ASCII85 uses checked_add so a five-char overflow group fails cleanly instead of wrapping. Predictor handling no longer borrows lopdf's lenient path: TIFF predictor 2 reconstructs 8/16-bit horizontal differencing (other depths are rejected rather than corrupted), and PNG predictors use ceil(Colors*BPC/8) bytes-per-pixel with fully checked row-size arithmetic, so packed sub-byte components stay aligned and malformed dimensions cannot overflow before the stream bound applies. pixel_bound multiplies Width*Height with checked arithmetic, and list validation keeps every ordinal candidate for ambiguous markers like (i) so alpha runs (i)(j)(k) and roman runs both validate. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(1u8 << bpc) panics when bpc is 8 — a valid index size for Indexed color spaces — and silently produces a zero mask in release builds. Derive the mask by shifting 0xFF right instead. Found while running the Docling PDF groundtruth through the DocLang export; four of the eighteen documents hit the panic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Export Docling's serialized PDF DoclingDocuments to DocLang and use them, with their PDF sources, as the source of truth for the DocLang export: every document must produce well-formed .dclx whose text covers at least 95% of the reference's characters. The harness reads PDFs from a sibling Docling checkout (DOCLING_PDF_TESTDATA or ../docling/tests/data/pdf) and skips cleanly when absent. Reference exports are vendored under tests/fixtures/doclang and regenerated by scripts/export_docling_doclang.py. All outputs also round-trip through Docling's own DocLang deserializer. Known divergences are documented in the test: newspaper-00 fails in pdf-inspector, right_to_left_* keeps visual order where Docling emits logical order, and Docling's layout model detects structures beyond the tagged/ruled-table and font heuristics. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
2 issues found across 21 files (changes from recent commits).
Confidence score: 4/5
scripts/export_docling_doclang.pycan report success with an empty groundtruth directory, leaving fixtures stale or missing. Reject empty input before regenerating.scripts/export_docling_doclang.pyrelies on the host locale when reading Docling JSON, so regeneration can fail on non-ASCII text. Read the JSON as UTF-8 explicitly.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="scripts/export_docling_doclang.py">
<violation number="1" location="scripts/export_docling_doclang.py:60">
P2: Reject an empty groundtruth directory instead of reporting successful regeneration; a wrong or incomplete checkout can currently leave stale or missing fixtures while returning success.</violation>
<violation number="2" location="scripts/export_docling_doclang.py:65">
P2: Read the Docling JSON as UTF-8 explicitly; relying on the host locale can fail regeneration as soon as a document contains non-ASCII text.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
# Conflicts: # Cargo.toml
A /Title without a BOM is PDFDocEncoding, not Latin-1: bytes in the 0x80-0x9F range are printable punctuation (bullets, dashes, quotes) rather than C1 controls. Use lopdf's text-string decoder, which also covers the UTF-8 BOM the manual path missed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
name_for_index now returns Option<Result<Cow>> and ZipFile::name returns Result; flatten both layers in the test helpers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
tests/fixtures is the corpus sweep input set; reference exports do not belong there and broke the snapshot harness. Move them to tests/groundtruth and harden the regeneration script: reject an empty groundtruth dir, read JSON as UTF-8, and prune stale .doclang outputs only after a fully successful run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
All reported issues were addressed across 6 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add export to the DocLang 0.9 standard format for documents. It will help to have more exchange between Firecrawl and the Docling communities. Also, we get a nice viewer for parsed documents for free (https://doclang.ai/viewer/).
This PR is still initial work and limited to PDF documents. My agent had a blast building it, I did some reasonable review of the code and tests.
Summary
to_doclang/to_doclang_bytes: PDFs export to.dclx(OPC ZIP) archives containingdocument.xml, extracted image/page rasters underassets/andpages/, and the original PDF under/source.<picture>elements with location boxes —NeedsOcris no longer the outcome for this path.<field_item>), drawn images, and page geometry are preserved where practically possible.<list>/<ldiv>/<marker>rather than flattened text./Tablestructure-tree elements are matched by MCID, and a ruled-table engine clusters drawn axis-aligned rules into grids (spans →lcel/ucel/xcel); both emit OTSL-token<table>blocks.DoclangHints::pages_with_tablesscopes the geometric detector.src/render/doclang/— DocLang is an output format, not an input format.ResourceLimit), page content and Form XObjects decode once per page, and the document parses at most twice (upstream parse-dedup filed as Expose &lopdf::Document variants of *_mem APIs to avoid double parsing pdf-inspector#662).Test plan
cargo test— 315 unit + integration tests (lists, tagged/ruled tables, spans, headers, q/Q CTM, malformed operands, image caps, packaging limits,.dclxround-trip)cargo clippy -- -D warningscleancargo fmt --checkcleanexamples/convert.rsGenerated with Devin
Summary by cubic
Adds DocLang
.dclxarchive export for PDFs through newto_doclang/to_doclang_bytesAPIs, now also exposed from Python. Scanned pages no longer end inNeedsOcr— image-only pages export as<picture>elements with extracted rasters, so this path works without OCR.<list>/<table>blocks via tagged/Tablestructure-tree elements and a ruled-line grid engine./source; rasters pass through as JPEG/JP2 or wrap to PNG/TIFF underassets/andpages/.DoclangHintstoggles list/table detection and scopes the geometric table detector to specific pages; Pythonto_doclang/to_doclang_bytesaccept the samelists,tables, andpages_with_tablesoptions.Robustness
Written for commit e741f06. Summary will update on new commits.