Skip to content

feat(knowledge): parse SpreadsheetML xls exports - #2332

Open
xyi74976-del wants to merge 1 commit into
TencentCloud:developfrom
xyi74976-del:feat/knowledge-spreadsheetml
Open

xyi74976-del wants to merge 1 commit into
TencentCloud:developfrom
xyi74976-del:feat/knowledge-spreadsheetml

Conversation

@xyi74976-del

Copy link
Copy Markdown

Summary

Closes #1879. Some business systems export Excel 2003 SpreadsheetML XML under a .xls filename. The existing importer passes it to xlrd, so the indexing job fails with “Expected BOF record” even though the data is a valid workbook.

Recognize XML exports and extract their cached cell values through a separate, bounded parser. Binary BIFF .xls files retain the xlrd path. The same extractor serves knowledge preview and indexing.

  • Preserve UTF-8/UTF-16 text, long string identifiers, sheet order and sparse columns. Compose ss:Index with LeftCell/TopCell and horizontal merge gaps before checking final coordinates.
  • Index cached formula values; leave missing caches and Excel error values empty. No formula calculation, links, styles or macros are executed.
  • Reject DTDs, malformed/foreign XML and invalid indices; bound input bytes, XML depth/elements, and Excel 2003 dimensions. Expanded-count metadata does not allocate blank rows.
  • Preserve uploaded originals, document the supported subset and limits, and verify real upload/preview/SQLite indexing and retrieval with synthetic vectors.

No new dependencies, schema changes, upload extension, or configuration knob.

Target branch

  • Base is develop
  • Base is main

Type of change

  • New feature / format compatibility
  • Bug fix

Test plan

  • Eight final valid XML export fixtures fail with real XLRDError through pristine develop@6ae53c2; they cover five encodings, sparse/multiple sheets, cached formulas/rich text and merge gaps.
  • 129 knowledge tests passed, including actual binary BIFF compatibility, XML rejection/resource bounds, original-byte preservation and real SQLite indexing. Readiness and embedding vectors are stubbed; no external model request is claimed.
  • Independent AI review found a Table-origin omission. Nine regressions reproduced it before correction. Read-only re-review and seven independent edge probes found no remaining actionable issue.
  • make all PYTEST_JOBS=2: 5157 passed, 34 skipped, Ruff and mypy passed.
  • Enabled pre-commit hook: 5157 passed, 69 skipped, followed by successful TypeScript/Vite/PWA build. No bypass.
  • git diff --check passed. No native Windows or live embedding-provider test was run; remote CI is separate.

This is text extraction, not full spreadsheet rendering/XSD validation. Number/DateTime text is retained without display-format conversion. Blank rows/sheets and uncached formulas contribute no text. Merged ranges are not fully validated or expanded vertically. XML extraction is additionally limited to 16 MiB, 500,000 elements and depth 64; the normal upload limit also applies.

Actual dashboard comparison

The same synthetic .xls was uploaded through the actual Octop HTTP endpoint in two isolated temporary instances. Authentication, background indexing, SQLite, preview API and React dashboard are real; embedding readiness/vectors and the unused harness are test fixtures. The before case substitutes only pristine parse.py at the existing jobs/service boundary.

Before: failed, zero chunks, XLRD BOF error.

Before: XML workbook cannot be indexed

After: ready, one chunk. Preview and actual SQLite vector search retain 订单 0001234567890123456789 and 12.25; original bytes are unchanged.

After: the same XML workbook is ready and indexed

Recorded job states and extracted/retrieved values

Checklist

  • Updated CHANGELOG.md and format/limit documentation
  • Real UI screenshots provided
  • Regression and integration tests added
  • All upstream CI workflows passing (pending execution)
  • AI-assisted scope aligned and independent review completed
  • One focused format-compatibility change

AI assistance: Codex for investigation, implementation, tests, integration/browser evidence, independent review and this description.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant