benchmark report
How We Test SolConverter: Internal Stress Tests and 6,695 Third-Party Benchmark Cases
See how SolConverter tests Markdown, PDF, Office documents, CSV, JSON, XML, EPUB, OCR, malformed files, and large workloads.
Explore SolConverter toolsSee how SolConverter is tested with a dedicated internal stress-test corpus and independent community benchmark suites covering Markdown, PDF, Office documents, CSV, JSON, XML, EPUB, OCR, malformed files, and large workloads.
Testing a file converter with a few clean example files tells you very little about how it behaves when the documents become difficult.
Real files contain unusual encodings, nested structures, malformed syntax, enormous tables, empty worksheets, broken containers, mathematical expressions, diagrams, OCR layers, multi-column PDFs, and combinations of features that rarely appear in simple demo files.
SolConverter therefore uses two separate testing tracks.
The first is an internal stress-test corpus built specifically around the conversion problems SolConverter needs to handle.
The second uses independent third-party and standards-based test suites created outside SolConverter.
In the latest third-party benchmark run, dated August 7, 2026, SolConverter recorded 6,614 passing outcomes from 6,695 benchmark cases, for a benchmark pass rate of 98.79%.
We intentionally report the internal dataset and third-party benchmark separately. They answer different questions and should not be combined into one inflated test count.
Want to test a real file instead of a benchmark fixture? Explore SolConverter tools.
What are we actually testing?
File conversion is not one problem.
A Markdown-to-PDF converter needs to handle syntax, diagrams, math, fonts, images, and pagination. A PDF-to-Markdown converter needs to reconstruct text, tables, formulas, layout, and reading order. Spreadsheet converters need to reason about worksheets, headers, merged cells, types, and formulas. CSV, JSON, and XML converters face parser, encoding, nesting, and malformed-input problems.
That means a useful test program needs to evaluate several dimensions separately:
| Test dimension | What it tries to expose |
|---|---|
| Parsing | Whether the source format is interpreted correctly |
| Conversion | Whether meaningful source information reaches the target format |
| Rendering | Whether visual output such as PDF remains usable |
| Structure | Whether rows, columns, headings, tables, and ordering survive conversion |
| Error handling | Whether invalid input produces controlled behavior |
| Large-file behavior | Whether problems appear as input size increases |
| Regression | Whether a previously fixed problem returns |
| Compatibility | Whether unusual but real format variants are handled consistently |
A single percentage cannot describe all of these dimensions.
That is why the benchmark keeps individual case classifications instead of treating every successful HTTP response as proof of exact output correctness.
Why do we use two separate testing tracks?
Internal tests and third-party benchmarks catch different types of failures.
Internal fixtures let us deliberately stress SolConverter-specific behavior.
Independent suites reduce the risk of building a test dataset around the same assumptions used to build the product.
We therefore use this model:
SolConverter testing
│
├── Internal test corpus
│ ├── product-specific edge cases
│ ├── large-file stress tests
│ ├── rendering tests
│ ├── OCR tests
│ └── regression fixtures
│
└── Third-party benchmark suites
├── standards-based cases
├── community corpora
├── malformed-input cases
├── reference outputs
└── independently created edge cases
Neither track replaces the other.
A converter can pass a parser conformance test and still struggle with a massive real-world file.
It can also process a 300 MB internal stress fixture successfully while failing a tiny standards edge case created by an independent project.
How does SolConverter test its converters internally?
The internal test dataset spans the actual conversion workflows supported by SolConverter and deliberately includes difficult, unusual, and high-load inputs.
The current internal corpus covers approximately 18 conversion and viewing workflows, including:
- Markdown to PDF;
- README to PDF;
- PDF to Markdown;
- Word to Markdown;
- Excel to Markdown;
- PowerPoint to Markdown;
- HTML to Markdown;
- EPUB to Markdown;
- CSV to Markdown;
- JSON to Markdown;
- XML to Markdown;
- Text to Markdown;
- JSON to CSV;
- CSV to JSON;
- XML to CSV;
- Excel to JSON;
- JSON to Excel;
- CSV Viewer.
The internal dataset is separate from the 6,695-case external benchmark.
Large-file and load testing
Large-file fixtures are used to expose failures that may never appear in small demo documents.
Representative internal files include:
| Fixture | Approximate size | Test target |
|---|---|---|
json_array_500MiB.json |
~500 MB | Large JSON processing |
CSV_TO_JSON_LOAD_TEST_313MB.csv |
~313 MB | High-volume CSV conversion |
XML_TO_CSV_LOAD_TEST_313MB.xml |
~313 MB | Large XML parsing and conversion |
real_document.docx |
~22.5 MB | Complex Word processing |
scanned_book_OCR_DONE_has_text_layer.pdf |
~26.3 MB | Scanned PDF with OCR text layer |
json_to_excel_all_edge_cases.json |
~3.16 MB | Complex JSON flattening |
These files are intended to expose problems such as:
- memory pressure;
- parser slowdown;
- unexpectedly expensive transformations;
- incomplete output;
- timeouts;
- scaling problems;
- failures that only emerge after processing large amounts of data.
File size alone is not considered sufficient stress coverage. The corpus combines size with difficult structures and format-specific edge cases.
Markdown and Markdown-to-PDF testing
Internal Markdown tests combine individual syntax edge cases with very long and rendering-heavy documents.
One long-document fixture, 1200_page_long_test.md, is approximately 4.17 MB and is designed to exercise rendering across more than 1,200 pages.
It targets areas such as:
- long-running rendering;
- pagination;
- table-of-contents behavior;
- page numbering;
- document continuity.
Separate files such as 92_case.md and 30_case.md cover combinations of:
- deeply nested lists;
- nested blockquotes;
- fenced code blocks;
- complex tables;
- footnotes;
- bold and italic combinations;
- escaped characters;
- Unicode;
- emoji;
- unusual Markdown structures.
A dedicated mermaid_test.md fixture adds diagram scenarios including flowcharts, sequence diagrams, Gantt diagrams, and class diagrams.
The important part is interaction testing.
A table may work in isolation. A code block may work in isolation. Unicode may work in isolation. Problems often appear when several features occur in the same document.
Internal stress testing is intended to reveal those interactions.
PDF and OCR testing
PDF-to-Markdown testing focuses on document reconstruction rather than merely checking whether some text can be extracted.
The internal corpus includes difficult document types.
The Attention Is All You Need paper provides a scientific-document fixture containing:
- two-column layout;
- mathematical expressions;
- figures;
- tables;
- captions;
- references.
Two-column layouts are useful for detecting incorrect reading order, where extraction software can accidentally merge lines horizontally across columns.
RFC 9110 provides a different type of stress case:
- a long technical document;
- deeply nested headings;
- preformatted blocks;
- structured technical text;
- index-like content.
OCR scenarios are tested separately.
Files such as scan_image_only_ocr.pdf and ocr/ocr_test_scanned.pdf contain scanned content that cannot be handled through ordinary PDF text extraction alone.
Another fixture, scanned_book_OCR_DONE_has_text_layer.pdf, contains both scanned images and an existing OCR text layer.
That case helps reveal problems such as duplicated text when a system performs OCR on content that already contains recognized text.
Word-to-Markdown testing
Word testing evaluates how complex page-oriented structures survive conversion into linear Markdown.
The internal real_document.docx fixture is approximately 22.5 MB.
It includes structures designed to exercise:
- nested tables;
- images;
- inline and positioned elements;
- headers;
- footers;
- multiple sections;
- table-of-contents structures;
- complex formatting.
DOCX conversion is not equivalent to copying paragraph text.
Word supports page-oriented and positioned structures that do not have direct Markdown equivalents. The converter therefore needs to preserve useful information while producing a readable linear document.
Excel converter testing
Excel tests cover both ordinary spreadsheet data and assumptions that often break on unusual workbooks.
Internal spreadsheet cases include:
- multiple worksheets;
- empty sheets;
- merged cells;
- formulas;
- dates;
- percentages;
- currencies;
- mixed data types;
- shifted data;
- worksheets without obvious headers.
One group of benchmark failures revealed a specific header-detection problem.
An earlier benchmark run produced 74 cases associated with HEADER_ROW_OUT_OF_RANGE behavior.
The old handling assumed that a usable header would always exist where expected.
That breaks on empty worksheets and files where the first meaningful row is not a conventional header.
The converter was subsequently updated with header_mode behavior supporting modes such as:
required;auto;none.
The latest supplied benchmark summary reports that these 74 header-related cases were resolved in the subsequent run.
This is one of the main reasons we keep difficult fixtures after a bug has been fixed: they become regression tests.
PowerPoint-to-Markdown testing
PowerPoint testing focuses on preserving meaningful reading order from spatial slide layouts.
The internal real_presentation.pptx fixture includes:
- text boxes;
- shapes;
- images;
- independently positioned objects;
- multiple slides.
Unlike Word, slide content may not have a natural continuous reading order.
A PowerPoint-to-Markdown converter therefore has to translate spatial layout into a useful text sequence without silently losing important content.
CSV converter testing
CSV testing includes cases that frequently expose simplistic parser assumptions.
The internal CSV_TO_JSON_EDGE_CASES.csv fixture covers situations such as:
- commas inside quoted values;
- line breaks inside fields;
- escaped double quotes;
- uneven rows;
- Unicode;
- alternative character encodings.
Large-file behavior is tested separately with fixtures including a CSV workload of approximately 313 MB.
This separation is important.
A small file can be structurally difficult.
A large file can be structurally simple but computationally expensive.
Both need testing.
JSON converter testing
JSON tests focus heavily on the difficulty of mapping hierarchical structures into tabular formats.
The internal json_to_excel_all_edge_cases.json fixture is approximately 3.16 MB and includes:
- nested objects;
- arrays containing objects;
- nested arrays;
- null values;
- booleans;
- high-precision numeric values.
These structures are particularly relevant to JSON-to-CSV and JSON-to-Excel conversion because the converter must decide how hierarchy becomes rows and columns.
A separate approximately 500 MB JSON fixture is used for large-data stress testing.
XML converter testing
Internal XML tests cover both mapping complexity and file size.
The corpus includes:
- nested elements;
- attributes;
- malformed structures;
- flattening scenarios;
- large XML documents.
Representative load fixtures include approximately:
- 50 MB XML;
- 100 MB XML;
- 313 MB XML.
These cases test SolConverter-specific XML conversion behavior.
They are reported separately from the W3C XML Conformance Test Suite used in third-party benchmarking.
How does the third-party benchmark work?
The external benchmark exposes SolConverter to test material that was not created specifically for SolConverter.
The latest benchmark report includes 6,695 third-party cases.
SolConverter produced 6,614 passing outcomes, giving:
98.79% third-party benchmark pass rate
The 6,614 passes are divided into three categories.
| Result | Cases | Meaning |
|---|---|---|
| Verified Pass | 23 | The benchmark supplied sufficient expected-result information for direct verification |
| Unverified Success | 4,742 | Conversion succeeded, but sufficient target-format ground truth was unavailable for exact output verification |
| Handled Rejection | 1,849 | Rejection was the expected behavior and the input was rejected |
| Total Pass | 6,614 | 98.79% of 6,695 cases |
These three categories all count as passing benchmark outcomes.
They do not, however, provide exactly the same type of evidence.
What does a benchmark pass mean?
A pass means SolConverter produced the behavior expected by that particular test.
That expected behavior is not always “produce a converted file.”
Verified Pass
A Verified Pass has sufficient benchmark ground truth to compare the actual result with an explicit expectation.
These provide the strongest direct correctness evidence in the current result set.
Unverified Success
An Unverified Success is also a pass.
It means the test completed successfully according to the execution criteria, but the external corpus did not provide sufficient expected target output to claim an exact ground-truth match.
For example, a third-party Office corpus may contain an unusual workbook that is useful for determining whether the parser can process the file, but it may not provide a canonical Markdown representation of that workbook.
Calling the conversion an exact output match would overstate what the dataset proves.
Calling it a failed test would also be incorrect.
That is why it receives its own passing category.
Handled Rejection
A Handled Rejection is a pass when the test expects rejection.
For example:
| Expected | Actual | Result |
|---|---|---|
| Reject malformed file | File rejected | PASS |
| Reject unsafe input | Input blocked | PASS |
| Reject corrupt container | Container rejected | PASS |
A converter should not blindly attempt to process every file it receives.
For some inputs, stopping before conversion is exactly the correct behavior.
Which independent test suites do we use?
Different formats require different benchmark resources.
We currently draw from several established projects and standards-oriented datasets.
| Benchmark resource | Primary role | Relevant SolConverter workflows |
|---|---|---|
| cmark-gfm | Markdown parser/reference tests | Markdown to PDF, README to PDF |
| diff-pdf | Visual PDF comparison | PDF rendering regression |
| OmniDocBench | Document parsing benchmark | PDF to Markdown |
| Microsoft MarkItDown Tests | Office/document fixtures | Office to Markdown |
| Apache POI Test Data | Microsoft Office test corpus | Word, Excel, PowerPoint, Excel converters |
| Turndown Attendant | HTML-to-Markdown fixtures | HTML to Markdown |
| EPUBCheck | EPUB validation and conformance | EPUB to Markdown |
| csv-spectrum | CSV parser edge cases | CSV converters and viewer |
| JSONTestSuite | JSON parser test corpus | JSON converters |
| W3C XML Conformance Test Suite | XML processor/conformance cases | XML converters |
cmark-gfm
cmark-gfm is GitHub's extended version of the CommonMark reference implementation, including GitHub Flavored Markdown extensions.
It provides independently maintained Markdown behavior and testing material useful for parser-oriented checks.
diff-pdf
diff-pdf addresses a different problem.
Instead of deciding whether Markdown syntax parsed correctly, it compares final PDFs visually.
That distinction matters because a document can parse correctly while still producing:
- clipped diagrams;
- missing fonts;
- incorrect pagination;
- shifted tables;
- rendering regressions.
OmniDocBench
OmniDocBench provides document parsing evaluation material covering areas such as:
- text;
- formulas;
- tables;
- layout;
- reading order.
It is therefore useful for PDF-to-Markdown evaluation rather than simply testing whether an API produced a Markdown file.
Microsoft MarkItDown Tests
Microsoft's MarkItDown project includes test fixtures for document-to-Markdown workflows.
These provide additional independently created Office and document examples.
Apache POI Test Data
Apache POI maintains a substantial corpus of Microsoft Office files used by its own document-processing implementations.
This corpus is particularly useful because it contains unusual, historical, malformed, and difficult Office files that a small hand-written dataset is unlikely to reproduce.
Turndown Attendant
Turndown Attendant provides HTML-to-Markdown conversion fixtures.
These are relevant to HTML-to-Markdown behavior and regression testing.
EPUBCheck
EPUBCheck is the EPUB conformance checker maintained through the EPUB standards ecosystem.
Its validation corpus is useful for testing how EPUB inputs are classified before and during conversion.
csv-spectrum
csv-spectrum describes itself as a collection of CSV files intended as an acid test for CSV parsing libraries.
It includes cases involving formatting that often exposes parser differences.
JSONTestSuite
JSONTestSuite includes separate classes of JSON input that parsers should accept, reject, or may interpret differently.
This is useful for making a distinction that also matters to SolConverter:
rejecting invalid input can be correct behavior.
W3C XML Conformance Test Suite
The W3C XML test material contains different categories of XML processor cases, including valid, invalid, and not-well-formed examples.
These cases give the XML converters independently created structures that are distinct from SolConverter's own internal XML fixtures.
What did the benchmark reveal?
The benchmark has already exposed implementation assumptions that smaller happy-path tests did not reveal.
The most useful outcome of a large test corpus is not the final percentage.
It is the individual failure patterns.
Excel header handling
The 74 Excel header-related cases described earlier revealed that assuming a conventional first-row header was too restrictive.
The resulting fix changed converter behavior rather than simply changing the benchmark.
Mixed-case EPUB extensions
One case exposed a much smaller but still real problem.
An EPUB using the mixed-case .ePub extension could be rejected because extension handling was case-sensitive.
The extension check was subsequently normalized.
This illustrates why thousands of unusual cases are useful.
A test dataset containing only ordinary lowercase .epub files would never discover that behavior.
Unusual Office documents
Apache POI test data also exposes files that are:
- intentionally corrupt;
- fuzz-generated;
- encrypted;
- unusual historical variants;
- strict-format variants.
Those categories need triage rather than being treated as one generic converter failure.
An unsupported encrypted workbook is not the same engineering problem as a valid workbook that loses half its rows.
Evaluator differences
Some benchmark failures can come from the evaluation method itself.
For example, Markdown outputs can differ in whitespace or representation without necessarily losing the same amount of semantic information.
That does not mean the result should automatically pass.
It means the evaluator needs to measure the property that actually matters for the test.
Why do rejected files sometimes count as passes?
Because rejection can be the required behavior.
File-processing systems receive more than clean documents.
Inputs can be:
- malformed;
- corrupt;
- incomplete;
- unsupported;
- incorrectly named;
- intentionally adversarial.
Trying to process every input is not necessarily robust behavior.
SolConverter's XML pipeline, for example, includes an input-level XML security check described in the supplied benchmark material.
If input matches a condition that should be rejected and the system stops processing it accordingly, the benchmark outcome is a pass.
The important distinction is:
Did the system do what the test expected?
not:
Did every test produce a downloadable converted file?
How do benchmark failures improve the converters?
A benchmark becomes more valuable when every confirmed bug turns into a permanent regression test.
The basic cycle is:
- Run the benchmark.
- Identify an unexpected result.
- Reproduce the result.
- Confirm the expected behavior.
- Diagnose the underlying cause.
- Fix the converter if necessary.
- Keep the original file as a regression case.
- Run that case again in future benchmark versions.
This transforms an unusual user file from a one-time bug report into a repeatable engineering test.
The Excel header cases are an example.
The purpose was not merely to turn 74 red results into green results.
The purpose was to change an assumption in the converter and make sure that assumption cannot quietly return later.
What does the 98.79% result actually mean?
It means 6,614 of the 6,695 third-party benchmark cases produced the expected class of passing behavior under the benchmark rules.
It does not mean that every one of those cases received an exact output comparison.
The result is composed of:
- 23 Verified Pass cases;
- 4,742 Unverified Success cases;
- 1,849 Handled Rejection cases.
So the correct statement is:
SolConverter achieved a 98.79% pass rate on the 6,695-case third-party benchmark run.
The incorrect statement would be:
SolConverter has 98.79% universal conversion accuracy.
Those claims are not equivalent.
What are the limitations of this benchmark?
No fixed benchmark can represent every document that users will encounter.
Several limitations should be kept in mind.
Not every external dataset has target-format ground truth
This is why Unverified Success exists as a separate category.
Successful processing is useful evidence, but it is not automatically equivalent to exact semantic correctness.
Different converters need different metrics
There is no sensible universal comparison metric for all of:
- PDF rendering;
- OCR;
- Markdown;
- spreadsheets;
- nested JSON;
- CSV parsing;
- malformed XML.
A PDF visual diff and a spreadsheet row-count comparison answer different questions.
Performance depends on the workload
Large-file conversion time can change with:
- file size;
- number of elements;
- images;
- formulas;
- diagrams;
- OCR;
- fonts;
- page count;
- nesting;
- hardware;
- concurrency.
A benchmark timing is therefore an observed measurement under a defined workload, not a guaranteed processing time for every file of similar size.
The internal corpus evolves
New regressions become new test cases.
New converter capabilities introduce new fixtures.
This means internal test coverage should improve over time rather than remain frozen just to preserve a historical score.
Third-party suites also change
Independent projects update their tests, parsers, evaluators, and format support.
Benchmark results therefore need a date and enough methodology to understand what was actually tested.
How this benchmark report is verified
SolConverter benchmark claims in this article are separated by evidence source.
Internal test claims come from the SolConverter internal test inventory and regression-test documentation.
Third-party benchmark totals come from the August 7, 2026 SolConverter benchmark report.
Descriptions of external test projects are checked against the corresponding project or standards documentation.
The internal and third-party test counts are deliberately not merged.
The reported 98.79% refers specifically to the third-party benchmark pass rate:
6,614 passing outcomes from 6,695 benchmark cases.
It should not be interpreted as universal file-conversion accuracy.
Frequently asked questions
Did SolConverter pass 98.79% of the third-party benchmark?
Yes.
The August 7, 2026 benchmark recorded 6,614 passing outcomes from 6,695 benchmark cases.
That produces a benchmark pass rate of 98.79%.
Are Unverified Success cases passes?
Yes.
An Unverified Success is counted as a passing benchmark outcome because the conversion completed successfully under the benchmark rules.
It is reported separately because the external dataset did not provide sufficient target-format ground truth for an exact output comparison.
Are Handled Rejections passes?
Yes.
If a test expects an invalid, malformed, corrupt, unsafe, or unsupported input to be rejected and SolConverter rejects it correctly, the test passes.
Does 98.79% mean 98.79% exact conversion accuracy?
No.
The 98.79% figure is a benchmark pass rate.
Only the Verified Pass category represents cases with sufficient expected-result information for direct verification under that category.
Are the internal tests part of the 6,695 cases?
No.
The 6,695 cases refer specifically to the third-party benchmark.
The SolConverter internal stress-test corpus is maintained and reported separately.
What does SolConverter test internally?
The internal corpus spans large files, Markdown syntax, PDF rendering, Mermaid diagrams, OCR, Word documents, Excel workbooks, PowerPoint presentations, CSV, JSON, XML, malformed inputs, and converter-specific regression cases.
Some internal structured-data fixtures reach hundreds of megabytes.
Why use public benchmark suites if SolConverter already has internal tests?
Because independent datasets can reveal assumptions that an internally designed corpus may miss.
Internal tests let us target known SolConverter problems.
External suites expose the converter to cases created independently of the product.
Does every successful conversion have verified ground truth?
No.
That is exactly why the benchmark distinguishes Verified Pass from Unverified Success.
Can I use the benchmark to predict how fast my file will convert?
Not precisely.
Processing time depends on the contents of the file as well as its size.
A large plain-text document and a similarly sized document containing thousands of images, equations, diagrams, or OCR pages can have very different processing costs.
Next step
The purpose of these benchmarks is not to replace testing with your own files.
It is to make the conversion pipeline encounter more difficult cases before users do.
If you have a document that has caused problems in another converter, that is often a more useful test than a clean sample created specifically for a demo.
Explore all SolConverter tools and test the format you actually use.
As new edge cases are discovered, confirmed regressions can be added to the internal corpus so future converter changes can be tested against the same input again.