Getting Clean Text Out of PDF, DOCX, and HTML

Short summary

Clean text extraction from PDFs is the unglamorous bottleneck behind every summarization, embedding, and TTS pipeline. The author covers column detection via horizontal projection, reading-order sorting, boilerplate removal by cross-page repetition, footnote separation by font size, and the honest limits of borderless table extraction. Real font metrics and per-page analysis beat naive heuristics.

• •Detect PDF columns geometrically before ordering text — sort by y within columns, then columns left-to-right
• •Remove running heads and page numbers by detecting repetition across pages, not by position
• •Footnotes and tables need separate extraction streams; borderless table structure recovery is unreliable and should be flagged as a product decision

Generated with AI, which can make mistakes.