Text Distance from Nested and Hierarchical Repetitions: A Compression-Based Perspective

Short summary

The paper introduces Ladderpath, a structural sequence analysis method grounded in Algorithmic Information Theory that extracts nested and hierarchical repetition patterns from text. Three distance measures derived from this representation, when paired with a k-NN classifier, outperform both gzip-based NCD and BERT on out-of-distribution and few-shot text classification tasks. The approach offers a lightweight, interpretable, training-free alternative for domain-agnostic sequence understanding.

• •Ladderpath extracts nested/hierarchical repetition structures from text using Algorithmic Information Theory
• •Three new distance measures beat gzip-NCD and BERT on OOD and few-shot classification
• •Method is training-free, lightweight, and interpretable

Generated with AI, which can make mistakes.