NestDiff Checker

Structured file comparison tool using AI-based consistency evaluation.

Features

• Compare JSON files using HHEM or Qwen embedding models
• Handle nested structures and arrays with dot notation paths
• Configurable consistency thresholds
• Detailed field-level difference reporting

Installation

pip install -r requirements.txt

Quick Start

from nestdiff import Checker

checker = Checker(judge_name="qwen", consistency_threshold=0.7)
report = checker.check(predictions_dir=Path("sample/predictions"), references_dir=Path("sample/references"))
checker.report(report, Path("sample/comparison_report.txt"))  # save report to file

Configuration

Judge Models:

• "hhem": vectara/hallucination_evaluation_model
• "qwen": Qwen/Qwen3-Embedding-0.6B

Threshold: 0.0-1.0 (higher = stricter comparison for similarity)

Sample

Check sample/ directory for book summary use case. Run with python sample/main.py.

Output

Returns ComparisonReport with:

• total_files, inconsistent_files, consistency_ratio
• differences: List of FieldDifference objects with file, field path, actual and expected values, and scores