Define several duplicate levels
Separate byte-identical files, repeated exports, visually equivalent documents, and duplicated business events. Two PDFs may encode the same invoice differently, while two rows with the same amount may represent separate purchases. The correct unit of deduplication depends on what the model or evaluation is supposed to learn.
Preserve revisions and status
An amended contract or corrected invoice may look almost identical to its predecessor but contain the most important change in the workflow. Use document identifiers, timestamps, status, and explicit revision links where available. Do not remove a near-duplicate automatically when the task involves version comparison or change detection.
Inspect clusters and source patterns
Review representative clusters from each duplication rule and measure how much material they account for. Repeated monthly exports can inflate the number of apparent independent records. Ask for unique entity, event, and template counts alongside file counts so collection scale is interpretable.
Apply split rules consistently
Keep duplicate families and closely related revisions together when independent evaluation requires it. Document the chosen grouping policy and any exceptions. Ask Fiund about source export practices and available stable identifiers before agreeing on a sourcing brief, because deduplication feasibility depends on the information the source actually preserves. A useful test cluster contains an original invoice, its corrected replacement, and two repeated exports of each. The expected handling should preserve the two meaningful versions while identifying redundant copies. Document which information establishes that distinction and what remains uncertain when stable identifiers or revision markers are absent from the source.
Review checklist
- Define file and event duplicate rules
- Preserve meaningful revision chains
- Inspect duplicate clusters manually
- Group related artifacts consistently across splits