Semantic De-Duplication of Shared Cyber Threat Indicators using Large Language Model Embeddings

Abstract

As defence agencies, allied partners, and cybersecurity firms exchange threat intelligence, the same indicator is frequently reported many times in slightly altered form, producing large-scale duplication across intelligence repositories. Traditional de-duplication relies on exact matching using hashes or literal text comparison and fails when information is reworded, paraphrased, or contextually recast. This paper proposes a semantic de-duplication approach that uses Large Language Model (LLM) embeddings to identify duplicate threat indicators on the basis of meaning rather than form. The method encodes each report into a contextual vector, measures pairwise similarity, clusters semantically equivalent reports, and retains a single representative entry per cluster. Situated within the literature on record linkage, entity resolution, and near-duplicate detection, the paper specifies an evaluation protocol based on precision, recall, F1-score, cluster-level agreement, and latency over open-source and synthetic corpora, together with the baselines and reproducibility requirements against which the design should be tested. That protocol is defined here but has not yet been executed: no dataset has been processed and no numerical results are reported. The behaviour described in the discussion is the behaviour the design is expected to exhibit, offered as hypotheses to be tested rather than as measured findings, and the anticipated benefits of consolidation, namely reduced noise, less exaggerated risk assessment, and lower computational and storage cost, are stated in those terms throughout. The paper contributes a meaning-based de-duplication design for defence CTI, an evaluation protocol for validating it, and a discussion of the accuracy, efficiency, and governance implications of deploying it.