Duplicate identification allows you to identify if your records are duplicated across all content sources managed by the platform. This feature helps to highlight any risks associated with the distribution of content and allows you to take the necessary actions to mitigate these risks.

How duplicate identification works

RecordPoint compares what is inside a file, not what the file is called or where it is stored. Two files with different names, saved in different content sources by different people, are duplicates if their content is the same. The comparison runs automatically as records are ingested, so no manual review is needed to find them.

The process works as follows.

1. The record’s binary is ingested. RecordPoint retrieves the file itself, rather than the metadata describing it.
2. The content is extracted from the file. This extracted content is what the platform compares.
3. A fingerprint is created. RecordPoint applies a hashing algorithm, MD5, to the extracted content. Hashing returns a short, fixed-length string, and that string is the fingerprint.
4. The fingerprint is paired with the file size. Together they form a composite key. Both parts of the key must match before two binaries are treated as duplicates.
5. Keys are stored and compared. As records are ingested, their keys are compared against the keys already held for every content source managed by RecordPoint. Where a match is found, the records are flagged as duplicates.

What a fingerprint is

A fingerprint is a short string calculated from the content of a file. It is useful because of two properties:

• Identical content always produces an identical fingerprint.
• Any change to the content, however small, produces a completely different fingerprint.

Together those properties let the platform answer the question “is this the same content?” by comparing two short strings, rather than comparing two whole files against each other.

Why the file size is included

The fingerprint on its own is not the whole test. Pairing it with the size of the file makes a false match very unlikely. For two different files to be flagged against each other incorrectly, they would need to produce the same fingerprint and be exactly the same size.

What gets matched

Duplicates are identified at the binary level rather than the record level. A single record can have more than one binary, and each binary is fingerprinted independently. When you view duplicates for a record, you are seeing the binaries that match that record’s binary.

Matching runs across every content source managed by RecordPoint, and across subdivisions of a single content source, even where users have different security access to those locations. A file in SharePoint and the identical file in a file share are matched with each other, not only against files held in the same repository.

RecordPoint describes the relationship between two binaries in one of the following ways.

What is not matched

A minimum binary size applies. Files below that size are left out of duplicate identification, because trivially small or near-empty files would otherwise match each other in very large numbers and report duplicates that are of no practical use.

Viewing duplicates for a record

To view the duplicates for an individual record:

1. Sign in to RecordPoint.
2. Go to the Browse page.
3. Select any record to open the Record Details page.
4. Select the Duplicates tab.
   • Any duplicates of the record binary are displayed.

Here you can see a number of duplicates of a CV, with content sources across SharePoint and Exchange:

Managing duplicates using file plans

To manage duplicates, you can create a classification and manually reassign duplicates to it.

1. Create a classification called Duplicate. You can assign a disposal date, such as one week after creation, to make destroying duplicate records easier.
2. Open a record and go to the Duplicates tab.
3. Select a duplicate, then select Reclassify.
4. Select the Duplicate classification.
5. The record is reclassified, and is searchable and reportable using the duplicate classification.

Duplicates Reports

You can use the Intelligence Signalling report to discover duplicates across your data landscape. 
See Enterprise Reporting & Analytics.

Download the Signal Intelligence Report

1. Sign in to RecordPoint.
2. Go to the Administration page by clicking the Settings icon on the top Main Toolbar.
3. Select Reporting from the Navigation Menubar.
4. Select the Enterprise tab.
5. Download the Power BI file named Signal Intelligence Report.

Read Enterprise Reporting & Analytics to learn how to connect your report to your tenant, understand Power BI, and customize your report.

The report contains two tabs, Duplicates and Duplicates details. Use these reports to identify duplicates across your connected sources.

1. The total items under management in RecordPoint
2. The total number of duplicates
3. A gauge that shows the number of duplicates in relation to the total number of items. Use this gauge to set Power BI automation or alerts.
4. Filter by the size of the file
5. Filter by the content source
6. The grid groups duplicates so they can be explored:
   • Select a row to investigate it.
   • Right click and drill into the detail report.

Duplicate Images, Audio, or Video Files

The Duplicate Report does not support detection of images, audio, or video files.