Skip to content
fingerprint

go · simhash + phash · near-dup

Find near-duplicate text and images.

Two content-fingerprinting primitives for near-duplicate detection — a shingled Charikar SimHash for text and a DCT perceptual hash for images. Both return a uint64; pairwise Hamming distance measures closeness.

Install Go Reference

MIT licensed · one small dependency for images · Go 1.25+

fingerprint.go 
a := fingerprint.Compute(docA) // 64-bit text fingerprint
b := fingerprint.Compute(docB)

if fingerprint.Similarity(a, b) >= 0.85 {
    // near-duplicate: minor edits, template fills, revisions
}

what you get

Two hashes, one number, one metric.

Fingerprint a document or an image down to a single uint64, then measure closeness with Hamming distance — the same primitive works for both.

Text SimHash

Compute / Distance / Similarity — a 64-bit locality-sensitive hash for finding near-duplicate documents.

Shingled (3-word)

Keys on phrasing, so unrelated prose drops to ~0.55 instead of the ~0.90 that single-token SimHash spuriously produces.

Image pHash

PHash / PHashFromImage (plus hex helpers): a 64-bit DCT hash for visually-similar images regardless of scale or minor edits.

One number, one metric

Everything is a uint64; Hamming Distance (and Similarity = 1 − d/64) measures closeness across both text and images.

Robust

pHash downscales to 32×32 greyscale, runs a 2-D DCT, and keeps the low-frequency signs — stable across resizes and re-encodes.

Small

The SimHash half is pure stdlib; the pHash half uses golang.org/x/image for high-quality downscaling. Go 1.25+.

usage

Text or images — same shape.

Fingerprint each side to a uint64, then threshold on Similarity or Distance. Image hashes round-trip through hex for storage.

// Text: near-duplicate documents
a := fingerprint.Compute(docA)
b := fingerprint.Compute(docB)

if fingerprint.Similarity(a, b) >= 0.85 {
    // minor edits, template fills, revisions
}
// Images: visually-similar pictures
h, err := fingerprint.PHash(file)
if err == nil && fingerprint.Distance(h, want) <= 10 {
    // same image, resized or re-encoded
}

hex := fingerprint.PHashHex(h)       // store
h2, _ := fingerprint.PHashFromHex(hex) // load
  • Compute
  • Distance
  • Similarity
  • PHash
  • PHashFromImage
  • PHashHex
  • PHashFromHex

install

Add it to your module.

One import for the whole package. Requires Go 1.25+.

go get go get github.com/richardwooding/fingerprint
import import "github.com/richardwooding/fingerprint"

Extracted from file-search-on (it powers find_near_duplicates / image_similar_to). Full API on the Go Reference.