Skip to content
fingerprint

go · simhash + phash · near-dup

Find near-duplicate text, images and audio.

Content-fingerprinting primitives for near-duplicate detection — a shingled Charikar SimHash for text, a DCT perceptual hash for images, and the two inputs ISO 24138 needs to mint an ISCC: normalised pixels, and a pure-Go Chromaprint.

Install Go Reference

MIT licensed · no cgo · two small dependencies, one per medium · Go 1.25+

fingerprint.go 
a := fingerprint.Compute(docA) // 64-bit text fingerprint
b := fingerprint.Compute(docB)

if fingerprint.Similarity(a, b) >= 0.85 {
    // near-duplicate: minor edits, template fills, revisions
}

what you get

Two hashes, and the two inputs an ISCC needs.

Fingerprint a document or an image down to a single uint64 and measure closeness with Hamming distance — or produce the normalised pixels and the Chromaprint vector that ISO 24138's own code generators take.

Text SimHash

Compute / Distance / Similarity — a 64-bit locality-sensitive hash for finding near-duplicate documents.

Shingled (3-word)

Keys on phrasing, so unrelated prose drops to ~0.55 instead of the ~0.90 that single-token SimHash spuriously produces.

Image pHash

PHash / PHashFromImage (plus hex helpers): a 64-bit DCT hash for visually-similar images regardless of scale or minor edits.

One number, one metric

Everything is a uint64; Hamming Distance (and Similarity = 1 − d/64) measures closeness across both text and images.

Robust

pHash downscales to 32×32 greyscale, runs a 2-D DCT, and keeps the low-frequency signs — stable across resizes and re-encodes.

Audio Chromaprint

ChromaprintFromWAV and ChromaprintFromMP3 — the acoustic fingerprint an ISCC Audio-Code is computed from, matching fpcalc bit for bit. Every other Go option needs cgo or the fpcalc binary.

Small

Every bit of the Chromaprint arithmetic is hand-written stdlib — FFT, resampler, classifiers. Only decoding pulls anything in: x/image for images, go-mp3 for MP3. Go 1.25+.

usage

Text or images — same shape.

Fingerprint each side to a uint64, then threshold on Similarity or Distance. Image hashes round-trip through hex for storage.

// Text: near-duplicate documents
a := fingerprint.Compute(docA)
b := fingerprint.Compute(docB)

if fingerprint.Similarity(a, b) >= 0.85 {
    // minor edits, template fills, revisions
}
// Images: visually-similar pictures
h, err := fingerprint.PHash(file)
if err == nil && fingerprint.Distance(h, want) <= 10 {
    // same image, resized or re-encoded
}

hex := fingerprint.PHashHex(h)       // store
h2, _ := fingerprint.PHashFromHex(hex) // load
  • Compute
  • Distance
  • Similarity
  • PHash
  • PHashFromImage
  • PHashHex
  • PHashFromHex

install

Add it to your module.

One import for the whole package. Requires Go 1.25+.

go get go get github.com/richardwooding/fingerprint
import import "github.com/richardwooding/fingerprint"

Extracted from file-search-on (it powers find_near_duplicates / image_similar_to). Full API on the Go Reference.