Skip to content
go-hashset

go · md5 / sha-1 / sha-256 · nsrl-scale

Hash allow/deny lookup, at NSRL scale.

Filter known-good files against an NSRL reference set, or flag known-bad hashes from a threat-intel feed — one Set interface over a fast in-memory store or a bbolt-backed store for tens of millions of entries.

Install Go Reference

MIT licensed · single dependency (bbolt) · MD5 · SHA-1 · SHA-256

hashset.go 
set, _ := hashset.LoadTextFile("known-bad.txt")
defer set.Close()

if set.Contains("sha256", digest) {
    // flag it
}
fmt.Println(set.Counts()) // map[md5:… sha1:… sha256:…]

what you get

One Set interface, allow or deny.

The same store powers NSRL known-good filtering and known-bad flagging — from a one-off text list to a fifty-million-entry on-disk database.

Allow and deny in one API

The same Set backs both allowlist (NSRL known-good) and denylist (known-bad) workflows — the difference is your data, not your code.

MD5, SHA-1, SHA-256

The algorithm is auto-detected by hash length — 32, 40, or 64 hex characters — so mixed lists just work.

Two backends

A fast in-memory store (~1M entries, O(1)) or a bbolt-backed on-disk store (~50M NSRL-scale, O(log N)) — same lookups either way.

Ingests real lists

Reads plain text hash lists — with # comments and mixed algorithms — and NSRL NSRLFile.txt quoted-CSV.

Auto-detecting Open

Open(path) picks the backend straight from the file — a bbolt database or a text list — so callers don't have to choose.

Lean

A single dependency (go.etcd.io/bbolt), with progress callbacks during builds so long ingests stay observable.

usage

In-memory list, or NSRL on disk.

Load a text list straight into memory for quick lookups, or build a bbolt database once and reopen it read-only for tens of millions of entries.

// In-memory — load a text list and query
set, err := hashset.LoadTextFile("known-bad.txt")
if err != nil { /* ... */ }
defer set.Close()

if set.Contains("sha256", digest) {
    // flag it
}
// NSRL scale — build once, then open read-only
in, _ := os.Open("NSRLFile.txt")
defer in.Close()
_ = hashset.Build(in, "nsrl.hashset", hashset.BuildOpts{
    Format:   "auto",
    Progress: func(n int64) { log.Printf("%d hashes", n) },
})

set, _ := hashset.OpenBolt("nsrl.hashset")
defer set.Close()
  • LoadTextFile
  • LoadText
  • Build
  • OpenBolt
  • Open
  • Contains
  • Counts

install

Add it to your module.

One import and a single transitive dependency (go.etcd.io/bbolt).

go get go get github.com/richardwooding/go-hashset
import import hashset "github.com/richardwooding/go-hashset"

Extracted from file-search-on, where it backs the known-good / known-bad search predicates. Full API on the Go Reference.