sluice

A Go library for pulling free threat intelligence feeds — and knowing what you are allowed to do with what comes out.

reg := sluice.NewRegistry(sluice.Credentials{AbuseCh: key},
    sluice.WithUserAgent("acme-soc/1.0 (+https://acme.example; soc@acme.example)"))

feed, _ := reg.Feed("threatfox")
obs, _ := feed.Fetch(ctx, http.DefaultClient)

// Every observation carries the terms it arrived under.
cleared, blocked := sluice.NewGate(sluice.UseRedistribute).Filter(obs)

No dependencies. go.mod has no require block and CI fails the build if one appears.

go get github.com/yourorg/sluice

Requires Go 1.23 or later.

---

Why this exists

Two problems, both small enough that everyone works around them and nobody fixes them.

Feed adapters rot quietly. In 2025 abuse.ch put every community API behind a mandatory Auth-Key header. Tooling that had worked for years started returning HTTP 200 with an error in the body, parsers yielded zero results, and the failure looked exactly like a quiet week. There is no canonical Go module that keeps these adapters correct, so everyone maintains a private copy that breaks on a different Tuesday.

Nobody tracks what they're allowed to republish. Feed terms vary a lot. OpenPhish prohibits redistribution. abuse.ch is free under fair use but says commercial use may need a paid subscription. PhishTank permits commercial use but wants attribution. Once you aggregate them, an indicator corroborated by four feeds is governed by the intersection of four sets of terms — and the usual answer to that is a spreadsheet, or nothing.

MISP's distribution levels answer a different question: who in your sharing community can see this. They don't answer whether your upstream provider lets you publish it at all.

sluice does both, and nothing else. It is not a platform. It has no database, no scheduler, no UI. Those are your decisions.

---

The licence model

Every Observation carries the Licence it arrived under, so a record stays self-describing after it leaves the library.

type Licence struct {
    Class        Class      // coarse label for humans scanning a list
    Commercial   Permission // Allowed | Ask | Prohibited | PermissionUnknown
    Redistribute Permission
    Attribution  string     // required credit line, empty if none
    Note         string
    TermsURL     string
}

Ask is not yes

Ask exists because most feed terms are not binary. abuse.ch offers its community API free under fair use and states that commercial use may require a subscription. That is neither permission nor refusal, and flattening it either way is how people end up in breach.

The gate treats Ask as refused by default. PermitAsk() relaxes that, and you should only call it once you have actually asked and been told yes.

gate := sluice.NewGate(sluice.UseCommercial).PermitAsk()

PermitAsk never overrides an outright prohibition.

Unknown is the strictest state

A licence nobody filled in has PermissionUnknown, which the gate refuses. If nobody read the terms, the answer is no. This is what makes it safe to add a feed adapter without auditing every call site: an undeclared licence fails closed, and a test asserts every registered feed declares one.

Corroboration intersects terms

This is the case aggregation creates and nothing else models. Once four feeds agree on an address, the terms governing that indicator are the intersection of all four, not the friendliest one.

groups := sluice.GroupByKey(observations)       // dedup across feeds
cleared, blocked := gate.FilterMerged(groups)   // strictest terms win per indicator

Merge accumulates attribution rather than picking one, because publishing a derived list that credits only some of your sources is its own problem.

Both Filter and FilterMerged return the rejects with reasons. Silently dropping records is how people lose track of why an export shrank.

---

Normalisation

The load-bearing function is Normalise. If two feeds report the same infrastructure in different shapes and those don't collapse to one key, corroboration counts under-report and every confidence score built on top is wrong in a way nobody notices.

It handles the cases that actually occur in feed data:

Input	Normalised
192.0.2.1:443	192.0.2.1 — C2 feeds append ports
192[.]0[.]2[.]1	192.0.2.1 — feeds echo defanged reports
hxxp://evil[.]invalid/a	http://evil.invalid/a
www.evil.invalid	evil.invalid
http://evil.invalid:80/a	http://evil.invalid/a
192.0.2.55/24	192.0.2.0/24
2001:DB8::1	2001:db8::1

It deliberately does not collapse evil.invalid and mail.evil.invalid, or http:// and https:// variants — over-collapsing manufactures false corroboration, which is worse than missing some. TestNormaliseKeepsDistinct guards that direction.

FamilyLabel collapses vendor naming so RedLine, redline_stealer and win.redline aggregate. It is heuristic and will occasionally be wrong; the original label stays in Raw.

---

Feeds

Slug	What it is	Key	Commercial	Redistribute
urlhaus	Malware distribution URLs	abuse.ch	ask	no
threatfox	Mixed IOCs with family attribution	abuse.ch	ask	no
malwarebazaar	Sample hashes and family labels	abuse.ch	ask	no
feodotracker	Active botnet C2 addresses	none	ask	no
openphish	Community phishing URLs	none	no	no
phishtank	Community-verified phishing	free	yes	no
blocklist_de	Hosts reported by fail2ban sensors	none	yes	yes
cins_army	Poor-reputation addresses	none	yes	no
spamhaus_drop	Hijacked netblocks	none	ask	no
tor_exits	Tor exit relays (context, not threat)	none	yes	yes
anyrun	Sandbox STIX bundle	paid	no	no

One free key from auth.abuse.ch unlocks URLhaus, ThreatFox and MalwareBazaar together. Feeds whose credential is absent are not registered at all, so ranging over reg.All() never yields something guaranteed to fail.

Three things worth knowing before you plan around them

abuse.ch requires authentication. Every community API. There is no anonymous fallback worth implementing.

VirusTotal has no free bulk feed. The free tier is lookup-only, roughly 4 requests/minute and 500/day, non-commercial, results not redistributable. It belongs in your enrichment path, not your ingest path, which is why there is no VT adapter here. Bulk address reputation comes from Feodo Tracker, Spamhaus DROP, CINS and blocklist.de.

ANY.RUN's live TI feed is a paid product. Only the demo bundle is open and it lags roughly six months. Useful for retro-hunting and for exercising the STIX parser; useless for live detection. The adapter registers only when you supply a key, and its licence class is restricted.

Provider TTLs are honoured

ThreatFox retires indicators after six months because cloud addresses change tenants. Resurrecting them manufactures false positives against whoever rented the address next. Observation.ExpiresAt mirrors upstream where the provider states a retention, and Expired() is yours to call before you act on anything.

Adding your own

Any STIX 2.x or TAXII collection works without new code:

feed := sluice.NewSTIXFeed(sluice.Descriptor{
    Slug: "internal", Name: "Our sandbox", Vendor: "us",
    MinInterval: time.Hour,
    Licence: sluice.Licence{
        Class: sluice.ClassPublicDomain,
        Commercial: sluice.Allowed, Redistribute: sluice.Allowed,
    },
}, "https://ti.internal.example/stix", authHeader, 90*24*time.Hour)

For anything else, implement Feed. Two methods.

---

Politeness

These are free services run on someone else's budget. The library helps you not be a problem, but it does not enforce anything.

• Set a User-Agent with a contact address. PhishTank rate limits or challenges blank and generic agents. abuse.ch would rather email you than ban your key. WithUserAgent is the single most valuable option here.
• Respect MinInterval. Each descriptor states the shortest interval that makes sense given how often the provider actually publishes. OpenPhish refreshes roughly every six hours; polling it every minute returns the same file 360 times.
• Jitter your startup. Ten feeds firing in the same second after a restart looks like abuse and reliably earns a rate limit.
• HTTPError.Retryable() reports whether backing off might help — 429 and 5xx yes, 401 and 403 no.

---

Testing

The suite runs against recorded fixtures in testdata/. No network, no credentials, no flakes.

go test ./...

Fixtures use RFC 5737 documentation addresses and .invalid domains. Shape is real; values are not. Nobody should be able to mistake a fixture for live intelligence, and no fixture should end up in someone's blocklist because a test wrote to the wrong place.

Every adapter takes WithBaseURL, which is what makes them testable against a local fixture server.

The canary

The expensive failure in feed tooling is a silent one. A provider renames a field, the adapter keeps returning HTTP 200, the parser yields nothing, and nobody notices until an incident review asks why a feed went quiet three weeks ago.

internal/canary polls the real endpoints and diffs the response shape against the fixtures:

go run ./internal/canary              # human readable
go run ./internal/canary -only=urlhaus
go run ./internal/canary -json        # machine readable

For JSON feeds it compares the set of field paths. A removed field is drift and fails the run; a new field is informational, since providers add columns routinely. For plain-text lists it measures what fraction of lines still classify as indicators, which catches a provider switching to CSV.

It also runs the actual adapter and fails if a live response yields zero observations, which is the failure mode that matters most.

.github/workflows/canary.yml runs it daily and maintains a single rolling issue labelled feed-drift, commenting on it rather than opening a new one each morning. A daily drip of duplicate issues gets muted, and a muted alarm is not an alarm. It closes the issue when everything is green again.

To enable it, add repository secrets ABUSECH_AUTH_KEY and PHISHTANK_APP_KEY, and a repository variable SLUICE_USER_AGENT. Feeds without credentials report as skipped rather than failed, so the canary is useful even on a fork with no secrets at all.

---

Examples

examples/export — collect every permitted feed, group by corroboration, score, and write JSON or CSV with the attribution block in the file:

go run ./examples/export -use=internal      > everything.json
go run ./examples/export -use=redistribute  > safe-to-publish.json

Run both and diff them. The gap is the part most tooling ships by accident.

The scoring there — sources × 28 × e^(−age/7d), capped at 100 — is an example, not a recommendation. It lives in the example rather than the library because your weighting depends on your environment, and a library that scores for you is a library you fight.

examples/console — a single HTML file that renders an export. Open it, pick a JSON file. No build step, no server.

---

What this is not

• Not a TIP. No storage, no scheduling, no correlation engine, no UI.
• Not legal advice. The licence metadata reflects a reading of published terms at a point in time. Terms change and this repo will lag them. TermsURL is on every licence for a reason — if the answer matters commercially, read it, and tell us when we're wrong.
• Not a VirusTotal client. See above.
• Not an enrichment layer. Passive DNS, WHOIS, geolocation and reputation lookups all belong on top.

If you want the whole platform, MISP and OpenCTI are mature, and both ingest every feed here. Use sluice when you want the adapters and the licence model without inheriting either.

---

Contributing

New feed adapters are the most useful contribution, and CONTRIBUTING.md walks through adding one: descriptor, licence, fixture, table test, canary probe. Roughly an hour, most of it reading the provider's terms.

Corrections to licence metadata are the second most useful. If you have written confirmation from a provider that changes a Permission, that is worth a PR on its own.

Licence

MIT. See LICENSE.

This covers the code. It says nothing about the data the code fetches, which is governed by each provider's own terms — which is the entire point of the Licence type.