zannotate

package module
v0.0.0-...-0c78c5a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 29, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

README

ZAnnotate

ZAnnotate is a Go utility that facilitates annotating large datasets with network metadata. Right now this includes:

  • Maxmind GeoIP2
  • AS/Routing Data (based on an MRT routing table)
  • IPInfo.io ASN and Geolocation data

For example, you can add Maxmind geolocation data to a list of IPs:

cat ips.csv | zannotate --geoip2 --geoip2-database=geoip2.mmdb

Installation

ZAnnotate can be installed using make install

make install

or if you don't have make installed, you can use the following command:

cd cmd/zannotate && go install

Either way, this will install the zannotate binary in your $GOPATH/bin directory.

Check that it was installed correctly with:

zannotate --help

Acquiring Datasets

[!NOTE] URLs and instructions may change over time. These are up-to-date as of September 2025.

Below are instructions for getting datasets from the below providers.

GeoLite2-ASN
  1. Sign-up form for MaxMind GeoLite Access
  2. Login to your account
  3. Go to the "GeoIP / GeoLite" > "Download files" section where you should see a list of available databases
  4. Download the .mmdb files for GeoLite ASN
  5. Unzip the downloaded file and test with:
echo "1.1.1.1" | zannotate --geoasn --geoasn-database=/path-to-downloaded-file/GeoLite2-ASN_20250923/GeoLite2-ASN.mmdb
{"ip":"1.1.1.1","geoasn":{"asn":13335,"org":"CLOUDFLARENET"}}
BGP Routing Tables
  1. Go to https://archive.routeviews.org/route-views2/bgpdata/
  2. Select a month directory (e.g. 2025.09)
  3. Select the RIBS/ directory
  4. Download a BZiped MRT file (e.g. rib.20250923.1200.bz2)
  5. Unzip the file with:
bzip2 -d ./path-to-downloaded-file/rib.20250923.1200.bz2
  1. Test with:
echo "1.1.1.1" | ./zannotate --routing --routing-mrt-file=/tmp/rib.20250923.1200
{"ip":"1.1.1.1","routing":{"prefix":"1.1.1.0/24","asn":13335,"path":[3561,209,3356,13335]}}

Input/Output

Output

By default, ZAnnotate reads new-line delimited IP addresses from standard input and outputs a JSON object per line to standard output like:

echo "1.1.1.1" | zannotate --rdns --geoasn --geoasn-database=/path-to-geo-asn.mmdb
{"ip":"1.1.1.1","geoasn":{"asn":13335,"org":"CLOUDFLARENET"},"rdns":{"domain_names":["one.one.one.one"]}}

If an IP address cannot be annotated, either because of an error or lack of data, there will be an empty field for that annotation. For example, if an IP address is private and therefore has no RDNS or ASN data, the output will look like:

echo "127.0.0.1" | zannotate --rdns --geoasn --geoasn-database=/path-to-geo-asn.mmdb
{"geoasn":{},"rdns":{},"ip":"127.0.0.1"}

Input

You may wish to annotate data that is already in JSON format. You'll then need to use the --input-file-type=json flag. This will insert a zannotate field into the existing JSON object. For example:

echo '{"ip": "1.1.1.1"}'  | ./zannotate --rdns --geoasn --geoasn-database=/path-to-geo-asn.mmdb --input-file-type=json    
{"ip":"1.1.1.1","zannotate":{"geoasn":{"asn":13335,"org":"CLOUDFLARENET"},"rdns":{"domain_names":["one.one.one.one"]}}}

Acquiring Datasets

[!NOTE] URLs and instructions may change over time. These are up-to-date as of September 2025. Below are instructions for getting datasets from the below providers.

IPInfo.io

IPInfo.io provides a free dataset that includes ASN and geolocation data, scoped to the country level. Paid tiers provide more granular geo-location data.

  1. Sign up for a free account at IPInfo.io.
  2. Navigate to the Data Download page
  3. Download the mmdb file IPInfo Download Page
  4. Example CLI usage
echo "1.1.1.1" | ./zannotate --ipinfo --ipinfo-database=./path-to-ipinfo-db.mmdb
{"ip":"1.1.1.1","ipinfo":{"country":"Australia","country_code":"AU","continent":"Oceania","continent_code":"OC","asn":"AS13335","as_name":"Cloudflare, Inc.","as_domain":"cloudflare.com"}}

Documentation

Index

Constants

This section is empty.

Variables

View Source
var Annotators []AnnotatorFactory

Functions

func AnnotateInputDecode

func AnnotateInputDecode(conf *GlobalConf, inChan <-chan string,
	outChan chan<- inProcessIP, wg *sync.WaitGroup, i int)

multiple workers that decode raw lines from AnnotateRead from JSON/CSV into native golang objects

func AnnotateOutputEncode

func AnnotateOutputEncode(conf *GlobalConf, inChan <-chan inProcessIP,
	outChan chan<- string, wg *sync.WaitGroup, i int)

func AnnotateRead

func AnnotateRead(conf *GlobalConf, path string, in chan<- string)

single worker that reads from file and queues raw lines

func AnnotateWorker

func AnnotateWorker(a Annotator, inChan <-chan inProcessIP,
	outChan chan<- inProcessIP, fieldName string, wg *sync.WaitGroup, i int)

func AnnotateWrite

func AnnotateWrite(path string, out <-chan string, wg *sync.WaitGroup)

func DoAnnotation

func DoAnnotation(conf *GlobalConf)

func RegisterAnnotator

func RegisterAnnotator(af AnnotatorFactory)

Types

type Annotator

type Annotator interface {
	Initialize() error
	Annotate(ip net.IP) interface{}
	GetFieldName() string
	Close() error
}

type AnnotatorFactory

type AnnotatorFactory interface {
	Initialize(c *GlobalConf) error
	AddFlags(flags *flag.FlagSet)
	GetWorkers() int
	IsEnabled() bool
	MakeAnnotator(i int) Annotator
	Close() error
}

type BasePluginConf

type BasePluginConf struct {
	Threads int
	Enabled bool
}

type GeoIP2Annotator

type GeoIP2Annotator struct {
	Factory *GeoIP2AnnotatorFactory
	Reader  *geoip2.Reader
	Id      int
}

func (*GeoIP2Annotator) Annotate

func (a *GeoIP2Annotator) Annotate(ip net.IP) interface{}

func (*GeoIP2Annotator) Close

func (a *GeoIP2Annotator) Close() error

func (*GeoIP2Annotator) GeoIP2FillStruct

func (a *GeoIP2Annotator) GeoIP2FillStruct(in *geoip2.City) *GeoIP2Output

func (*GeoIP2Annotator) GetFieldName

func (a *GeoIP2Annotator) GetFieldName() string

func (*GeoIP2Annotator) Initialize

func (a *GeoIP2Annotator) Initialize() error

type GeoIP2AnnotatorFactory

type GeoIP2AnnotatorFactory struct {
	BasePluginConf
	Path       string
	Mode       string
	Language   string
	RawInclude string

	Conf *GlobalConf
	// what data to include
	IncludeCity               bool
	IncludeCountry            bool
	IncludeContinent          bool
	IncludePostal             bool
	IncludeLatLong            bool
	IncludeTraits             bool
	IncludeSubdivisions       bool
	IncludeRepresentedCountry bool
	IncludeRegisteredCountry  bool
}

func (*GeoIP2AnnotatorFactory) AddFlags

func (a *GeoIP2AnnotatorFactory) AddFlags(flags *flag.FlagSet)

GeoIP2 Annotator Factory (Global)

func (*GeoIP2AnnotatorFactory) Close

func (a *GeoIP2AnnotatorFactory) Close() error

func (*GeoIP2AnnotatorFactory) GetWorkers

func (a *GeoIP2AnnotatorFactory) GetWorkers() int

func (*GeoIP2AnnotatorFactory) Initialize

func (a *GeoIP2AnnotatorFactory) Initialize(conf *GlobalConf) error

func (*GeoIP2AnnotatorFactory) IsEnabled

func (a *GeoIP2AnnotatorFactory) IsEnabled() bool

func (*GeoIP2AnnotatorFactory) MakeAnnotator

func (a *GeoIP2AnnotatorFactory) MakeAnnotator(i int) Annotator

type GeoIP2City

type GeoIP2City struct {
	Name      string `json:"name,omitempty"`
	GeoNameId uint   `json:"id,omitempty"`
}

type GeoIP2Country

type GeoIP2Country struct {
	Name      string `json:"name,omitempty"`
	Code      string `json:"code,omitempty"`
	GeoNameId uint   `json:"id,omitempty"`
}

type GeoIP2LatLong

type GeoIP2LatLong struct {
	AccuracyRadius uint16  `json:"accuracy_radius,omitempty"`
	Latitude       float64 `json:"latitude,omitempty"`
	Longitude      float64 `json:"longitude,omitempty"`
	MetroCode      uint    `json:"metro_code,omitempty"`
	TimeZone       string  `json:"time_zone,omitempty"`
}

type GeoIP2Output

type GeoIP2Output struct {
	City               *GeoIP2City    `json:"city,omitempty"`
	Country            *GeoIP2Country `json:"country,omitempty"`
	Continent          *GeoIP2Country `json:"continent,omitempty"`
	Postal             *GeoIP2Postal  `json:"postal,omitempty"`
	LatLong            *GeoIP2LatLong `json:"latlong,omitempty"`
	RepresentedCountry *GeoIP2Country `json:"represented_country,omitempty"`
	RegisteredCountry  *GeoIP2Country `json:"registered_country,omitempty"`
	Traits             *GeoIP2Traits  `json:"metadata,omitempty"`
}

type GeoIP2Postal

type GeoIP2Postal struct {
	Code string `json:"code,omitempty"`
}

type GeoIP2Traits

type GeoIP2Traits struct {
	IsAnonymousProxy    bool `json:"is_anonymous_proxy,omitempty"`
	IsSatelliteProvider bool `json:"is_satellite_provider,omitempty"`
}

type GeoIPASNAnnotator

type GeoIPASNAnnotator struct {
	Factory *GeoIPASNAnnotatorFactory
	Reader  *geoip2.Reader
	Id      int
}

func (*GeoIPASNAnnotator) Annotate

func (anno *GeoIPASNAnnotator) Annotate(ip net.IP) interface{}

func (*GeoIPASNAnnotator) Close

func (anno *GeoIPASNAnnotator) Close() error

func (*GeoIPASNAnnotator) GetFieldName

func (anno *GeoIPASNAnnotator) GetFieldName() string

func (*GeoIPASNAnnotator) Initialize

func (anno *GeoIPASNAnnotator) Initialize() error

type GeoIPASNAnnotatorFactory

type GeoIPASNAnnotatorFactory struct {
	BasePluginConf
	Path string
	Mode string
}

func (*GeoIPASNAnnotatorFactory) AddFlags

func (fact *GeoIPASNAnnotatorFactory) AddFlags(flags *flag.FlagSet)

func (*GeoIPASNAnnotatorFactory) Close

func (fact *GeoIPASNAnnotatorFactory) Close() error

func (*GeoIPASNAnnotatorFactory) GetWorkers

func (fact *GeoIPASNAnnotatorFactory) GetWorkers() int

func (*GeoIPASNAnnotatorFactory) Initialize

func (fact *GeoIPASNAnnotatorFactory) Initialize(conf *GlobalConf) error

func (*GeoIPASNAnnotatorFactory) IsEnabled

func (fact *GeoIPASNAnnotatorFactory) IsEnabled() bool

func (*GeoIPASNAnnotatorFactory) MakeAnnotator

func (fact *GeoIPASNAnnotatorFactory) MakeAnnotator(i int) Annotator

type GeoIPASNOutput

type GeoIPASNOutput struct {
	ASN    uint   `json:"asn,omitempty"`
	ASNOrg string `json:"org,omitempty"`
}

type GlobalConf

type GlobalConf struct {
	InputFilePath           string
	InputFileType           string
	OutputFilePath          string
	MetadataFilePath        string
	LogFilePath             string
	Verbosity               int
	Threads                 int
	JSONIPFieldName         string
	JSONAnnotationFieldName string
	InputDecodeThreads      int
	OutputEncodeThreads     int
}

type IPInfoAnnotator

type IPInfoAnnotator struct {
	Factory *IPInfoAnnotatorFactory
	Id      int
}

func (*IPInfoAnnotator) Annotate

func (a *IPInfoAnnotator) Annotate(inputIP net.IP) interface{}

func (*IPInfoAnnotator) Close

func (a *IPInfoAnnotator) Close() error

func (*IPInfoAnnotator) GetFieldName

func (a *IPInfoAnnotator) GetFieldName() string

func (*IPInfoAnnotator) Initialize

func (a *IPInfoAnnotator) Initialize() error

type IPInfoAnnotatorFactory

type IPInfoAnnotatorFactory struct {
	BasePluginConf
	DatabaseFilePath string
	// contains filtered or unexported fields
}

func (*IPInfoAnnotatorFactory) AddFlags

func (a *IPInfoAnnotatorFactory) AddFlags(flags *flag.FlagSet)

func (*IPInfoAnnotatorFactory) Close

func (a *IPInfoAnnotatorFactory) Close() error

func (*IPInfoAnnotatorFactory) GetWorkers

func (a *IPInfoAnnotatorFactory) GetWorkers() int

func (*IPInfoAnnotatorFactory) Initialize

func (a *IPInfoAnnotatorFactory) Initialize(conf *GlobalConf) (err error)

func (*IPInfoAnnotatorFactory) IsEnabled

func (a *IPInfoAnnotatorFactory) IsEnabled() bool

func (*IPInfoAnnotatorFactory) MakeAnnotator

func (a *IPInfoAnnotatorFactory) MakeAnnotator(i int) Annotator

type IPInfoMMDBOutput

type IPInfoMMDBOutput struct {
	City              string `maxminddb:"city"`
	Region            string `maxminddb:"region"`
	RegionCode        string `maxminddb:"region_code"`
	Country           string `maxminddb:"country"`
	CountryCode       string `maxminddb:"country_code"`
	Continent         string `maxminddb:"continent"`
	ContinentCode     string `maxminddb:"continent_code"`
	Latitude          string `maxminddb:"latitude"`
	Longitude         string `maxminddb:"longitude"`
	Timezone          string `maxminddb:"timezone"`
	PostalCode        string `maxminddb:"postal_code"`
	DMACode           string `maxminddb:"dma_code"`
	GeonameID         string `maxminddb:"geoname_id"`  // GeoNames database identifier (if available).
	Radius            string `maxminddb:"radius"`      // Accuracy radius in kilometers (if available).
	GeoChanged        string `maxminddb:"geo_changed"` // Timestamp or flag indicating when the geolocation last changed (if available).
	ASN               string `maxminddb:"asn"`
	ASName            string `maxminddb:"as_name"`
	ASDomain          string `maxminddb:"as_domain"`
	ASType            string `maxminddb:"as_type"`
	ASChanged         string `maxminddb:"as_changed"`
	CarrierName       string `maxminddb:"carrier_name"` // Name of the mobile carrier (if available).
	MobileCountryCode string `maxminddb:"mcc"`
	MobileNetworkCode string `maxminddb:"mnc"`
	PrivacyName       string `maxminddb:"privacy_name"` // Specific name of the privacy or anonymization service detected (e.g., “NordVPN”).
	IsProxy           string `maxminddb:"is_proxy"`
	IsRelay           string `maxminddb:"is_relay"`     // Boolean flag indicating use of a general relay service
	IsTOR             string `maxminddb:"is_tor"`       // Whether the IP is a known TOR exit node.
	IsVPN             string `maxminddb:"is_vpn"`       // Flag indicating use of a VPN Service
	IsAnonymous       string `maxminddb:"is_anonymous"` // True if the IP is associated with VPN, proxy, Tor, or a relay service.
	IsAnycast         string `maxminddb:"is_anycast"`   // Whether the IP is using anycast routing.
	IsHosting         string `maxminddb:"is_hosting"`   // True if the IP address is an internet service hosting IP address
	IsMobile          string `maxminddb:"is_mobile"`    // True if the IP address is associated with a mobile network or carrier.
	IsSatellite       string `maxminddb:"is_satellite"` // True if the IP address is associated with a satellite connection
}

IPInfoMMDBOutput includes both the Plus/Core/Lite IPInfo fields and their maxminddb tags. We'll convert this into a IPInfoModuleOutput for JSON output (converting string fields to appropriate types).

func (*IPInfoMMDBOutput) ToModuleOutput

func (in *IPInfoMMDBOutput) ToModuleOutput() *IPInfoModuleOutput

type IPInfoModuleOutput

type IPInfoModuleOutput struct {
	City              string  `json:"city,omitempty"`
	Region            string  `json:"region,omitempty"`
	RegionCode        string  `json:"region_code,omitempty"`
	Country           string  `json:"country,omitempty"`
	CountryCode       string  `json:"country_code,omitempty"`
	Continent         string  `json:"continent,omitempty"`
	ContinentCode     string  `json:"continent_code,omitempty"`
	Latitude          float64 `json:"latitude,omitempty"`
	Longitude         float64 `json:"longitude,omitempty"`
	Timezone          string  `json:"timezone,omitempty"`
	PostalCode        string  `json:"postal_code,omitempty"`
	DMACode           string  `json:"dma_code,omitempty"`    // Nielsen Designated Market Area code (if available).
	GeonameID         uint64  `json:"geoname_id,omitempty"`  // GeoNames database identifier (if available).
	Radius            uint64  `json:"radius,omitempty"`      // Accuracy radius in kilometers (if available).
	GeoChanged        string  `json:"geo_changed,omitempty"` // Timestamp or flag indicating when the geolocation last changed (if available).
	ASN               string  `json:"asn,omitempty"`
	ASName            string  `json:"as_name,omitempty"`
	ASDomain          string  `json:"as_domain,omitempty"`
	ASType            string  `json:"as_type,omitempty"`
	ASChanged         string  `json:"as_changed,omitempty"`
	CarrierName       string  `json:"carrier_name,omitempty"` // Name of the mobile carrier (if available).
	MobileCountryCode string  `json:"mobile_country_code,omitempty"`
	MobileNetworkCode string  `json:"mobile_network_code,omitempty"`
	PrivacyName       string  `json:"privacy_name,omitempty"` // Specific name of the privacy or anonymization service detected (e.g., “NordVPN”).
	IsProxy           *bool   `json:"is_proxy,omitempty"`
	IsRelay           *bool   `json:"is_relay,omitempty"`     // Boolean flag indicating use of a general relay service
	IsTOR             *bool   `json:"is_tor,omitempty"`       // Whether the IP is a known TOR exit node.
	IsVPN             *bool   `json:"is_vpn,omitempty"`       // Flag indicating use of a VPN Service
	IsAnonymous       *bool   `json:"is_anonymous,omitempty"` // True if the IP is associated with VPN, proxy, Tor, or a relay service.
	IsAnycast         *bool   `json:"is_anycast,omitempty"`   // Whether the IP is using anycast routing.
	IsHosting         *bool   `json:"is_hosting,omitempty"`   // True if the IP address is an internet service hosting IP address
	IsMobile          *bool   `json:"is_mobile,omitempty"`    // True if the IP address is associated with a mobile network or carrier.
	IsSatellite       *bool   `json:"is_satellite,omitempty"` // True if the IP address is associated with a satellite connection
}

IPInfoModuleOutput is the final output struct with appropriate types for JSON output

type RDNSAnnotator

type RDNSAnnotator struct {
	Factory *RDNSAnnotatorFactory
	Id      int
	// contains filtered or unexported fields
}

func (*RDNSAnnotator) Annotate

func (a *RDNSAnnotator) Annotate(ip net.IP) interface{}

Annotate performs a reverse DNS lookup for the given IP address and returns the results. If an error occurs or a lookup fails, it returns nil

func (*RDNSAnnotator) Close

func (a *RDNSAnnotator) Close() error

func (*RDNSAnnotator) GetFieldName

func (a *RDNSAnnotator) GetFieldName() string

func (*RDNSAnnotator) Initialize

func (a *RDNSAnnotator) Initialize() (err error)

type RDNSAnnotatorFactory

type RDNSAnnotatorFactory struct {
	BasePluginConf
	RawResolvers string
	// contains filtered or unexported fields
}

func (*RDNSAnnotatorFactory) AddFlags

func (a *RDNSAnnotatorFactory) AddFlags(flags *flag.FlagSet)

func (*RDNSAnnotatorFactory) Close

func (a *RDNSAnnotatorFactory) Close() error

func (*RDNSAnnotatorFactory) GetWorkers

func (a *RDNSAnnotatorFactory) GetWorkers() int

func (*RDNSAnnotatorFactory) Initialize

func (a *RDNSAnnotatorFactory) Initialize(_ *GlobalConf) error

func (*RDNSAnnotatorFactory) IsEnabled

func (a *RDNSAnnotatorFactory) IsEnabled() bool

func (*RDNSAnnotatorFactory) MakeAnnotator

func (a *RDNSAnnotatorFactory) MakeAnnotator(i int) Annotator

type RDNSOutput

type RDNSOutput struct {
	DomainNames []string `json:"domain_names,omitempty"`
}

type RoutingAnnotator

type RoutingAnnotator struct {
	Factory *RoutingAnnotatorFactory

	Id int
	// contains filtered or unexported fields
}

func (*RoutingAnnotator) Annotate

func (a *RoutingAnnotator) Annotate(ip net.IP) interface{}

func (*RoutingAnnotator) Close

func (a *RoutingAnnotator) Close() error

func (*RoutingAnnotator) GetFieldName

func (a *RoutingAnnotator) GetFieldName() string

func (*RoutingAnnotator) Initialize

func (a *RoutingAnnotator) Initialize() error

type RoutingAnnotatorFactory

type RoutingAnnotatorFactory struct {
	BasePluginConf
	RoutingTablePath string
	ASNamesPath      string
	ASDataPath       string
	// contains filtered or unexported fields
}

func (*RoutingAnnotatorFactory) AddFlags

func (a *RoutingAnnotatorFactory) AddFlags(flags *flag.FlagSet)

Routing Annotator Factory (Global)

func (*RoutingAnnotatorFactory) Close

func (a *RoutingAnnotatorFactory) Close() error

func (*RoutingAnnotatorFactory) GetWorkers

func (a *RoutingAnnotatorFactory) GetWorkers() int

func (*RoutingAnnotatorFactory) Initialize

func (a *RoutingAnnotatorFactory) Initialize(conf *GlobalConf) error

func (*RoutingAnnotatorFactory) IsEnabled

func (a *RoutingAnnotatorFactory) IsEnabled() bool

func (*RoutingAnnotatorFactory) MakeAnnotator

func (a *RoutingAnnotatorFactory) MakeAnnotator(i int) Annotator

Directories

Path Synopsis
cmd
mrt2json command
zannotate command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL