SpiderFoot is a reconnaissance tool that
automatically queries over 100 public data sources (OSINT) to gather
intelligence on IP addresses, domain names, e-mail addresses, names and
more. You simply specify the target you want to investigate, pick which
modules to enable and then SpiderFoot will collect data to build up an
understanding of all the entities and how they relate to each other.
What is OSINT?
OSINT (Open Source Intelligence) is data
available in the public domain which might reveal interesting
information about your target. This includes DNS, Whois, Web pages,
passive DNS, spam blacklists, file meta data, threat intelligence lists
as well as services like SHODAN, HaveIBeenPwned? and more. See the full list of data sources SpiderFoot utilises.
What can I do with SpiderFoot?
The data returned from a SpiderFoot scan
will reveal a lot of information about your target, providing insight
into possible data leaks, vulnerabilities or other sensitive information
that can be leveraged during a penetration test, red team exercise or
for threat intelligence. Try it out against your own network to see what
you might have exposed!
The growing numbrs of OSINT sources out there is mind-boggling, and
most remain free or at least provide API keys free of charge for low
query volumes. In this release, eight new modules have been introduced:
SecurityTrails (sfp_securitytrails): One of my favourite recent discoveries, SecurityTrails
has truly a shedload of DNS and Whois data that any threat intelligence
analyst, security analyst or investigator should look into. This module
will query their API for IP addresses, domain names, e-mail addresses
and owned netblocks to identify co-hosted sites, domains registered
under the same e-mail address and more. An API key is required, however
limited free usage is provided. Check out their blog post about the integration.
FullContact.com (sfp_fullcontact): FullContact.com
has loads of data about people and companies. This module uses their
API (API key required) to look up domain names, e-mail addresses and
names in an attempt to identify further e-mail addresses and names, but
also physical locations and phone numbers.
ARIN (sfp_arin): ARIN (American Registry for Internet Numbers)
is similar to RIPE (for which SpiderFoot already has a module -
sfp_ripe) in that they provide an API to query information about network
ranges. But more interestingly from an OSINT perspective, you can query
by first and last name, and likewise query by domain name to get
affiliated names. This module will take any identified domain name and
return a list of human names and ARIN registry data, which will then be
scanned by other modules to idenify potential e-mail addresses and
hostnames. It will also look up any names to identify potential relevant
Hacked-Emails.com (sfp_hackedemails): Similar to haveibeenpwned.com, hacked-emails.com
provides a free service to identify e-mail addresses mentioned in data
leaks. This module will query their API for any e-mail address
identified during a scan.
Citadel.pw (sfp_citadel): As above, citadel.pw
provides a way to search a large number of leaks for mention of an
e-mail address, which is what this module will do. Thanks to citadel.pw -
at - protonmail.com for this contribution and for providing a public
API key free of charge!
CIRCL.LU (sfp_circllu): CIRCL.LU
(Computer Incident Response Center, Luxembourg) provide a free, however
upon-request API to query their rich database of historical SSL and DNS
data. This module will take hostnames, owned netblocks, IP addresses
and domain names and identify further IP addresses and hostnames, plus
SSL certificates and co-hosts related to your target.
Quad9.net (sfp_quad9): Quad9.net
aggregate a number of threat intelligence data sources and integrate
them into their resolver, which anyone can point to (188.8.131.52). The
resolver will not resolve anything malicious according to the data feeds
they have integrated. This module will attempt to resolve identified
hostnames, affiliates and co-hosts using 184.108.40.206, and if they fail to
resolve there but do resolve using the configured resolver, will report
them as malicious.
RiskIQ / PassiveTotal (sfp_riskiq): RiskIQ
provide a threat intelligence platform with an API (API key required)
to query their passive DNS and other data. This module will query their
API for any hostname, IP address, domain name or e-mail address
identified, and return owned netblocks, further IP addresses, co-hosted
sites and domain names also registered by the provided e-mail address
Dockerfile is now using the Alpine Linux base image, plus some
other improvements to bring the image down from about 500MB to 90MB. See
this tutorial to try it out.
Stopped reporting IPv6 addresses from the sfp_ripe module, as it
made the malicious modules spin forever on the huge IPv6 address spaces
identified. This will be re-visited sometime when IPv6 sees wider
Updated the sfp_robtex module to honor throttling and be more configurable.
Improved sfp_ripe’s ability to identify netblocks possibly owned by the target.
Handle re-directions when looking for S3 buckets, which will
result in many more being found as Amazon returns 30x in many cases,
which before was being ignored by SpiderFoot.
sfp_whois will now perform Whois lookups for affiliate domains and co-hosted sites.
sfp_onioncity updated to use onion.link.
Enhancements / Bug fixes
Misc. minor bug fixes, performance improvements and tweaks.