Since the last newsletter, we have been working on a language to perform searches on the ONYPHE search engine. It has been put in production last week, that is, end of april. Thanks to that, new APIs are now also possible and have been made available.
But there is more, we have enriched threatlist and inetnum categories of data by adding geolocation information. We also added subnet information for any single IP address to synscan and datascan categories of data. It will make it easy to pivot on a host and find anything else on the same subnet.
Finally, we industrialized Dark Web scanning and added two new protocols we are now watching: RDP (Remote Desktop Protocol) and DNS (Domain Name System). This makes a total of 13 protocols we are able to identify, whatever the listening port.
ONYPHE query language
We are in the last mile regarding the launch of our commercial offer. We had to finalize the query language, or the filters, before being able to sell the service. This is now done but not yet accessible to the mass (be patient).
This language allows users to query ONYPHE data with filters and CIDR mask, for instance. It will be as easy as typing key:value and keeping adding these filters to just get the information you want.
Another special filter is the category one. By default, ONYPHE will search for the datascan category (application data), but you may want to search for resolver data (passive DNS) or pastries (like pastebin).
category:datascan product:Apache port:443 os:Windows category:synscan port:23 country:FR os:Linux category:synscan ip:126.96.36.199/21 os:Linux port:23 category:inetnum organization:"OVH SAS" category:inetnum netname:APNIC-LABS category:threatlist country:RU category:threatlist ip:188.8.131.52 category:pastries ip:184.108.40.206/24 category:pastries domain:amazonaws.com category:resolver ip:220.127.116.11/16 category:resolver domain:example.com
With a full access, it is nearly 70 filters that are accessible. You may use these filters from the Web site on the search line or from the API, as described on the dedicated documentation. We will document this language more thoroughly when it will be available for end-users.
Geolocation enrichment for threatlist and inetnum categories
Starting from february 2017, we added geolocation information to threatlists. It was not done before because threatlists we aggregate were not giving this information. We considered that it may make sens, that if an IP was classified as a threat and its geolocation changed from one day to another, it may not be harmful anymore. We let the user decide how to consider that new information.
The same is true for inetnum: it is good to get these netblocks from RIR, but only the country is given. By adding geolocation information, we can enrich it with organization and GPS coordinates, for instance. Adding organization allows to perform such a thing as a netname to organization lookup (or the reverse).
Both of these enrichments are now readily available on any new data.
When searching for information about your own IP addresses, you may find yourself in the situation where you want to find everything on your complete subnet. For that to work, the subnet information has to be put somewhere. This is now done, subnet information is added to synscan and datascan, and every category where geolocation is applied.
Thanks to that, you will be able to pivot on ip or subnet data by a simple click (or query filter) when the commercial offer we be available.
Scanning the Dark Web
Another addition is the scanning of the so-called Dark Web. Those .onion Web sites reachable only from the Tor network. We have compiled a first pass list of nearly 40,000 onion sites. Thanks to that list, we will be able to crawl the Dark Web and enrich this list by discovering new onion links, just like any search engine.
At the time of writing and taking into account this list, we have indexed more than 5,100 active hidden sites.
Note: don’t try the displayed search query as it is only available for ONYPHE purposes.
New watched protocols and fingerprinting
Finally, we added two new protocols along with fingerprinting of services: RDP (Remote Desktop Protocol) and DNS (Domain Name System). For RDP, we are able to differentiate between the Microsoft implementation and the XRDP one. That’s a start and should be very helpful. Thanks to that, we can enrich the information with the os.
For the DNS protocol, we simply use the version.bind request. And here is the TOP10 product in use on the Internet, being a resolver or authoritative server. The percentage is about this TOP10 only, not about all detected servers. Thus, BIND accounts for 78% of the TOP10 products discovered on the Internet.
As you can see, we have many new addition to share with you in this newsletter. The next time we share something with you will be the final pricing and the opening of commercial subscriptions.
In the meantime, for those not already registered, you can create your free user account and gain access to your API key by registering here: