Osweep - Don't Simply Search Osint, Sweep It

If you lot operate inward information technology security, therefore you lot most probable role OSINT to assistance you lot sympathise what it is that your SIEM alerted you lot on together with what everyone else inward the globe understands most it. More than probable you lot are using to a greater extent than than 1 OSINT service because most of the fourth dimension OSINT volition entirely render you lot amongst reports based on the final analysis of the IOC. For some, that's practiced enough. They do network together with electronic mail blocks, do novel rules for their IDS/IPS, update the content inward the SIEM, do novel alerts for monitors inward Google Alerts together with DomainTools, etc etc. For others, they deploy these same countermeasures based on provided reports from their third-party tools that the fellowship is paying THOUSANDS of dollars for.

The work amongst both of these is that the analyst needs to dig a footling deeper (ex. FULLY deobfuscate a PowerShell ascendency establish inward a malicious macro) to assemble all of the IOCs. And what if the additional IOC(s) you lot are basing your analysis on has nil to do amongst what is truthful most that site today? And therefore you lot larn pwned? And therefore other questions from management arise...

See where this is headed? You're most to larn a pinkish sideslip together with walked out of the edifice therefore you lot tin plough over notice start looking for roughly other task inward a dissimilar describe of work.

So why did you lot larn pwned? You know that if you lot wasted fourth dimension gathering all the IOCs for that 1 warning manually, it would receive got taken you lot one-half of your shift to consummate together with you lot would've got pwned regardless.

The fix? OSweep™.

Prerequisites

• Splunk 7.1.3 >
• Python 2.7.14 > ($SPLUNK_HOME/bin/python)

Setup

1. Open a terminal together with run the next commands equally the user running Splunk:

cd /opt/splunk/etc/apps git clone https://github.com/leunammejii/osweep.git mv osweep-master osweep sudo -H -u $SPLUNK_USER /opt/splunk/bin/splunk restart # $SPLUNK_USER = User running Splunk

2. Edit "config.py" together with add together the necessary values equally strings to the config file:

vim ./osweep/etc/config.py

Note: Values for the proxies should live the total URL including the port (ex. http://<IP Adress>:<Port>).
3. Save "config.py" together with closed the terminal.
4. Install Pip packages:

cd /opt/splunk/etc/apps/osweep/bin bash py_pkg_update.sh

Commands

• crtsh - https://crt.sh/
• cybercrimeTracker - http://cybercrime-tracker.net/index.php
• cymon - https://cymon.io/
• greyNoise - https://greynoise.io/
• phishingCatcher - https://github.com/x0rz/phishing_catcher
• phishingKitTracker - https://github.com/neonprimetime/PhishingKitTracker
• ransomwareTracker - https://ransomwaretracker.abuse.ch/
• threatcrowd - https://www.threatcrowd.org/
• twitter - https://twitter.com/
• urlhaus - https://urlhaus.abuse.ch/
• urlscan - https://urlscan.io/

Usage
Feed Overview - Dashboard
Three of the dashboards below role lookup tables to shop the information feed from the sources. This dasboard shows the electrical flow stats compared to the previous day.

The Round Table - Dashboard

1. Switch to the The Round Table dashboard inward the OSweep™ app.
2. Add the listing of IOCs to the "IOC (+)" textbox to know which source has the most information.
3. Click "Submit".
4. After the panels receive got populated, click on 1 to live redirected to the corresponding dashboard to encounter the results.

Certificate Search - Dashboard

1. Switch to the Certificate Search dashboard inward the OSweep™ app.
2. Add the listing of IOCs to the "Domain, IP (+)" textbox.
3. Select "Yes" or "No" from the "Wildcard" dropdown to search for subdomains.
4. Click "Submit".

Certificate Search - Adhoc

| crtsh <DOMAINS> | fillnull value="-" | search NOT "issuer ca id"="-" | dedup "issuer ca id" "issuer name" "name value" "min cert id" "min entry timestamp" "not before" "not after" | tabular array "issuer ca id" "issuer name" "name value" "min cert id" "min entry timestamp" "not before" "not after" | variety - "min cert id"

or to search for subdomains,

| crtsh subdomain <DOMAINS> | fillnull value="-" | search NOT "issuer ca id"="-" | dedup "issuer ca id" "issuer name" "name value" "min cert id" "min entry timestamp" "not before" "not after" | tabular array "issuer ca id" "issuer name" "name value" "min cert id" "min entry timestamp" "not before" "not after" | variety - "min cert id"

or to search for wildcard,

| crtsh wildcard <DOMAINS> | fillnull value="-" | search NOT "issuer ca id"="-" | dedup "issuer ca id" "issuer name" "name value" "min cert id" "min entry timestamp" "not before" "not after" | tabular array "issuer ca id" "issuer name" "name value" "min cert id" "min entry timestamp" "not before" "not after" | variety - "min cert id"

CyberCrime Tracker - Dashboard

1. Switch to the CyberCrime Tracker dashboard inward the OSweep™ app.
2. Add the listing of IOCs to the 'Domain, IP (+)' textbox.
3. Select whether the results volition live grouped together with how from the dropdowns.
4. Click 'Submit'.

CyberCrime Tracker - Adhoc

| cybercrimeTracker <IOCs> | fillnull value="-" | search NOT date="-" | dedup appointment url ip "vt latest scan" "vt ip info" type | tabular array appointment url ip "vt latest scan" "vt ip info" type

Cymon - Dashboard

1. Switch to the Cymon dashboard inward the OSweep™ app.
2. Add the listing of IOCs to the "Domain, IP, MD5, SHA256 (+)" textbox.
3. Select whether the results volition live grouped together with how from the dropdowns.
4. Click "Submit".

Cymon - Adhoc

| cymon <IOCs> | tabular array "feed id" feed championship description tags timestamp ip url hostname domain md5 sha1 sha256 ssdeep "reported by" terra firma urban center lat lon

GreyNoise - Dashboard

1. Manually download information feed (one-time only)

| greyNoise feed

2. Switch to the GreyNoise dashboard inward the OSweep™ app.
3. Add the listing of IOCs to the 'Domain, IP, Scanner Name (+)' textbox.
4. Select whether the results volition live grouped together with how from the dropdowns.
5. Click 'Submit'.

GreyNoise - Adhoc

| greynoise <IOCs> | fillnull value="-"  | search NOT "last updated"="-"  | dedup category confidence "last updated" advert ip intention "first seen" datacenter tor "rdns parent" link org bone asn rdns | tabular array category confidence "last updated" advert ip intention "first seen" datacenter tor "rdns parent" link org bone asn rdns | variety - "Last Updated"

Phishing Catcher - Dashboard

1. Switch to the Phishing Catcher dashboard inward the OSweep™ app.
2. Select whether you lot desire to monitor the logs inward realtime or add together a listing of domains.
3. If Monitor Mode is "Yes":
   • Add a search string to the 'Base Search' textbox.
   • Add the plain advert of the plain containing the domain to the "Field Name" textbox.
   • Select the fourth dimension arrive at to search.
4. If Monitor Mode is "No":
   • Add the listing of domains to the 'Domain (+)' textbox.
5. Click 'Submit'.

Phishing Catcher - Adhoc

| phishingCatcher <DOMAINS> | tabular array domain "threat level" score

Phishing Kit Tracker - Dashboard

1. Manually download information feed (one-time only)

| phishingKitTracker feed

2. Switch to the Phishing Kit Tracker dashboard inward the OSweep™ app.

Ransomare Tracker - Dashboard

1. Manually download information feed (one-time only)

| ransomwareTracker feed

2. Switch to the Ransomare Tracker dashboard inward the OSweep™ app.
3. Add the listing of IOCs to the 'Domain, IP, Malware, Status, Threat, URL (+)' textbox.
4. Select whether the results volition live grouped together with how from the dropdowns.
5. Click 'Submit'.

Ransomare Tracker - Adhoc

| ransomwareTracker <DOMAINS> | fillnull value="-"  | search NOT "firstseen (utc)"="-" | dedup "firstseen (utc)" threat malware host "ip address(es)" url condition registrar asn(s) terra firma | tabular array "firstseen (utc)" threat malware host "ip address(es)" url condition registrar asn(s) terra firma | variety "firstseen (utc)"

ThreatCrowd - Dashboard

1. Switch to the ThreatCrowd dashboard inward the OSweep™ app.
2. Add the listing of IOCs to the 'IP, Domain, or Email (+)' textbox.
3. Select the IOC type.
4. Click 'Submit'.

Twitter - Dashboard

1. Switch to the Twitter dashboard inward the OSweep app.
2. Add the listing of IOCs to the "Search Term (+)" textbox.
3. Click "Submit".

Twitter - Adhoc

| twitter <IOCs> | eval epoch=strptime(timestamp, "%+") | fillnull value="-"  | search NOT timestamp="-"  | dedup timestamp tweet url  | variety - epoch | tabular array timestamp tweet url hashtags "search term"

URLhaus - Dashboard

1. Manually download information feed (one-time only)

| urlhaus feed

2. Switch to the URLhaus dashboard inward the OSweep™ app.
3. Add the listing of IOCs to the 'Domain, IP, MD5, SHA256, URL (+)' textbox.
4. Select whether the results volition live grouped together with how from the dropdowns.
5. Click 'Submit'.

URLhaus - Adhoc

| urlhaus <IOCs> | fillnull value="-" | search NOT "provided ioc"="-" | dedup id dateadded url payload "url status" threat tags "urlhaus link" | tabular array id dateadded url payload "url status" threat tags "urlhaus link"

urlscan.io - Dashboard

1. Switch to the urlscan.io dashboard inward the OSweep™ app.
2. Add the listing of IOCs to the 'Domain, IP, SHA256 (+)' textbox.
3. Select whether the results volition live grouped together with how from the dropdowns.
4. Click 'Submit'.

urlscan.io - Adhoc

| urlscan <IOCs> | fillnull value="-"  | search NOT url="-" | dedup url domain ip ptr server urban center terra firma asn asnname filename filesize mimetype sha256  | tabular array url domain ip ptr server urban center terra firma asn asnname filename filesize mimetype sha256  | variety sha256

Destroy
To take the projection completely, run the next commands:

rm -rf /opt/splunk/etc/apps/osweep

Things to know
All commands convey input from the pipeline. Either role the fields or table ascendency to select 1 plain containing the values that the ascendency accepts together with pipage it to the ascendency amongst the start declaration beingness the plain name.

<search> | fields <FIELD NAME> | <OSWEEP COMMAND> <FIELD NAME>

ex. The next volition allow a user to discovery other URLs analyzed past times URLhaus that are hosting the same Emotet malware equally ahsweater[d]com together with grouping it past times the payload:

| urlhaus ahsweater.com | fields payload | urlhaus payload | stats values(url) AS url BY payload

You tin plough over notice also pipage the results of 1 ascendency into a totally dissimilar ascendency to correlate data.

Dashboards Coming Soon

• Alienvault
• Censys
• Hybrid-Analysis
• Malshare
• PulseDive

Please fork, do merge requests, together with assistance brand this better.

Download Osweep