Posts

Showing posts with the label Scraping

Skiptracer - Osint Webscaping Framework

Image
Initial gear upward on vectors for recon commonly involve utilizing pay-for-data/API (Recon-NG), or paying to utilise transforms (Maltego) to teach information mining results. Skiptracer utilizes to a greater extent than or less basic python webscraping (BeautifulSoup) of PII paywall sites to compile passive information on a target on a ramen noodle budget. Example: Installation $ git clone https://github.com/xillwillx/skiptracer.git skiptracer $ cd skiptracer Install requirements $ pip install -r requirements.txt Run $ python skiptracer.py -l (phone|email|sn|name|plate) Usage Full details on how to purpose Skiptracer are on the wiki located here Download Skiptracer

Machinae V1.4.8 - Safety Word Collector

Image
Machinae is a tool for collecting tidings from world sites/feeds nearly diverse security-related pieces of data: IP addresses, domain names, URLs, electronic mail addresses, file hashes, together with SSL fingerprints. It was inspired past times Automater , about other first-class tool for collecting information. The Machinae projection was born from wishing to amend Automater inwards four areas: Codebase - Bring Automater to python3 compatibility piece making the code to a greater extent than pythonic Configuration - Use a to a greater extent than human readable configuration format (YAML) Inputs - Support JSON parsing out-of-the-box without the ask to write regular expressions, but even hence back upward regex scraping when needed Outputs - Support additional output types, including JSON, piece making extraneous output optional Installation Machinae tin live installed using pip3: pip3 install machinae Or, if you're feeling adventurous, tin live installed stra...

Scavenger - Crawler Searching For Credential Leaks On Dissimilar Glue Sites

Image
Just the code of my OSINT bot searching for sensitive information leaks on dissimilar glue sites. Search terms: credentials private RSA keys Wordpress configuration files MySQL connect strings onion links links to files hosted within the onion network (PDF, DOC, DOCX, XLS, XLSX) Keep inwards mind: This bot is non beautiful. The code is non consummate so far. Some parts similar integrating the credentials inwards a database are missing inwards this online repository. If you lot desire to piece of occupation this code, experience costless to create so. Keep inwards hear you lot accept to customize things to become inwards run on your system. IMPORTANT The bot tin give the sack locomote run inwards ii major modes: API mode Scraping fashion (using TOR) Is highly recommend using the API mode. It is the intended method of scraping pastes from Pastebin.com too it is only fair to create so. The only matter you lot require is a Pastebin.com PRO draw of pi...

Crosslinked - Linkedin Enumeration Tool To Extract Valid Employee Names From An Scheme Through Search Engine Scraping

Image
CrossLinked simplifies the processes of searching LinkedIn to collect valid employee names when performing password spraying or around other safety testing against an organization. Using similar search engine scraping capabilities flora inwards tools similar subscraper as well as pymeta , CrossLinked volition abide by valid employee names as well as aid format the information according to the organization's concern human relationship naming convention. Results volition live written to a 'names.txt' file inwards the electrical flow directory for farther testing. Setup git clone https://github.com/m8r0wn/crosslinked cd crosslinked pip3 install -r requirements.txt Examples python3 crosslinked.py -f '{first}.{last}@domain.com' company_name python3 crosslinked.py -f 'domain\{f}{last}' -t 45 -j 0.5 company_name Usage -h, --help exhibit this aid message as well as dice -t TIMEOUT Timeout [seconds] for search threads (Default: 25) -...

Amass - In-Depth Dns Enumeration As Well As Network Mapping

Image
The OWASP Amass tool suite obtains subdomain names past times scraping information sources, recursive animate beingness forcing, crawling spider web archives, permuting/altering names as well as contrary DNS sweeping. Additionally, Amass uses the IP addresses obtained during resolution to notice associated netblocks as well as ASNs. All the information is as well as hence used to range maps of the target networks. Information Gathering Techniques Used: DNS: Basic enumeration, Brute forcing (upon request), Reverse DNS sweeping, Subdomain cite alterations/permutations, Zone transfers (upon request) Scraping: Ask, Baidu, Bing, CommonCrawl, DNSDB, DNSDumpster, DNSTable, Dogpile, Exalead, FindSubdomains, Google, IPv4Info, Netcraft, PTRArchive, Riddler, SiteDossier, ThreatCrowd, VirusTotal, Yahoo Certificates: Active pulls (upon request), Censys, CertDB, CertSpotter, Crtsh, Entrust APIs: BinaryEdge, BufferOver, CIRCL, HackerTarget, PassiveTotal, Robtex, SecurityTrails, Shodan...