Just grabbed the user agent out of apache (combined) log file: cat my_logfile.log | awk ‘{print$14}’ |cut -d’/’ -f1 |sort |uniq -dc |sort -nr xxxx SemrushBot xxxx MJ12bot Apache (combined) log file cat my_logfile.log | awk ‘{print$12}’ |cut -d'”‘ -f2 |cut -d’/’ -f1 |sort |uniq -dc |sort -nr xxxx DomainCrawler xxxx Mozilla xxxx Baiduspider xxxx ..