Basic Web Automation for Collecting Activity

[scottalanmiller]

Some people have asked for this and it has been a while since I knew where the code was stored but I stumbled on it the other day and figured that I would share. People are often curious how practical screen scraping is done. This is a very crufty example but it works and is easily automatable (throw it into cron and you are done.) No special tools needed:

#!/usr/bin/bash
#
# Version 1.0
# 11 January 2014
#
# Abstract: To automate the collection of Spiceworks activity data at the current time through the direct querying of the SW Community

##  Initialize temp file
echo "" > /tmp/swreport

## Generate Data
for i in $(cat /opt/scripts/staff); do
    echo $i $(curl -b /opt/scripts/cookies.txt http://community.spiceworks.com/people/$i/activity 2>/dev/null| grep Points \
    | cut -d">" -f 3 | cut -d "<" -f1) $(curl -b /opt/scripts/cookies.txt http://community.spiceworks.com/people/$i/activity \
    2>/dev/null| grep Answer | cut -d">" -f 3 | cut -d "<" -f1)  $(curl -b /opt/scripts/cookies.txt \
    http://community.spiceworks.com/people/$i/activity 2>/dev/null| grep Posts| cut -d">" -f 3 | cut -d "<" -f1) \
    $(curl -b /opt/scripts/cookies.txt http://community.spiceworks.com/people/$i/activity 2>/dev/null |  grep '"title"' | \
    cut -d"<" -f2 | cut -d">" -f2) | sed 's/,//g' >> /tmp/swreport
done

## Format Report
echo "Screenname Points BAs HPs Pepper" > /tmp/swreport.sorted
echo " " >> /tmp/swreport.sorted
sort -k2nr /tmp/swreport >> /tmp/swreport.sorted

column -c 4 -t -s $' ' /tmp/swreport.sorted > /tmp/swreport.col

echo "This daily report of Spiceworks standings is generated automatically by the swreport.sh script on to-lnx-dev. \
      This report is created by directly querying the Spiceworks Community at the time of creation and is completely \
      up to date at creation time." > /tmp/swreport.for
echo "" >> /tmp/swreport.for
cat /tmp/swreport.col >> /tmp/swreport.for

## Send Out Report
mail -s "Spiceworks Daily Report - Straight from the Server" [email protected] < /tmp/swreport.for

The script is quick and dirty with hard coded locations. It requires the text file /opt/scripts/staff to contain the list of user names to query. Add as many names to the list as you want in the report.

[scottalanmiller]

Before this script will run properly, you have to use curl and a valid account to acquire a cookie to pass with the script, as well.