Generic Metrics Software

Dear lazyweb,

I want to track some metrics about something. I want to download a particular URL at a particular frequency, apply a regexp or do a count to extract a number from the returned info, and store it in a database. I then want to do pretty charts and graphs, all the usual metrics stuff.

So for example, I might want to do:

  • Daily; https://api-dev.bugzilla.mozilla.org/latest/count?component=BzAPI; /data: (\d+)/
  • Daily; http://hg.mozilla.org/mozilla-central/shortlog; count: /diff/
  • Weekly: https://wiki.mozilla.org/index.php?title=Special:RecentChanges&days=7; count: /hist/

And so on. Basically, a generic metrics gathering and display engine. (Or two bits of software, one for each part.) Does this software exist? If not, anyone want to build it?

4 thoughts on “Generic Metrics Software

  1. Sounds like a short Perl script on a cron job.
    Perhaps something along these lines…

    use DateTime; use WWW::Mechanize;
    my $mech = WWW::Mechanize->new();
    require ‘db.pl’; # database abstraction layer based on DBI
    my $today = DateTime->now();
    open CFG, ‘) {
    my ($label, $freq, $url, $regex) = split($_, “; “);
    if (($freq eq ‘Daily’) or
    (($freq eq ‘Weekly’) and ($now->wday() == 1)) or
    (($freq eq ‘Monthly’) and ($now->mday() == 1))) {
    $mech->get($url);
    my ($answer) = $mech->content() =~ $regex;
    db_addrecord(‘tablename’, $label, $now->ymd(), $answer);
    }
    }

    More elaborate things can be done, of course (e.g., if you need to log into to the site before you can access the data you want, or some pages might need real parsing e.g. with HTML::TreeBuilder), but the above should handle the simplest cases.

    Once you have the data in a database, generating the graphs can be a separate script, probably based on GD::Graph or somesuch. I don’t have experience with the making graphs part, but people do it all the time, so how hard can it be?