Monitor Go Agents with Graphite




Say we want to use Graphite to monitor free memory on all Go agents. There are many ways to do it (e.g. cron job, free -s) but you will have to set it up on each agent. Here is one way to do it using a combination of Go’s features.

We’ll send data using the plaintext protocol.

echo "local.random.diceroll 4 `date +%s`" | nc ${SERVER} ${PORT};

I find redirection cooler, so:

echo ”server-name.freemem 23 `date +%s`” >/dev/tcp/carbon-server/2003

where 23 is percentage free memory. To get free memory as a single number, we use this tip:

free | grep Mem | awk '{print $4/$2 * 100.0}'

Thus, we have the following Go custom command that also reports hostname dynamically:

<exec command="/bin/bash">
   <arg>
-c</arg>
   <arg>
echo "`hostname`.freemem `free | grep Mem | awk '{print $4/$2 * 100.0}'` `date +%s`" &gt; /dev/tcp/carbon-server/2003</arg>
</exec>


Next, we set up a timer trigger based pipeline (with a dummy material, fetch set to false) so that it runs this command every x seconds.

To ensure that it always runs on the agent you want to monitor, you could use a job resource with value equal to the hostname of the agent. On the other hand, you could check run-on-all-agents on the job to monitor every agent - present and future, for now and ever!