Say we want to use Graphite to monitor free memory on all Go agents. There are many ways to do it (e.g. cron job, free -s) but you will have to set it up on each agent. Here is one way to do it using a combination of Go’s features.
We’ll send data using the plaintext protocol.
echo "local.random.diceroll 4 `date +%s`" | nc ${SERVER} ${PORT};
I find redirection cooler, so:
echo ”server-name.freemem 23 `date +%s`” >/dev/tcp/carbon-server/2003
where 23 is percentage free memory. To get free memory as a single number, we use this tip:
free | grep Mem | awk '{print $4/$2 * 100.0}'
Thus, we have the following Go custom command that also reports hostname dynamically:
<exec command="/bin/bash">
<arg>-c</arg>
<arg>echo "`hostname`.freemem `free | grep Mem | awk '{print $4/$2 * 100.0}'` `date +%s`" > /dev/tcp/carbon-server/2003</arg>
</exec>
Next, we set up a timer trigger based pipeline (with a dummy material, fetch set to false) so that it runs this command every x seconds.
To ensure that it always runs on the agent you want to monitor, you could use a job resource with value equal to the hostname of the agent. On the other hand, you could check run-on-all-agents on the job to monitor every agent - present and future, for now and ever!