Sunday, May 17, 2015

Pushing Hystrix statistics to Graphite

If you plan to use Hystrix for your remote calls, you’ll need great monitoring. Actually Hystrix contains awesome dashboard that is free and easy to use for everyone but it make sense to have at least a general overview of your services in one monitoring app for your whole application. Using the same tool for everyhing allows you to easily find correlations between different metrics (e.g. high latency and low number of sales).


We use Graphite for most of our business and performance metrics so we wanted to plot Hystrix statistics on our dashboards too. And for a newcomer, the path might not be so obvious in the first time.


Good thing is that Netflix engineers thought about users wanting to plug in their own monitoring tools and prepared two plugin endpoints that will help you accomplish this. Detailed descriptions of plugins can be found in the documentation.


Event notifier
If basic statistics about the command being run and finished enough for you, event publisher is good enough choice. If you are using statsd for metrics aggregation it is also the easier one. The important thing to note here - time you spent in Event notifier will count as a time of command execution. So whatever you do, it should be extremely fast. Luckily, StatsD supports UDP metrics reporting and firing up an UDP packet is extremely fast.


Event notifier has two endpoints:
  • markCommandExecution - run at the end of command execution, reports all events that occured during the execution
  • markEvent - for every unique event in every command this method is executed
So in the end, it is just matter of how often do you need to do the reporting. I think that for statsd metrics, markCommandExecution is good enough.

In the end, simple event notifier implementation might look similar as the example below.

public class StatsDEventNotifier extends HystrixEventNotifier {

    private StatsDClient stats;
    private final String keyPrefix = "hystrix.";

    public StatsDEventNotifier(StatsDClient stats) {
        this.stats = stats;
    }

    @Override
    public void markEvent(HystrixEventType eventType, HystrixCommandKey key) {
        String commandPrefix = getCommandKey(key);
        if (eventType == HystrixEventType.SUCCESS) {
            stats.count(commandPrefix + "success");
        }
        if (eventType == HystrixEventType.FAILURE) {
            stats.count(commandPrefix + "failure");
        }
        if (eventType == HystrixEventType.FALLBACK_SUCCESS) {
            stats.count(commandPrefix + "fallback_success");
        }
        if (eventType == HystrixEventType.FALLBACK_FAILURE) {
            stats.count(commandPrefix + "fallback_failure");
        }
        if (eventType == HystrixEventType.EXCEPTION_THROWN) {
            stats.count(commandPrefix + "exception");
        }
    }

    private String getCommandKey(HystrixCommandKey key) {
        return keyPrefix + stats.encode(key.name().toLowerCase()) + ".";
    }
}

3 comments:

  1. That is very interesting; you are a very skilled blogger. I have shared your website in my social networks!


    SAP training in Chennai

    ReplyDelete
  2. Truly an incredible expansion. I have perused this magnificent post. A debt of gratitude is in order for sharing data about it. I truly like that. Much obliged so parcel for your assemble. Solution Case Study

    ReplyDelete
  3. Actuary Assignment Help, Actuarial science assignment help in the actuarial field of science from primary to complex level for the understudies of US UK. Actuary Finance Assignment Help

    ReplyDelete