Elastic Watcher Tips & Tricks, Part 1: Speeding Up Creating & Testing

This is the first in a multi-part series about Elastic Watcher. We would walk through some tips and tricks to speed up creating and testing Elastic watch.

How to Create a watch?

To create Elastic Watch, we will start from Kibana Watcher UI or Watcher APIs. In this article, we will focus on creating an advanced watch.

A watch request body mainly consists of a trigger, input, condition, actions, and metadata. You can check the Elastic document to check the detailed watch definition and the flow of watch execution.

PUT _watcher/watch/my_first_watch
{
  "trigger" : {},
  "input" : {},
  "condition" : {},
  "transform" : {},
  "actions" : {},
  "metadata" : {}
}

Tip #1: How to create an input query?

A simple request like the following is a good starting.

"request": {
  "indices": "app-logs",
  "body": {
    "size": 0,
    "query": {
      "match": {
        "logLevel": "error"
      }
    }
  }
}

But in the real world, the case will be much more complicated. Your boss may come to you with more requirements.

• Could you exclude the log from class com.example.businesstransactionapp.controller.health? Develop log some messages as ERROR by mistake.
• Downstream app AccountService API sometimes is slow and we got an intermittent timeout error. We have a retry mechanism in our businesstransactionapp and all transactions are successful. You can ignore those timeout errors now.

The beginner who is not familiar with Elasticsearch Query DSL may spend hours and hours trying to correct the query to meet this requirement. So the tip here is you should use kibana to generate and test queries instead of using watcher directly.

Tip #2: Watch Test

You set your Watch to trigger every 15 minutes. Every time when you make the change to your watch will you wait for 15 minutes and then check the result? Or you may say that I would not waste my time, let me change the interval to 10s to test watch. Then you got a production call….. A few hours later after you solved your production issue and came back to continue working on your watch and you found out your email box was flooded with hundreds of email sending from your test watch. Your co-workers were not able to use kibana because your inefficient query in the watch slowed down the whole elasticsearch cluster.

So the tip here is you don’t need to change the trigger interval or waiting for the watch is a trigger, using the watch API to execute the watch and check the result immediately.