How to use AND in logstash queries?

asked 2018-11-21 04:40:13 -0500

ssbarnea gravatar image

OpenStack build logs a loaded to http://logstash.openstack.org which is queried by humans or elastic-recheck service.

The queries used by elastic-recheck do look like https://github.com/openstack-infra/el...

The problem that I had is that I only found possible to use the most basic queries like the ones with a single condition.

Any other attempts to paste a query defined in the yaml files into logstash search failed to get desired results and I suspect this may be releated to the syntax expected.

Logstash UI popup seach help window states that it uses lucene syntax and pointing to http://www.elasticsearch.org/guide/en...

image description

The first problem here is that the example does not include any AND examples and it states clearly that implicit operator is OR.

In a huge number of cases we have to look for log line start looks like "FOO*BAR" but used of wilcards is against recomandations due to its serious performance impact, indication being to use AND on the same field to achieve a very similar behavior.

I have clear example that should work but it does not and I am 100% sure there are errors of this type in the last 7 days:

message:"Failed connect to mirror." AND
message:".openstack.org" AND
tags:"console"
  • What is wrong?
  • Is the syntax expected different for those queries?
  • Is there a common syntax that can be used that is compatible with both?

Please note that you may have to update the default time resolution in the top-right corner as by default it is set to lookup only to last-hour.

edit retag flag offensive close merge delete

Comments

1

Can you link to a job log which you expect this query to match? Searching the past week for even just message:"connect to mirror" turns up 0 lines.

fungi gravatar imagefungi ( 2018-11-21 08:23:23 -0500 )edit

2 answers

Sort by ยป oldest newest most voted
0

answered 2018-11-21 10:12:00 -0500

ssbarnea gravatar image

Few weeks back when I tried this query I did got results, I am sure. Still, now I am unable to do the same because the last occurent is no longer on stash. Even the logs from logs.openstack.org were removed.

This was original CR https://review.openstack.org/#/c/608968/ which indicates bug https://bugs.launchpad.net/tripleo/+b... that contains links to logs from 2018-10-08 - ~6 weeks old.

This makes impossible to test its validity. On the other hand I performed other similar queries succesfully which makes me believe that we could close this as "unable to reproduce".

Maybe we should put a big red banner on logstash.openstack.org that reports that it contain logs only from the last 7 days.

edit flag offensive delete link more

Comments

I think it's closer to 10 days? But yes, the Kibana interface allows you to select longer timeframes than we keep data. I wonder if that's configurable...

fungi gravatar imagefungi ( 2018-11-21 13:11:57 -0500 )edit
0

answered 2018-11-21 13:10:11 -0500

Per subsequent IRC discussion in #openstack-infra your syntax looks fine but you were querying for patterns which haven't been seen as recently as the data retention in the ES cluster so that's why there were no results.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2018-11-21 04:40:13 -0500

Seen: 199 times

Last updated: Nov 21 '18