<div id="readability-page-1" class="page"><div id="post-body-2483740859559321780" itemprop="articleBody">
DB 데이터 연동.
<div><pre><code>[2025-09-07T11:06:48,019][INFO ][logstash.agent&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;] Pipelines running {:count=&gt;1, :running_pipelines=&gt;[:main], :non_running_pipelines=&gt;[]}[2025-09-07T11:07:00,219][INFO ][logstash.inputs.jdbc&nbsp; &nbsp; &nbsp;][main][d56fa9ad43d4dc4ae19f64f322de33854c6e9d33b5fc7c06354603bf613db2ee] (0.001000s)select a.cid, a.timestamp, b.sig_name, inet_ntoa(c.ip_src), inet_ntoa(c.ip_dst), unhex(d.data_payload)from event a, signature b, iphdr c, data dwhere a.signature = b.sig_idand a.sid = c.sid and a.cid = c.cidand a.sid = d.sid and a.cid = d.cidand a.cid &gt; 0<a name="more"></a>{&nbsp; &nbsp; &nbsp; "inet_ntoa(c.ip_src)" =&gt; "192.168.31.230",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"sig_name" =&gt; "sql-injection-and",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"@version" =&gt; "1",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "timestamp" =&gt; 2018-01-08T04:54:52.000Z,&nbsp; &nbsp; "unhex(d.data_payload)" =&gt; #&lt;Sequel::SQL::Blob:0xad0 bytes=190 start="GET /s?ie=" end="t: */*\r\n\r\n"&gt;,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "cid" =&gt; 4,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"@timestamp" =&gt; 2025-09-07T02:07:00.232Z,&nbsp; &nbsp; &nbsp; "inet_ntoa(c.ip_dst)" =&gt; "119.63.197.151"}</code></pre></div>
로그스태시가 데이터를 가져온 시간인 @timestamp와 데이터의 원본 시간인 timestamp의 포맷이 똑같은 ISO8601이다. 그래서 엘라스틱은 둘 다 시간 필드로 인식함. 굳이 date 필터 작업을 안 해도 된다는 얘기.&nbsp;단 %{+yyyy.MM.dd} 등의 구문을 사용해서 시간 정보 활용을 시도하면 @timestamp에 접근하기 때문에 헷갈리기 싫으면 그냥 date 필터 작업을 해주는 게 낫다.
<div><pre><code>filter {&nbsp;date {&nbsp; match =&gt; ["timestamp", "ISO8601"]&nbsp;}}</code></pre></div>
그런데 안 됨(..)
<div><pre><code>{&nbsp; &nbsp; &nbsp; "inet_ntoa(c.ip_src)" =&gt; "192.168.31.230",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"sig_name" =&gt; "sql-injection-and",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"@version" =&gt; "1",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "timestamp" =&gt; 2018-01-08T04:54:47.000Z,&nbsp; &nbsp; "unhex(d.data_payload)" =&gt; #&lt;Sequel::SQL::Blob:0xbb4 bytes=351 start="GET /s?ie=" end="t: */*\r\n\r\n"&gt;,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "cid" =&gt; 1,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"tags" =&gt; [&nbsp; &nbsp; &nbsp; &nbsp; [0] "_dateparsefailure"&nbsp; &nbsp; ],&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"@timestamp" =&gt; 2025-09-07T02:10:00.041Z,&nbsp; &nbsp; &nbsp; "inet_ntoa(c.ip_dst)" =&gt; "119.63.197.139"}</code></pre></div>
date 필터가 ISO8601 포맷을 해석하지 못하는 모양. date 필터를 걸기 전에 먼저 문자열로 바꿔줘야 한다.
<div><pre><code>filter {&nbsp;mutate {&nbsp; convert =&gt; {"timestamp" =&gt; "string"}&nbsp;}&nbsp;date {&nbsp; match =&gt; ["timestamp", "ISO8601"]&nbsp;}}</code></pre></div>
<div><pre><code>{&nbsp; &nbsp; &nbsp; "inet_ntoa(c.ip_src)" =&gt; "192.168.31.230",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"sig_name" =&gt; "sql-injection-and",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"@version" =&gt; "1",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "timestamp" =&gt; "2018-01-08T04:55:12.000Z",&nbsp; &nbsp; "unhex(d.data_payload)" =&gt; #&lt;Sequel::SQL::Blob:0xcbe bytes=264 start="GET /s?ie=" end="t: */*\r\n\r\n"&gt;,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "cid" =&gt; 5,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"@timestamp" =&gt; 2018-01-08T04:55:12.000Z,&nbsp; &nbsp; &nbsp; "inet_ntoa(c.ip_dst)" =&gt; "119.63.197.151"}</code></pre></div>
<div>관련 글<ul><li><a href="https://kangmyounghun.blogspot.com/2021/02/logstash-date-3rd.html" target="">Logstash 필터 date - 3rd</a></li><li><a href="https://kangmyounghun.blogspot.com/2019/01/logstash-date.html" target="">Logstash 필터 date</a></li><li><a href="http://kangmyounghun.blogspot.com/2017/06/elasticsearch-grok.html" target="_blank">Logstash 필터 grok</a></li><li><a href="http://kangmyounghun.blogspot.com/2017/07/logstash-mutate.html" target="_blank">Logstash 필터 mutate</a></li><li><a href="http://kangmyounghun.blogspot.kr/2017/10/logstash-ruby.html" target="_blank">Logstash 필터 ruby</a></li><li><a href="http://kangmyounghun.blogspot.kr/2018/02/logstash-geoip.html" target="_blank">Logstash 필터&nbsp;geoip</a></li><li><a href="http://kangmyounghun.blogspot.kr/2018/04/logstash-dissect.html" target="_blank">Logstash 필터 dissect</a></li><li><a href="http://kangmyounghun.blogspot.com/2018/09/logstash-kv.html" target="_blank">Logstash 필터&nbsp;kv</a></li><li><a href="https://kangmyounghun.blogspot.com/2019/02/logstash-translate.html" target="_blank">Logstash 필터 translate</a></li><li><a href="https://kangmyounghun.blogspot.com/2019/08/logstash-drop.html" target="_blank">Logstash 필터 drop</a></li><li><a href="https://kangmyounghun.blogspot.com/2019/11/logstash-useragent.html" target="_blank">Logstash 필터 useragent</a></li><li><a href="https://kangmyounghun.blogspot.com/2020/02/logstash-elapsed.html" target="_blank">Logstash 필터 elapsed</a></li><li><a href="https://kangmyounghun.blogspot.com/2020/04/logstash-fingerprint.html" target="_blank">Logstash 필터 fingerprint</a></li><li><a href="https://kangmyounghun.blogspot.com/2022/12/logstash-csv.html" target="_blank">Logstash 필터 csv</a></li><li><a href="https://kangmyounghun.blogspot.com/2023/11/logstash-dns.html">Logstash 필터 dns</a></li><li><a href="https://kangmyounghun.blogspot.com/2025/01/logstash-split.html">Logstash 필터 split</a></li><li><a href="https://kangmyounghun.blogspot.com/2020/02/logstash-codec-multiline.html" target="_blank">Logstash codec 플러그인 multiline</a></li></ul></div>

</div></div>

케세라세라

DB 데이터 연동.
[2025-09-07T11:06:48,019][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2025-09-07T11:07:00,219][INFO ][logstash.inputs.jdbc     ][main][d56fa9ad43d4dc4ae19f64f322de33854c6e9d33b5fc7c06354603bf613db2ee] (0.001000s)
select a.cid, a.timestamp, b.sig_name, inet_ntoa(c.ip_src), inet_ntoa(c.ip_dst), unhex(d.data_payload)
from event a, signature b, iphdr c, data d
where a.signature = b.sig_id
and a.sid = c.sid and a.cid = c.cid
and a.sid = d.sid and a.cid = d.cid
and a.cid > 0
{
      "inet_ntoa(c.ip_src)" => "192.168.31.230",
                 "sig_name" => "sql-injection-and",
                 "@version" => "1",
                "timestamp" => 2018-01-08T04:54:52.000Z,
    "unhex(d.data_payload)" => #<Sequel::SQL::Blob:0xad0 bytes=190 start="GET /s?ie=" end="t: */*\r\n\r\n">,
                      "cid" => 4,
               "@timestamp" => 2025-09-07T02:07:00.232Z,
      "inet_ntoa(c.ip_dst)" => "119.63.197.151"
}
로그스태시가 데이터를 가져온 시간인 @timestamp와 데이터의 원본 시간인 timestamp의 포맷이 똑같은 ISO8601이다. 그래서 엘라스틱은 둘 다 시간 필드로 인식함. 굳이 date 필터 작업을 안 해도 된다는 얘기. 
단 %{+yyyy.MM.dd} 등의 구문을 사용해서 시간 정보 활용을 시도하면 @timestamp에 접근하기 때문에 헷갈리기 싫으면 그냥 date 필터 작업을 해주는 게 낫다.
filter {
 date {
  match => ["timestamp", "ISO8601"]
 }
}
그런데 안 됨(..)
{
      "inet_ntoa(c.ip_src)" => "192.168.31.230",
                 "sig_name" => "sql-injection-and",
                 "@version" => "1",
                "timestamp" => 2018-01-08T04:54:47.000Z,
    "unhex(d.data_payload)" => #<Sequel::SQL::Blob:0xbb4 bytes=351 start="GET /s?ie=" end="t: */*\r\n\r\n">,
                      "cid" => 1,
                     "tags" => [
        [0] "_dateparsefailure"
    ],
               "@timestamp" => 2025-09-07T02:10:00.041Z,
      "inet_ntoa(c.ip_dst)" => "119.63.197.139"
}
date 필터가 ISO8601 포맷을 해석하지 못하는 모양. date 필터를 걸기 전에 먼저 문자열로 바꿔줘야 한다.
filter {
 mutate {
  convert => {"timestamp" => "string"}
 }
 date {
  match => ["timestamp", "ISO8601"]
 }
}
{
      "inet_ntoa(c.ip_src)" => "192.168.31.230",
                 "sig_name" => "sql-injection-and",
                 "@version" => "1",
                "timestamp" => "2018-01-08T04:55:12.000Z",
    "unhex(d.data_payload)" => #<Sequel::SQL::Blob:0xcbe bytes=264 start="GET /s?ie=" end="t: */*\r\n\r\n">,
                      "cid" => 5,
               "@timestamp" => 2018-01-08T04:55:12.000Z,
      "inet_ntoa(c.ip_dst)" => "119.63.197.151"
}
관련 글
Logstash 필터 date - 3rd
Logstash 필터 date
Logstash 필터 grok
Logstash 필터 mutate
Logstash 필터 ruby
Logstash 필터 geoip
Logstash 필터 dissect
Logstash 필터 kv
Logstash 필터 translate
Logstash 필터 drop
Logstash 필터 useragent
Logstash 필터 elapsed
Logstash 필터 fingerprint
Logstash 필터 csv
Logstash 필터 dns
Logstash 필터 split
Logstash codec 플러그인 multiline

Logstash 필터 date - 4th

댓글