<div id="readability-page-1" class="page"><div id="post-body-471602865853486287" itemprop="articleBody">
geoip 필터를 이용하면 위경도를 포함한 ip 주소의 다양한 지리 정보를 사용할 수 있다.&nbsp;
<div><pre><code>{&nbsp; &nbsp; "message" =&gt; "122.213.34.56",&nbsp; &nbsp; &nbsp; "geoip" =&gt; {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"longitude" =&gt; 139.6297,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "ip" =&gt; "122.213.34.56",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"country_code3" =&gt; "JP",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "latitude" =&gt; 35.5843,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "timezone" =&gt; "Asia/Tokyo",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"city_name" =&gt; "Chitose",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"region_name" =&gt; "Kanagawa",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"region_code" =&gt; "14",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "location" =&gt; {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "lat" =&gt; 35.5843,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "lon" =&gt; 139.6297<a name="more"></a>&nbsp; &nbsp; &nbsp; &nbsp; },&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "country_name" =&gt; "Japan",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"postal_code" =&gt; "213-0021",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"country_code2" =&gt; "JP",&nbsp; &nbsp; &nbsp; &nbsp; "continent_code" =&gt; "AS"&nbsp; &nbsp; }}</code></pre></div>
그런데 위경도 정보가 이미 있다면?
<div><pre><code>filter {&nbsp;mutate {&nbsp; remove_field =&gt; ["@timestamp", "@version", "path", "host"]&nbsp;}&nbsp;csv {&nbsp; columns =&gt; ["lat", "lon"]&nbsp; add_field =&gt; {"point" =&gt; "%{lat}"}&nbsp; add_field =&gt; {"point" =&gt; "%{lon}"}&nbsp; convert =&gt; {"point" =&gt; "float"}&nbsp;}}</code></pre></div>
<div><pre><code>{&nbsp; &nbsp; "message" =&gt; "37.48854368,126.9094842\r",&nbsp; &nbsp; &nbsp; &nbsp; "lat" =&gt; "37.48854368",&nbsp; &nbsp; &nbsp; &nbsp; "lon" =&gt; "126.9094842",&nbsp; &nbsp; &nbsp; "point" =&gt; [&nbsp; &nbsp; &nbsp; &nbsp; [0] "37.48854368",&nbsp; &nbsp; &nbsp; &nbsp; [1] "126.9094842"&nbsp; &nbsp; ]}</code></pre></div>

point 필드에 위경도 정보 통합. 그런데 숫자 타입 변환이 안 된다. 옵션 순서 문제인가? 타입 변환 과정만 분리.
<div><pre><code>filter {&nbsp;mutate {&nbsp; remove_field =&gt; ["@timestamp", "@version", "path", "host"]&nbsp;}&nbsp;csv {&nbsp; columns =&gt; ["lat", "lon"]&nbsp; add_field =&gt; {"point" =&gt; "%{lat}"}&nbsp; add_field =&gt; {"point" =&gt; "%{lon}"}&nbsp;}&nbsp;mutate {&nbsp; convert =&gt; {"point" =&gt; "float"}&nbsp;}}</code></pre></div>
<div><pre><code>{&nbsp; &nbsp; "message" =&gt; "37.48854368,126.9094842\r",&nbsp; &nbsp; &nbsp; &nbsp; "lat" =&gt; "37.48854368",&nbsp; &nbsp; &nbsp; &nbsp; "lon" =&gt; "126.9094842",&nbsp; &nbsp; &nbsp; "point" =&gt; [&nbsp; &nbsp; &nbsp; &nbsp; [0] 37.48854368,&nbsp; &nbsp; &nbsp; &nbsp; [1] 126.9094842&nbsp; &nbsp; ]}</code></pre></div>
숫자 타입 변환은 됐는데, 이번엔 인뎅싱이 안 된다. 에러 메시지를 보니 위도 정보값이 이상하다고? 위경도 정보에 순서가 있는 모양.
<pre><code>Could not index event to Elasticsearch. {:status=&gt;400, :action=&gt;["index", {:_id=&gt;nil, :_index=&gt;"geotest", :routing=&gt;nil}, {"message"=&gt;"37.48854368,126.9094842\r", "lat"=&gt;"37.48854368", "lon"=&gt;"126.9094842", "point"=&gt;[37.48854368, 126.9094842]}], :response=&gt;{"index"=&gt;{"status"=&gt;400, "error"=&gt;{"type"=&gt;"document_parsing_exception",&nbsp;"reason"=&gt;"[1:112] failed to parse: illegal latitude value [126.9094842] for point", "caused_by"=&gt;{"type"=&gt;"illegal_argument_exception", "reason"=&gt;"illegal latitude value [126.9094842] for point"}}}}}</code></pre>
경도 정보 먼저 저장.
<div><pre><code>filter {&nbsp;mutate {&nbsp; remove_field =&gt; ["@timestamp", "@version", "path", "host"]&nbsp;}&nbsp;csv {&nbsp; columns =&gt; ["lat", "lon"]&nbsp; add_field =&gt; {"point" =&gt; "%{lon}"}&nbsp; add_field =&gt; {"point" =&gt; "%{lat}"}&nbsp;}&nbsp;mutate {&nbsp; convert =&gt; {"point" =&gt; "float"}&nbsp;}}</code></pre></div>
<div><pre><code>{&nbsp; &nbsp; "message" =&gt; "37.48854368,126.9094842\r",&nbsp; &nbsp; &nbsp; &nbsp; "lat" =&gt; "37.48854368",&nbsp; &nbsp; &nbsp; &nbsp; "lon" =&gt; "126.9094842",&nbsp; &nbsp; &nbsp; "point" =&gt; [&nbsp; &nbsp; &nbsp; &nbsp; [0] 126.9094842,&nbsp; &nbsp; &nbsp; &nbsp; [1] 37.48854368&nbsp; &nbsp; ]}</code></pre></div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjiD9U46o-9KwDmqTxz2z-hqC9c5h5laJmY9chaTnLte4VILQo5pz_CHVL0G7uyc4fd3mjzVdV_l6Cau4BhT6Gr3kkV0j8V3IAt6qaPbFdbswTn8lpiM-3C6Y2J-JqJrKHFWoL5FxH8IVdRcyKUF9D2oEwxwWxNxm4mDK2IeZL9B_LVsopl0wcrXRshNOK/s1642/geo_point.png"><img data-original-height="904" data-original-width="1642" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjiD9U46o-9KwDmqTxz2z-hqC9c5h5laJmY9chaTnLte4VILQo5pz_CHVL0G7uyc4fd3mjzVdV_l6Cau4BhT6Gr3kkV0j8V3IAt6qaPbFdbswTn8lpiM-3C6Y2J-JqJrKHFWoL5FxH8IVdRcyKUF9D2oEwxwWxNxm4mDK2IeZL9B_LVsopl0wcrXRshNOK/s16000/geo_point.png"></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5GmKAdTAdZexY07sqn4CwHabpXmfkg2-8-s5O2kROaapc43zFY8BKaXGSh8z0oomevbJM2KmoMrtzVTC8xPnMszaT-A_h2kw666NNISyuxK9G6nhJe2KTUCYZR0Q8xORuCdYxnnENA3Af60aV4JNTgH6vfB0z5LzbQeNKqXw7oDGt16etS-WgZ70-Jkfe/s1642/cluster_map.png"><img data-original-height="955" data-original-width="1642" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5GmKAdTAdZexY07sqn4CwHabpXmfkg2-8-s5O2kROaapc43zFY8BKaXGSh8z0oomevbJM2KmoMrtzVTC8xPnMszaT-A_h2kw666NNISyuxK9G6nhJe2KTUCYZR0Q8xORuCdYxnnENA3Af60aV4JNTgH6vfB0z5LzbQeNKqXw7oDGt16etS-WgZ70-Jkfe/s16000/cluster_map.png"></a>geo_point 타입 필드는 당연히&nbsp;인덱스 매핑에&nbsp;미리 만들어놔야 한다.<div>관련&nbsp;글<ul><li><a href="http://kangmyounghun.blogspot.kr/2018/02/logstash-geoip.html" target="_blank">Logstash 필터&nbsp;geoip</a></li><li><a href="http://kangmyounghun.blogspot.com/2017/06/elasticsearch-grok.html" target="_blank">Logstash 필터 grok</a></li><li><a href="http://kangmyounghun.blogspot.com/2017/07/logstash-mutate.html" target="_blank">Logstash 필터 mutate</a></li><li><a href="http://kangmyounghun.blogspot.kr/2017/10/logstash-ruby.html" target="_blank">Logstash 필터 ruby</a></li><li><a href="http://kangmyounghun.blogspot.kr/2018/04/logstash-dissect.html" target="_blank">Logstash 필터 dissect</a></li><li><a href="http://kangmyounghun.blogspot.com/2018/09/logstash-kv.html" target="_blank">Logstash 필터&nbsp;kv</a></li><li><a href="https://kangmyounghun.blogspot.com/2019/01/logstash-date.html" target="_blank">Logstash 필터 date</a></li><li><a href="https://kangmyounghun.blogspot.com/2019/02/logstash-translate.html" target="_blank">Logstash 필터 translate</a></li><li><a href="https://kangmyounghun.blogspot.com/2019/08/logstash-drop.html" target="_blank">Logstash 필터 drop</a></li><li><a href="https://kangmyounghun.blogspot.com/2019/11/logstash-useragent.html" target="_blank">Logstash 필터 useragent</a></li><li><a href="https://kangmyounghun.blogspot.com/2020/02/logstash-elapsed.html" target="_blank">Logstash 필터 elapsed</a></li><li><a href="https://kangmyounghun.blogspot.com/2020/04/logstash-fingerprint.html" target="_blank">Logstash 필터 fingerprint</a></li><li><a href="https://kangmyounghun.blogspot.com/2022/12/logstash-csv.html" target="_blank">Logstash 필터 csv</a></li><li><a href="https://kangmyounghun.blogspot.com/2023/11/logstash-dns.html">Logstash 필터 dns</a></li><li><a href="https://kangmyounghun.blogspot.com/2025/01/logstash-split.html">Logstash 필터 split</a></li><li><a href="https://kangmyounghun.blogspot.com/2020/02/logstash-codec-multiline.html" target="_blank">Logstash codec 플러그인 multiline</a></li></ul></div>

</div></div>

케세라세라

geoip 필터를 이용하면 위경도를 포함한 ip 주소의 다양한 지리 정보를 사용할 수 있다. 
{
    "message" => "122.213.34.56",
      "geoip" => {
             "longitude" => 139.6297,
                    "ip" => "122.213.34.56",
         "country_code3" => "JP",
              "latitude" => 35.5843,
              "timezone" => "Asia/Tokyo",
             "city_name" => "Chitose",
           "region_name" => "Kanagawa",
           "region_code" => "14",
              "location" => {
            "lat" => 35.5843,
            "lon" => 139.6297
        },
          "country_name" => "Japan",
           "postal_code" => "213-0021",
         "country_code2" => "JP",
        "continent_code" => "AS"
    }
}
그런데 위경도 정보가 이미 있다면?
filter {
 mutate {
  remove_field => ["@timestamp", "@version", "path", "host"]
 }
 csv {
  columns => ["lat", "lon"]
  add_field => {"point" => "%{lat}"}
  add_field => {"point" => "%{lon}"}
  convert => {"point" => "float"}
 }
}
{
    "message" => "37.48854368,126.9094842\r",
        "lat" => "37.48854368",
        "lon" => "126.9094842",
      "point" => [
        [0] "37.48854368",
        [1] "126.9094842"
    ]
}
point 필드에 위경도 정보 통합. 그런데 숫자 타입 변환이 안 된다. 옵션 순서 문제인가? 타입 변환 과정만 분리.
filter {
 mutate {
  remove_field => ["@timestamp", "@version", "path", "host"]
 }
 csv {
  columns => ["lat", "lon"]
  add_field => {"point" => "%{lat}"}
  add_field => {"point" => "%{lon}"}
 }
 mutate {
  convert => {"point" => "float"}
 }
}
{
    "message" => "37.48854368,126.9094842\r",
        "lat" => "37.48854368",
        "lon" => "126.9094842",
      "point" => [
        [0] 37.48854368,
        [1] 126.9094842
    ]
}
숫자 타입 변환은 됐는데, 이번엔 인뎅싱이 안 된다. 에러 메시지를 보니 위도 정보값이 이상하다고? 위경도 정보에 순서가 있는 모양.
Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"geotest", :routing=>nil}, {"message"=>"37.48854368,126.9094842\r", "lat"=>"37.48854368", "lon"=>"126.9094842", "point"=>[37.48854368, 126.9094842]}], :response=>{"index"=>{"status"=>400, "error"=>{"type"=>"document_parsing_exception", 
"reason"=>"[1:112] failed to parse: illegal latitude value [126.9094842] for point", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"illegal latitude value [126.9094842] for point"}}}}}
경도 정보 먼저 저장.
filter {
 mutate {
  remove_field => ["@timestamp", "@version", "path", "host"]
 }
 csv {
  columns => ["lat", "lon"]
  add_field => {"point" => "%{lon}"}
  add_field => {"point" => "%{lat}"}
 }
 mutate {
  convert => {"point" => "float"}
 }
}
{
    "message" => "37.48854368,126.9094842\r",
        "lat" => "37.48854368",
        "lon" => "126.9094842",
      "point" => [
        [0] 126.9094842,
        [1] 37.48854368
    ]
}
geo_point 타입 필드는 당연히 인덱스 매핑에 미리 만들어놔야 한다.
관련 글
Logstash 필터 geoip
Logstash 필터 grok
Logstash 필터 mutate
Logstash 필터 ruby
Logstash 필터 dissect
Logstash 필터 kv
Logstash 필터 date
Logstash 필터 translate
Logstash 필터 drop
Logstash 필터 useragent
Logstash 필터 elapsed
Logstash 필터 fingerprint
Logstash 필터 csv
Logstash 필터 dns
Logstash 필터 split
Logstash codec 플러그인 multiline

Logstash 필터 geoip - 2nd

댓글