<div id="readability-page-1" class="page"><div id="post-body-2338678447716787599" itemprop="articleBody">
filebeat를 이용한 웹로그 전처리.
<div><pre><code>processors:&nbsp; - include_fields:&nbsp; &nbsp; &nbsp; fields: "message"&nbsp; - if.contains:&nbsp; &nbsp; &nbsp; message: "?"&nbsp; &nbsp; then:&nbsp; &nbsp; &nbsp; - dissect:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url}?%{param} %{}" %{status} %{}'&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; field: "message"&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; target_prefix: ""&nbsp;&nbsp; &nbsp; else:&nbsp; &nbsp; &nbsp; - dissect:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url} %{}" %{status} %{}'&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; field: "message"&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; target_prefix: ""&nbsp;</code></pre></div>
<div><pre><code>{ "@timestamp": "2025-01-13T09:07:47.134Z",
 "@metadata": {
 "beat": "filebeat",
 "type": "_doc",
 "version": "8.17.0"
 },
 "status": "200",
 "timestamp": "12/Oct/2015:02:42:00",
 "uri": "/bbs/view.php?board_id=kor%5Fmedia&amp;gul_no=1106&amp;idx=17&amp;m=4&amp;upage=25&amp;tpage=&amp;PAGE=4",
 "clientip": "192.168.71.168",
 "url": "/bbs/view.php",
 "param": "board_id=kor%5Fmedia&amp;gul_no=1106&amp;idx=17&amp;m=4&amp;upage=25&amp;tpage=&amp;PAGE=4",
 "message": "192.168.71.168 - - [12/Oct/2015:02:42:00 +0900] \"GET /bbs/view.php?board_id=kor%5Fmedia&amp;gul_no=1106&amp;idx=17&amp;m=4&amp;upage=25&amp;tpage=&amp;PAGE=4 HTTP/1.1\" 200 37727",
 "method": "GET"
}
{
 "@timestamp": "2025-01-13T09:07:47.134Z",
 "@metadata": {
 "beat": "filebeat",
 "type": "_doc",
 "version": "8.17.0"
 },
 "status": "200",
 "clientip": "192.168.71.168",
 "uri": "/bbs/view.html",
 "url": "/bbs/view.html",
 "message": "192.168.71.168 - - [12/Oct/2015:02:42:00 +0900] \"GET /bbs/view.html HTTP/1.1\" 200 37727",
 "timestamp": "12/Oct/2015:02:42:00",
 "method": "GET"
}</code></pre></div>

<div><pre><code>processors:&nbsp; - include_fields:&nbsp; &nbsp; &nbsp; fields: "message"&nbsp; - dissect:&nbsp; &nbsp; &nbsp;tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{uri} %{}" %{status} %{}'&nbsp; &nbsp; &nbsp;field: "message"&nbsp; &nbsp; &nbsp;target_prefix: ""&nbsp; - script:&nbsp; &nbsp; &nbsp; lang: javascript&nbsp; &nbsp; &nbsp; source: &gt;&nbsp; &nbsp; &nbsp; &nbsp; function process(evt) {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; if (evt.Get("uri").match("\\?")) {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; var result = evt.Get("uri").split("?");&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; evt.Put("url", result[0]);&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; evt.Put("param", result[1]);&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; } else {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; evt.Put("url", evt.Get("uri"))&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; - if.contains:&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; message: "?"&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; then:&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; - dissect:&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url}?%{param} %{}" %{status} %{}'&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; field: "message"&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; target_prefix: ""&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; else:&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; - dissect:&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url} %{}" %{status} %{}'&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; field: "message"&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; target_prefix: ""&nbsp;</code></pre></div>

그런데 안 되네? 구문은 맞는데?
<div><pre><code>Exiting: error initializing processors: SyntaxError: inline.js: Line 9:3 Unexpected token ILLEGAL (and 14 more errors)PS D:\ELK\filebeat-8.17.0-windows-x86_64&gt;</code></pre></div>
끙끙대다 첫 번째 주석 라인의 들여쓰기를 제거하니 정상 실행됨. 들여쓰기 간격이 같으면 주석 라인도 이어지는 코드로 인식한다는 얘기.
<div><pre><code>processors:&nbsp; - include_fields:&nbsp; &nbsp; &nbsp; fields: "message"&nbsp; - dissect:&nbsp; &nbsp; &nbsp;tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{uri} %{}" %{status} %{}'&nbsp; &nbsp; &nbsp;field: "message"&nbsp; &nbsp; &nbsp;target_prefix: ""&nbsp; - script:&nbsp; &nbsp; &nbsp; lang: javascript&nbsp; &nbsp; &nbsp; source: &gt;&nbsp; &nbsp; &nbsp; &nbsp; function process(evt) {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; if (evt.Get("uri").match("\\?")) {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; var result = evt.Get("uri").split("?");&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; evt.Put("url", result[0]);&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; evt.Put("param", result[1]);&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; } else {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; evt.Put("url", evt.Get("uri"))&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; }#&nbsp; - if.contains:&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; message: "?"&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; then:&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; - dissect:&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url}?%{param} %{}" %{status} %{}'&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; field: "message"&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; target_prefix: ""&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; else:&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; - dissect:&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url} %{}" %{status} %{}'&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; field: "message"&nbsp; &nbsp; &nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; target_prefix: ""&nbsp;</code></pre></div>
윈도우용 VIM이 주편집기인데 yml 파일 편집을 하다보면 들여쓰기가 자동으로 적용될 때가 있다. (리눅스 VIM은 안 그런다)&nbsp;그동안은 그냥 참고 썼는데 이번에 제대로 빡쳐서 yml 자동 들여쓰기 해제 기능 폭풍 검색. <a href="https://github.com/tpope/vim-sensible/issues/188" target="_blank">_vimrc에 다음 설정을 추가하면 해결</a>된다.
<pre><code>autocmd FileType yaml setlocal ts=2 sts=2 sw=2 expandtab autoindent smartindent indentexpr=</code></pre>
역시 빡침 정도가 커야 문제 해결 가능성도 커진다.<div>관련 글<div><ul><li><a href="https://kangmyounghun.blogspot.com/2024/04/filebeat-processors-4rd.html">Filebeat의 processors - 4th</a></li><li><a href="https://kangmyounghun.blogspot.com/2023/09/convert-ip-to-decimal.html">convert ip to decimal</a></li><li><a href="https://kangmyounghun.blogspot.com/2021/10/key-value.html" target="">엘라스틱의 key-value 처리</a></li><li><a href="https://kangmyounghun.blogspot.com/2021/06/processors-registereddomain.html" target="">도메인 정규화 Processor: registered_domain</a></li></ul></div></div>

</div></div>

케세라세라

filebeat를 이용한 웹로그 전처리.
processors:
  - include_fields:
      fields: "message"
  - if.contains:
      message: "?"
    then:
      - dissect:
          tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url}?%{param} %{}" %{status} %{}'
          field: "message"
          target_prefix: "" 
    else:
      - dissect:
          tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url} %{}" %{status} %{}'
          field: "message"
          target_prefix: "" 
{
  "@timestamp": "2025-01-13T09:07:47.134Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.17.0"
  },
  "status": "200",
  "timestamp": "12/Oct/2015:02:42:00",
  "uri": "/bbs/view.php?board_id=kor%5Fmedia&gul_no=1106&idx=17&m=4&upage=25&tpage=&PAGE=4",
  "clientip": "192.168.71.168",
  "url": "/bbs/view.php",
  "param": "board_id=kor%5Fmedia&gul_no=1106&idx=17&m=4&upage=25&tpage=&PAGE=4",
  "message": "192.168.71.168 - - [12/Oct/2015:02:42:00 +0900] \"GET /bbs/view.php?board_id=kor%5Fmedia&gul_no=1106&idx=17&m=4&upage=25&tpage=&PAGE=4 HTTP/1.1\" 200 37727",
  "method": "GET"
}
{
  "@timestamp": "2025-01-13T09:07:47.134Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.17.0"
  },
  "status": "200",
  "clientip": "192.168.71.168",
  "uri": "/bbs/view.html",
  "url": "/bbs/view.html",
  "message": "192.168.71.168 - - [12/Oct/2015:02:42:00 +0900] \"GET /bbs/view.html HTTP/1.1\" 200 37727",
  "timestamp": "12/Oct/2015:02:42:00",
  "method": "GET"
}
script 프로세서를 사용하면 좀 더 우아(?)한 코드 작성도 가능.
processors:
  - include_fields:
      fields: "message"
  - dissect:
     tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{uri} %{}" %{status} %{}'
     field: "message"
     target_prefix: ""
  - script:
      lang: javascript
      source: >
        function process(evt) {
          if (evt.Get("uri").match("\\?")) {
            var result = evt.Get("uri").split("?");
            evt.Put("url", result[0]);
            evt.Put("param", result[1]);
          } else {
              evt.Put("url", evt.Get("uri"))
          }
        }
        #  - if.contains:
        #      message: "?"
        #    then:
        #      - dissect:
        #          tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url}?%{param} %{}" %{status} %{}'
        #          field: "message"
        #          target_prefix: "" 
        #    else:
        #      - dissect:
        #          tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url} %{}" %{status} %{}'
        #          field: "message"
        #          target_prefix: "" 
그런데 안 되네? 구문은 맞는데?
Exiting: error initializing processors: SyntaxError: inline.js: Line 9:3 Unexpected token ILLEGAL (and 14 more errors)
PS D:\ELK\filebeat-8.17.0-windows-x86_64>
끙끙대다 첫 번째 주석 라인의 들여쓰기를 제거하니 정상 실행됨. 들여쓰기 간격이 같으면 주석 라인도 이어지는 코드로 인식한다는 얘기.
processors:
  - include_fields:
      fields: "message"
  - dissect:
     tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{uri} %{}" %{status} %{}'
     field: "message"
     target_prefix: ""
  - script:
      lang: javascript
      source: >
        function process(evt) {
          if (evt.Get("uri").match("\\?")) {
            var result = evt.Get("uri").split("?");
            evt.Put("url", result[0]);
            evt.Put("param", result[1]);
          } else {
              evt.Put("url", evt.Get("uri"))
          }
        }
#  - if.contains:
        #      message: "?"
        #    then:
        #      - dissect:
        #          tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url}?%{param} %{}" %{status} %{}'
        #          field: "message"
        #          target_prefix: "" 
        #    else:
        #      - dissect:
        #          tokenizer: '%{clientip} %{} %{} [%{timestamp} %{}] "%{method} %{url} %{}" %{status} %{}'
        #          field: "message"
        #          target_prefix: "" 
윈도우용 VIM이 주편집기인데 yml 파일 편집을 하다보면 들여쓰기가 자동으로 적용될 때가 있다. (리눅스 VIM은 안 그런다) 그동안은 그냥 참고 썼는데 이번에 제대로 빡쳐서 yml 자동 들여쓰기 해제 기능 폭풍 검색. _vimrc에 다음 설정을 추가하면 해결된다.
autocmd FileType yaml setlocal ts=2 sts=2 sw=2 expandtab autoindent smartindent indentexpr=
역시 빡침 정도가 커야 문제 해결 가능성도 커진다.
관련 글
Filebeat의 processors - 4th
convert ip to decimal
엘라스틱의 key-value 처리
도메인 정규화 Processor: registered_domain

Filebeat의 processors - 5th

댓글