<div id="readability-page-1" class="page"><div id="post-body-5823819460594179656" itemprop="articleBody">
프로토콜별로 불필요한 필드는 지우고 싶다.
<div><pre><code>processors:&nbsp; - include_fields:&nbsp; &nbsp; &nbsp; fields: "message"&nbsp; - dissect:&nbsp; &nbsp; &nbsp; tokenizer: "%{timestamp} %{+timestamp} %{action} %{protocol} %{sip} %{dip} %{sport} %{dport} %{tcpflags} %{icmptype} %{icmpcode}"&nbsp; &nbsp; &nbsp; target_prefix: ""</code></pre></div>
<div><pre><code>{&nbsp; "@timestamp": "2024-03-14T10:46:36.387Z",&nbsp; "@metadata": {&nbsp; &nbsp; "beat": "filebeat",&nbsp; &nbsp; "type": "_doc",&nbsp; &nbsp; "version": "8.12.0"&nbsp; },&nbsp; "icmpcode": "-",&nbsp; "action": "ALLOW",&nbsp; "icmptype": "-",&nbsp; "message": "2024-03-05 00:03:04 ALLOW TCP 192.168.31.177 192.168.31.1 3638 8089 24 - -",&nbsp; "dport": "8089",&nbsp; "dip": "192.168.31.1",&nbsp; "protocol": "TCP",&nbsp; "tcpflags": "24",&nbsp; "sip": "192.168.31.177",&nbsp; "timestamp": "2024-03-05 00:03:04",&nbsp; "sport": "3638"}{&nbsp; "@timestamp": "2024-03-14T10:46:36.387Z",&nbsp; "@metadata": {&nbsp; &nbsp; "beat": "filebeat",&nbsp; &nbsp; "type": "_doc",&nbsp; &nbsp; "version": "8.12.0"&nbsp; },&nbsp; "sip": "192.168.31.177",&nbsp; "icmptype": "0",&nbsp; "action": "ALLOW",&nbsp; "protocol": "ICMP",&nbsp; "timestamp": "2024-03-05 00:03:04",&nbsp; "sport": "-",&nbsp; "message": "2024-03-05 00:03:04 ALLOW ICMP 192.168.31.177 192.168.31.1 - - - 0 8",&nbsp; "dport": "-",&nbsp; "icmpcode": "8",&nbsp; "dip": "192.168.31.1",&nbsp; "tcpflags": "-"}{&nbsp; "@timestamp": "2024-03-14T10:46:36.387Z",&nbsp; "@metadata": {&nbsp; &nbsp; "beat": "filebeat",&nbsp; &nbsp; "type": "_doc",&nbsp; &nbsp; "version": "8.12.0"&nbsp; },&nbsp; "timestamp": "2024-03-05 00:03:05",&nbsp; "protocol": "UDP",&nbsp; "icmpcode": "-",&nbsp; "message": "2024-03-05 00:03:05 ALLOW UDP 192.168.31.177 192.168.31.1 50958 53 - - -",&nbsp; "dport": "53",&nbsp; "sport": "50958",&nbsp; "tcpflags": "-",&nbsp; "icmptype": "-",&nbsp; "sip": "192.168.31.177",&nbsp; "dip": "192.168.31.1",&nbsp; "action": "ALLOW"}</code></pre></div>
프로토콜별로 다른 처리를 해주면 간단한데,
<pre><code>if...tcp...else if...udp...else...icmp...</code></pre>
else if 구문을 지원하지 않네? 이럴 때 중첩 if-then-else 구문이 대안이 될 수 있다.
<pre><code><div>processors:&nbsp; - include_fields:&nbsp; &nbsp; &nbsp; fields: "message"&nbsp; - dissect:&nbsp; &nbsp; &nbsp; tokenizer: "%{timestamp} %{+timestamp} %{action} %{protocol} %{sip} %{dip} %{sport} %{dport} %{tcpflags} %{icmptype} %{icmpcode}"&nbsp; &nbsp; &nbsp; target_prefix: ""</div><div>&nbsp; - if:&nbsp; &nbsp; &nbsp; equals.protocol: "TCP"&nbsp; &nbsp; then:&nbsp; &nbsp; &nbsp; - drop_fields:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; fields: ["icmptype", "icmpcode"]&nbsp; &nbsp; else:&nbsp; &nbsp; &nbsp; if:&nbsp; &nbsp; &nbsp; &nbsp; equals.protocol: "UDP"&nbsp; &nbsp; &nbsp; then:&nbsp; &nbsp; &nbsp; &nbsp; - drop_fields:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; fields: ["tcpflags", "icmptype", "icmpcode"]&nbsp; &nbsp; &nbsp; else:&nbsp; &nbsp; &nbsp; &nbsp; - drop_fields:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; fields: ["sport", "dport", "tcpflags"]</div></code></pre>
<div><pre><code>{&nbsp; "@timestamp": "2024-03-14T10:50:02.502Z",&nbsp; "@metadata": {&nbsp; &nbsp; "beat": "filebeat",&nbsp; &nbsp; "type": "_doc",&nbsp; &nbsp; "version": "8.12.0"&nbsp; },&nbsp; "message": "2024-03-05 00:03:04 ALLOW TCP 192.168.31.177 192.168.31.1 3638 8089 24 - -",&nbsp; "sip": "192.168.31.177",&nbsp; "sport": "3638",&nbsp; "timestamp": "2024-03-05 00:03:04",&nbsp; "protocol": "TCP",&nbsp; "dip": "192.168.31.1",&nbsp; "dport": "8089",&nbsp; "action": "ALLOW",&nbsp; "tcpflags": "24"}{&nbsp; "@timestamp": "2024-03-14T10:50:02.502Z",&nbsp; "@metadata": {&nbsp; &nbsp; "beat": "filebeat",&nbsp; &nbsp; "type": "_doc",&nbsp; &nbsp; "version": "8.12.0"&nbsp; },&nbsp; "icmpcode": "8",&nbsp; "timestamp": "2024-03-05 00:03:04",&nbsp; "action": "ALLOW",&nbsp; "icmptype": "0",&nbsp; "dip": "192.168.31.1",&nbsp; "protocol": "ICMP",&nbsp; "sip": "192.168.31.177",&nbsp; "message": "2024-03-05 00:03:04 ALLOW ICMP 192.168.31.177 192.168.31.1 - - - 0 8"}{&nbsp; "@timestamp": "2024-03-14T10:50:02.502Z",&nbsp; "@metadata": {&nbsp; &nbsp; "beat": "filebeat",&nbsp; &nbsp; "type": "_doc",&nbsp; &nbsp; "version": "8.12.0"&nbsp; },&nbsp; "message": "2024-03-05 00:03:05 ALLOW UDP 192.168.31.177 192.168.31.1 50958 53 - - -",&nbsp; "sip": "192.168.31.177",&nbsp; "protocol": "UDP",&nbsp; "timestamp": "2024-03-05 00:03:05",&nbsp; "action": "ALLOW",&nbsp; "dip": "192.168.31.1",&nbsp; "sport": "50958",&nbsp; "dport": "53"}</code></pre></div>
그냥 필요 없는 필드 다 지우고 시작하는 게 나을래나?
<pre><code><div>&nbsp; - script:&nbsp; &nbsp; &nbsp; lang: javascript&nbsp; &nbsp; &nbsp; source: &gt;&nbsp; &nbsp; &nbsp; &nbsp; function process(event) {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; var list = event.Get();&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; for (var i in list) {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; if (event.Get(i) == "-") {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; event.Delete(i);&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; }</div></code></pre>

<div>관련 글<div><ul><li><a href="https://kangmyounghun.blogspot.com/2023/04/filebeat-processors-2nd.html">Filebeat의 processors - 2nd</a></li><li><a href="https://kangmyounghun.blogspot.com/2023/09/convert-ip-to-decimal.html">convert ip to decimal</a></li><li><a href="https://kangmyounghun.blogspot.com/2021/10/key-value.html" target="_blank">엘라스틱의 key-value 처리</a></li><li><a href="https://kangmyounghun.blogspot.com/2021/06/processors-registereddomain.html" target="_blank">도메인 정규화 Processor: registered_domain</a></li></ul></div></div>

</div></div>

케세라세라

프로토콜별로 불필요한 필드는 지우고 싶다.
processors:
  - include_fields:
      fields: "message"
  - dissect:
      tokenizer: "%{timestamp} %{+timestamp} %{action} %{protocol} %{sip} %{dip} %{sport} %{dport} %{tcpflags} %{icmptype} %{icmpcode}"
      target_prefix: ""
{
  "@timestamp": "2024-03-14T10:46:36.387Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.12.0"
  },
  "icmpcode": "-",
  "action": "ALLOW",
  "icmptype": "-",
  "message": "2024-03-05 00:03:04 ALLOW TCP 192.168.31.177 192.168.31.1 3638 8089 24 - -",
  "dport": "8089",
  "dip": "192.168.31.1",
  "protocol": "TCP",
  "tcpflags": "24",
  "sip": "192.168.31.177",
  "timestamp": "2024-03-05 00:03:04",
  "sport": "3638"
}
{
  "@timestamp": "2024-03-14T10:46:36.387Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.12.0"
  },
  "sip": "192.168.31.177",
  "icmptype": "0",
  "action": "ALLOW",
  "protocol": "ICMP",
  "timestamp": "2024-03-05 00:03:04",
  "sport": "-",
  "message": "2024-03-05 00:03:04 ALLOW ICMP 192.168.31.177 192.168.31.1 - - - 0 8",
  "dport": "-",
  "icmpcode": "8",
  "dip": "192.168.31.1",
  "tcpflags": "-"
}
{
  "@timestamp": "2024-03-14T10:46:36.387Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.12.0"
  },
  "timestamp": "2024-03-05 00:03:05",
  "protocol": "UDP",
  "icmpcode": "-",
  "message": "2024-03-05 00:03:05 ALLOW UDP 192.168.31.177 192.168.31.1 50958 53 - - -",
  "dport": "53",
  "sport": "50958",
  "tcpflags": "-",
  "icmptype": "-",
  "sip": "192.168.31.177",
  "dip": "192.168.31.1",
  "action": "ALLOW"
}
프로토콜별로 다른 처리를 해주면 간단한데,
if
...tcp...
else if
...udp...
else
...icmp...
else if 구문을 지원하지 않네? 이럴 때 중첩 if-then-else 구문이 대안이 될 수 있다.
processors:
  - include_fields:
      fields: "message"
  - dissect:
      tokenizer: "%{timestamp} %{+timestamp} %{action} %{protocol} %{sip} %{dip} %{sport} %{dport} %{tcpflags} %{icmptype} %{icmpcode}"
      target_prefix: ""
  - if:
      equals.protocol: "TCP"
    then:
      - drop_fields:
          fields: ["icmptype", "icmpcode"]
    else:
      if:
        equals.protocol: "UDP"
      then:
        - drop_fields:
            fields: ["tcpflags", "icmptype", "icmpcode"]
      else:
        - drop_fields:
            fields: ["sport", "dport", "tcpflags"]
{
  "@timestamp": "2024-03-14T10:50:02.502Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.12.0"
  },
  "message": "2024-03-05 00:03:04 ALLOW TCP 192.168.31.177 192.168.31.1 3638 8089 24 - -",
  "sip": "192.168.31.177",
  "sport": "3638",
  "timestamp": "2024-03-05 00:03:04",
  "protocol": "TCP",
  "dip": "192.168.31.1",
  "dport": "8089",
  "action": "ALLOW",
  "tcpflags": "24"
}
{
  "@timestamp": "2024-03-14T10:50:02.502Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.12.0"
  },
  "icmpcode": "8",
  "timestamp": "2024-03-05 00:03:04",
  "action": "ALLOW",
  "icmptype": "0",
  "dip": "192.168.31.1",
  "protocol": "ICMP",
  "sip": "192.168.31.177",
  "message": "2024-03-05 00:03:04 ALLOW ICMP 192.168.31.177 192.168.31.1 - - - 0 8"
}
{
  "@timestamp": "2024-03-14T10:50:02.502Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.12.0"
  },
  "message": "2024-03-05 00:03:05 ALLOW UDP 192.168.31.177 192.168.31.1 50958 53 - - -",
  "sip": "192.168.31.177",
  "protocol": "UDP",
  "timestamp": "2024-03-05 00:03:05",
  "action": "ALLOW",
  "dip": "192.168.31.1",
  "sport": "50958",
  "dport": "53"
}
그냥 필요 없는 필드 다 지우고 시작하는 게 나을래나?
  - script:
      lang: javascript
      source: >
        function process(event) {
          var list = event.Get();
          for (var i in list) {
            if (event.Get(i) == "-") {
              event.Delete(i);
            }
          }
        }
관련 글
Filebeat의 processors - 2nd
convert ip to decimal
엘라스틱의 key-value 처리
도메인 정규화 Processor: registered_domain

Filebeat의 processors - 3rd

댓글