<div id="readability-page-1" class="page"><div id="post-body-1996772452309249723" itemprop="articleBody">
로그스태시의 동적 매핑을 사용하면 키워드 필드의 <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html" target="_blank">ignore_above</a> 값이 256으로 고정된다.<div>
<pre><code>{&nbsp; "test": {&nbsp; &nbsp; "aliases": {},&nbsp; &nbsp; "mappings": {&nbsp; &nbsp; &nbsp; "properties": {&nbsp; &nbsp; &nbsp; &nbsp; "message": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "text",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "fields": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "keyword": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "keyword",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "ignore_above": 256<a name="more"></a>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; }&nbsp; }}</code></pre>
결과적으로 길이를 초과하는 데이터는 키워드 필드에 저장되지 않기 때문에 집계 작업 시 왜곡 발생.</div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-bW-yw4Zy2iN4R2nIMCW0mJ7WsmAitcTvhubWCl7eHFW00GaKgcvKQ01A3rw064vT_dIlKNPrgjd6Mdwbk9hwF5-nWDud23r_fxpagoiY6gZJk1ytqj8sDVfdTteY4sSMlSFGelOnPkzzdLWoFUZxGxdcxpSr9pgiJu5A3N7IoJ3n7-hV-m6ejoEnOhen/s1280/ignore_above_256.png"><img data-original-height="604" data-original-width="1280" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-bW-yw4Zy2iN4R2nIMCW0mJ7WsmAitcTvhubWCl7eHFW00GaKgcvKQ01A3rw064vT_dIlKNPrgjd6Mdwbk9hwF5-nWDud23r_fxpagoiY6gZJk1ytqj8sDVfdTteY4sSMlSFGelOnPkzzdLWoFUZxGxdcxpSr9pgiJu5A3N7IoJ3n7-hV-m6ejoEnOhen/s280/ignore_above_256.png" width="280"></a>
인덱스 템플릿을 만들어서 정적 매핑을 적용해보자. 그런데 ignore_above 수정 옵션이 안 보이네?
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhp1THyDdOa6ZRx7T2i9MuVgswajuGG8xVPJceZU7L6Mo4IdM2Qpxa1XC1wQsBCSr9reXROQJ6cd0N8G58H1sm8Ve0DVI96gOsun7xK40nKXYfwcle9vl5vd4k4B3CL0jSgktBB7sPPSZkYoqzYTK9_-x3wy3Qh2qXpnuWOiXrwB_asCmYbMlfR1EmB-t0/s1280/index_template.png"><img data-original-height="718" data-original-width="1280" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhp1THyDdOa6ZRx7T2i9MuVgswajuGG8xVPJceZU7L6Mo4IdM2Qpxa1XC1wQsBCSr9reXROQJ6cd0N8G58H1sm8Ve0DVI96gOsun7xK40nKXYfwcle9vl5vd4k4B3CL0jSgktBB7sPPSZkYoqzYTK9_-x3wy3Qh2qXpnuWOiXrwB_asCmYbMlfR1EmB-t0/s280/index_template.png" width="280"></a>
만들어진 템플릿 내역을 살펴보자. ignore_above 옵션이 아예 없다. 정적 매핑일 때는 원래 사용되지 않는 옵션인가? 길이 제한이 없다는 뜻?
<pre><code>GET _index_template/test</code></pre>
<pre><code><div>{&nbsp; "index_templates": [&nbsp; &nbsp; {&nbsp; &nbsp; &nbsp; "name": "test",&nbsp; &nbsp; &nbsp; "index_template": {&nbsp; &nbsp; &nbsp; &nbsp; "index_patterns": [&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "test*"&nbsp; &nbsp; &nbsp; &nbsp; ],&nbsp; &nbsp; &nbsp; &nbsp; "template": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "mappings": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "properties": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "message": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "text",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "fields": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "keyword": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "keyword"&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; },&nbsp; &nbsp; &nbsp; &nbsp; "allow_auto_create": false&nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; }&nbsp; ]}</div></code></pre>
<div>왠지 될 것 같은 부푼 기대를 안고 다시 테스트. <strike>어림 없쥬</strike></div>
<pre><code><div>[2024-04-01T16:17:53,753][INFO ][logstash.agent&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;] Pipelines running {:count=&gt;1, :running_pipelines=&gt;[:main], :non_running_pipelines=&gt;[]}{&nbsp; &nbsp; "message" =&gt; "abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz\r"}{&nbsp; &nbsp; "message" =&gt; "abcdefghijklmnopqrstuvwxyz\r"}[2024-04-01T16:17:53,983][WARN ][logstash.outputs.elasticsearch][main][cb7112c4be65177af0c217a3a208d8b35fab773320b508b4c53a66a83af5889d] Could not index event to Elasticsearch. {:status=&gt;404, :action=&gt;["index", {:_id=&gt;nil, :_index=&gt;"test", :routing=&gt;nil}, {"message"=&gt;"abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz\r"}], :response=&gt;{"index"=&gt;{"status"=&gt;404, "error"=&gt;{"type"=&gt;"index_not_found_exception", "reason"=&gt;"no such index [composable template [test*] forbids index auto creation]", "index_uuid"=&gt;"_na_", "index"=&gt;"composable template [test*] forbids index auto creation"}}}}[2024-04-01T16:17:53,985][WARN ][logstash.outputs.elasticsearch][main][cb7112c4be65177af0c217a3a208d8b35fab773320b508b4c53a66a83af5889d] Could not index event to Elasticsearch. {:status=&gt;404, :action=&gt;["index", {:_id=&gt;nil, :_index=&gt;"test", :routing=&gt;nil}, {"message"=&gt;"abcdefghijklmnopqrstuvwxyz\r"}], :response=&gt;{"index"=&gt;{"status"=&gt;404, "error"=&gt;{"type"=&gt;"index_not_found_exception", "reason"=&gt;"no such index [composable template [test*] forbids index auto creation]", "index_uuid"=&gt;"_na_", "index"=&gt;"composable template [test*] forbids index auto creation"}}}}</div></code></pre>

<pre><code><div>PUT _index_template/test{&nbsp; "index_patterns": ["test*"],&nbsp; "template": {&nbsp; &nbsp; "mappings": {&nbsp; &nbsp; &nbsp; "properties": {&nbsp; &nbsp; &nbsp; &nbsp; "message": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "text",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "fields": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "keyword": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "keyword"&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; }&nbsp; },&nbsp; "allow_auto_create": true}</div></code></pre>
<pre><code>GET _index_template/test</code></pre>
<pre><code><div>{&nbsp; "index_templates": [&nbsp; &nbsp; {&nbsp; &nbsp; &nbsp; "name": "test",&nbsp; &nbsp; &nbsp; "index_template": {&nbsp; &nbsp; &nbsp; &nbsp; "index_patterns": [&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "test*"&nbsp; &nbsp; &nbsp; &nbsp; ],&nbsp; &nbsp; &nbsp; &nbsp; "template": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "mappings": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "properties": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "message": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "text",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "fields": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "keyword": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "keyword"&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; },&nbsp; &nbsp; &nbsp; &nbsp; "allow_auto_create": true&nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; }&nbsp; ]}</div></code></pre>
이제 됨.
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVpCihHngiJrThhweO7xkt3h9e1DO3glw-O8G58rrzLT2UEWSllAnotjNMynqVMmpsR1zrk_GJDgJl4gV2BrcnTrK_x4yq0zi7m7JOsaaFAm_k_Udcpxmp67XLN6PVOEDvLjtH9HYvRSpQU8QnkD4ER7xpCy37dsHt4ra51obnxCrs6ie80eJuYwUl2jL0/s1280/no_ignore_above.png"><img data-original-height="604" data-original-width="1280" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVpCihHngiJrThhweO7xkt3h9e1DO3glw-O8G58rrzLT2UEWSllAnotjNMynqVMmpsR1zrk_GJDgJl4gV2BrcnTrK_x4yq0zi7m7JOsaaFAm_k_Udcpxmp67XLN6PVOEDvLjtH9HYvRSpQU8QnkD4ER7xpCy37dsHt4ra51obnxCrs6ie80eJuYwUl2jL0/s280/no_ignore_above.png" width="280"></a>
참고로 키바나 UI 말고 REST API로 생성하면&nbsp;allow_auto_create&nbsp;옵션이 적용되지 않는다. 키바나 UI에 문제가 좀 있는 모양.
<pre><code><div>PUT _index_template/test{&nbsp; "index_patterns": ["test*"],&nbsp; "template": {&nbsp; &nbsp; "mappings": {&nbsp; &nbsp; &nbsp; "properties": {&nbsp; &nbsp; &nbsp; &nbsp; "message": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "text",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "fields": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "keyword": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "keyword"&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; }&nbsp; }}</div></code></pre>
<pre><code>GET _index_template/test</code></pre>
<pre><code><div>{&nbsp; "index_templates": [&nbsp; &nbsp; {&nbsp; &nbsp; &nbsp; "name": "test",&nbsp; &nbsp; &nbsp; "index_template": {&nbsp; &nbsp; &nbsp; &nbsp; "index_patterns": [&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "test*"&nbsp; &nbsp; &nbsp; &nbsp; ],&nbsp; &nbsp; &nbsp; &nbsp; "template": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "mappings": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "properties": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "message": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "text",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "fields": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "keyword": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "type": "keyword"&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; }&nbsp; ]}</div></code></pre>


</div></div>

케세라세라

로그스태시의 동적 매핑을 사용하면 키워드 필드의 ignore_above 값이 256으로 고정된다.
{
  "test": {
    "aliases": {},
    "mappings": {
      "properties": {
        "message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}
결과적으로 길이를 초과하는 데이터는 키워드 필드에 저장되지 않기 때문에 집계 작업 시 왜곡 발생.
인덱스 템플릿을 만들어서 정적 매핑을 적용해보자. 그런데 ignore_above 수정 옵션이 안 보이네?
만들어진 템플릿 내역을 살펴보자. ignore_above 옵션이 아예 없다. 정적 매핑일 때는 원래 사용되지 않는 옵션인가? 길이 제한이 없다는 뜻?
GET _index_template/test
{
  "index_templates": [
    {
      "name": "test",
      "index_template": {
        "index_patterns": [
          "test*"
        ],
        "template": {
          "mappings": {
            "properties": {
              "message": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                }
              }
            }
          }
        },
        "allow_auto_create": false
      }
    }
  ]
}
왠지 될 것 같은 부푼 기대를 안고 다시 테스트. 어림 없쥬
[2024-04-01T16:17:53,753][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
    "message" => "abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz\r"
}
{
    "message" => "abcdefghijklmnopqrstuvwxyz\r"
}
[2024-04-01T16:17:53,983][WARN ][logstash.outputs.elasticsearch][main][cb7112c4be65177af0c217a3a208d8b35fab773320b508b4c53a66a83af5889d] Could not index event to Elasticsearch. {:status=>404, :action=>["index", {:_id=>nil, :_index=>"test", :routing=>nil}, {"message"=>"abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz\r"}], :response=>{"index"=>{"status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index [composable template [test*] forbids index auto creation]", "index_uuid"=>"_na_", "index"=>"composable template [test*] forbids index auto creation"}}}}
[2024-04-01T16:17:53,985][WARN ][logstash.outputs.elasticsearch][main][cb7112c4be65177af0c217a3a208d8b35fab773320b508b4c53a66a83af5889d] Could not index event to Elasticsearch. {:status=>404, :action=>["index", {:_id=>nil, :_index=>"test", :routing=>nil}, {"message"=>"abcdefghijklmnopqrstuvwxyz\r"}], :response=>{"index"=>{"status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index [composable template [test*] forbids index auto creation]", "index_uuid"=>"_na_", "index"=>"composable template [test*] forbids index auto creation"}}}}
버전 7에서 잘 되던 게 왜 버전 8에서는 안 될까? 아무래도 allow_auto_create 옵션이 수상하다. 변경.
PUT _index_template/test
{
  "index_patterns": ["test*"],
  "template": {
    "mappings": {
      "properties": {
        "message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        }
      }
    }
  },
  "allow_auto_create": true
}
GET _index_template/test
{
  "index_templates": [
    {
      "name": "test",
      "index_template": {
        "index_patterns": [
          "test*"
        ],
        "template": {
          "mappings": {
            "properties": {
              "message": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                }
              }
            }
          }
        },
        "allow_auto_create": true
      }
    }
  ]
}
이제 됨.
참고로 키바나 UI 말고 REST API로 생성하면 allow_auto_create 옵션이 적용되지 않는다. 키바나 UI에 문제가 좀 있는 모양.
PUT _index_template/test
{
  "index_patterns": ["test*"],
  "template": {
    "mappings": {
      "properties": {
        "message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}
GET _index_template/test
{
  "index_templates": [
    {
      "name": "test",
      "index_template": {
        "index_patterns": [
          "test*"
        ],
        "template": {
          "mappings": {
            "properties": {
              "message": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                }
              }
            }
          }
        }
      }
    }
  ]
}
관련 글
엘라스틱 Runtime field - 7th

composable template forbids index auto creation

댓글