<div id="readability-page-1" class="page"><div>
                <div id="SE-289cbf4b-48c3-4ddf-a7ec-b553439feabf">
                                    <p id="SE-12b522dc-98d5-4e67-89e2-40019d3b2104"><span id="SE-6b781a19-0f9a-464e-bfec-2803dfd5095c">안녕하세요 멍개입니다.</span></p>
<p id="SE-2fcc7ae8-5db0-4c30-a1df-cfb70b2a0cbe"><span id="SE-e5e77cae-6a5b-4e05-849e-5fb468114dc5">​</span></p>
<p id="SE-92ac4815-c337-4c54-a663-b0582f7294ec"><span id="SE-ca2392dd-7f19-47af-a7bc-48668687685c">전에 lm studio와 ollama를 이용하여 로컬에서 LLM을 구동하는 방법을 다루어 보았습니다.</span></p>
<p id="SE-48a0266d-40d1-451f-9ed4-49dc6ef5ec32"><span id="SE-627d8ff9-fbae-4b3e-8b9f-5c226bd6ac13"><a href="https://blog.naver.com/pjt3591oo/223939267795" data-linktype="text" data-linkdata="{&quot;id&quot;: &quot;SE-627d8ff9-fbae-4b3e-8b9f-5c226bd6ac13&quot;, &quot;link&quot;: &quot;https://blog.naver.com/pjt3591oo/223939267795&quot;}" target="_blank">https://blog.naver.com/pjt3591oo/223939267795</a></span></p>

                            </div>                
                                
                <div id="SE-40646506-90a9-451b-83b4-ebb7d97bfc69">
                                    <p id="SE-51b46c41-f785-44fd-b020-b0e87d82d040"><span id="SE-8d1b77a1-3f85-42e5-a45e-f21c92becbfe">이번 포스트에서는 두 도구가 내부적으로 사용하는 llama.cpp를 다뤄봅니다. llama.cpp는 gguf를 서빙하는 도구입니다.</span></p>
<p id="SE-5bbf4e00-1d4b-4259-8c4e-3c4342f17165"><span id="SE-b837ff42-46e6-437c-b3c8-d634d2cd88cb">​</span></p>
<p id="SE-a8fade8c-8f62-4333-ad55-8a34044bc2e9"><span id="SE-0cab4e56-971e-40b4-80ed-9dd70df18d78">llama.cpp는 GGUF 포맷 모델을 구동하는 핵심 추론 엔진이며, Ollama와 LM Studio(LM Studio는 GGUF 이외에 MLX 포맷도 지원함.) 같은 툴들은 이 엔진을 내부적으로 감싸서 모델 관리·API 서버·UI 같은 편의 기능을 얹은 것 입니다. </span></p>
<p id="SE-b14ed4d4-f03a-41e8-b2cf-09887d128b3e"><span id="SE-590e2c88-9c5f-4073-8704-a326dcf584eb"><a href="https://blog.naver.com/pjt3591oo/224410064127" data-linktype="text" data-linkdata="{&quot;id&quot;: &quot;SE-590e2c88-9c5f-4073-8704-a326dcf584eb&quot;, &quot;link&quot;: &quot;https://blog.naver.com/pjt3591oo/224410064127&quot;}" target="_blank">https://blog.naver.com/pjt3591oo/224410064127</a></span></p>

                            </div>                
                                <div id="SE-b10bd514-9b28-41d0-afd8-420aa51ca431">
                                    <p>$ curl -LsSf https://llama.app/install.sh | sh</p>
                                </div>                <div id="SE-1299ec67-ca17-4813-ab68-50c5e385b808">
                                    <p>$ $ llama help all
Usage: llama &lt;command&gt; [options]

Available commands:
  serve           HTTP API server
  cli             Command-line interactive interface
  update          Update llama to the latest release
  download        Download a model
  completion      Text completion
  bench           Benchmark prompt processing and text generation
  batched-bench   Benchmark batched decoding performance
  fit-params      Compute parameters to fit a model in device memory
  quantize        Quantize a model
  perplexity      Compute model perplexity and KL divergence
  version         Show version
  licenses        Show third-party licenses
  help            Show available commands

Run 'llama &lt;command&gt; --help' for command-specific usage.</p>
                                </div>                                <div id="SE-9233cca0-e95b-44bb-9e2b-7b582929e220">
                                    <p>$ llama cli -hf Qwen/Qwen2.5-1.5B-Instruct-GGUF
Downloading qwen2.5-1.5b-instruct-q4_k_m.gguf ────────────────────── 100%


Loading model...

▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b10217-ddd4ec142
model      : Qwen/Qwen2.5-1.5B-Instruct-GGUF
ftype      : Q4_K - Medium
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read &lt;file&gt;        add a text file
  /glob &lt;pattern&gt;     add text files using globbing pattern



&gt; hi
Hello! How can I assist you today?

[ Prompt: 237.0 t/s | Generation: 174.4 t/s ]</p>
                                </div>                <div id="SE-44be8871-304e-4c0d-9a65-7c3e663a1865">
                                <p><a href="#" onclick="return false;" data-linktype="img" data-linkdata="{&quot;id&quot; : &quot;SE-44be8871-304e-4c0d-9a65-7c3e663a1865&quot;, &quot;src&quot; : &quot;https://mblogthumb-phinf.pstatic.net/MjAyNjA5MjZfMzQg/MDAxNzkwMzk3MTMxMDk3.6BEzy2eWrjnY6rsapAqc1-y2DWamCw4XUP7tbwS5t9Yg.GByR7J8jM92DPiJudNKcRNJF9QYd_QBxcmPD4IMOcHMg.PNG/image.png&quot;, &quot;originalWidth&quot; : &quot;1771&quot;, &quot;originalHeight&quot; : &quot;723&quot;, &quot;linkUse&quot; : &quot;false&quot;, &quot;link&quot; : &quot;&quot;, &quot;fileSize&quot;: &quot;62386&quot; }">
                                        <img src="https://mblogthumb-phinf.pstatic.net/MjAyNjA5MjZfMzQg/MDAxNzkwMzk3MTMxMDk3.6BEzy2eWrjnY6rsapAqc1-y2DWamCw4XUP7tbwS5t9Yg.GByR7J8jM92DPiJudNKcRNJF9QYd_QBxcmPD4IMOcHMg.PNG/image.png?type=w800" data-width="693" data-height="282" alt="">
                                    </a>
                                </p>
                        </div>
                <div id="SE-99074cac-7270-4c93-a222-6750b5c5af45">
                                    <p id="SE-84a4dc2a-3624-4b51-96b1-a8cd256afa27"><span id="SE-36237b70-d365-4128-9b36-486fb3f043db"><b>· 서버 구동</b></span></p>
<p id="SE-f2a3b88d-3f2a-46d7-938a-72d3b894d69b"><span id="SE-7e567458-13a7-4876-88f6-9f10789c027c">llama.cpp또한 openAI 스펙을 제공합니다.</span></p>

                            </div>                <div id="SE-c8d7776f-237c-45e7-857c-6ead46f94f0b">
                                    <p>$ llama serve -hf Qwen/Qwen2.5-1.5B-Instruct-GGUF -c 16384 --port 8080

0.00.576.195 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.740.646 W srv  llama_server: -----------------
0.00.740.665 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.740.666 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.740.666 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.740.666 W srv  llama_server: -----------------
0.00.742.263 I srv    load_model: loading model 'Qwen/Qwen2.5-1.5B-Instruct-GGUF'
0.00.971.347 W load: control-looking token: 128247 '&lt;/s&gt;' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.538.911 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 16384, kv_unified = 'true'
0.01.541.424 I srv  llama_server: model loaded
0.01.541.446 I srv  llama_server: listening on http://127.0.0.1:8080</p>
                                </div>                <div id="SE-52a43213-1098-4af4-a82a-9c167b69399e">
                                    <p>$ curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-1.5b-instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "안녕? 넌 누구야?"}
    ],
    "stream": false
  }'</p>
                                </div>                <div id="SE-b2179b5d-588c-41e7-a634-5ef69a26a316">
                                    <p>{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "안녕하세요, 저는 알리바바 플러스입니다. 알리바바 플러스는 Alibaba Group의 인공지능 모델입니다. 어떻게 도와드릴까요?"
      }
    }
  ],
  "created": 1790397289,
  "model": "Qwen/Qwen2.5-1.5B-Instruct-GGUF",
  "system_fingerprint": "b10217-ddd4ec142",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 43,
    "prompt_tokens": 27,
    "total_tokens": 70,
    "prompt_tokens_details": {
      "cached_tokens": 26
    }
  },
  "id": "chatcmpl-Vji8PNVYO25w72wHwoAIRdEfjX5oflB2",
  "timings": {
    "cache_n": 26,
    "prompt_n": 1,
    "prompt_ms": 4.253,
    "prompt_per_token_ms": 4.253,
    "prompt_per_second": 235.1281448389372,
    "predicted_n": 43,
    "predicted_ms": 168.328,
    "predicted_per_token_ms": 3.9146046511627905,
    "predicted_per_second": 255.45363813506964
  }
}</p>
                                </div>                                <div id="SE-58758db5-89d2-4e28-955d-7a11ec5aca61">
                                <p><a href="#" onclick="return false;" data-linktype="img" data-linkdata="{&quot;id&quot; : &quot;SE-58758db5-89d2-4e28-955d-7a11ec5aca61&quot;, &quot;src&quot; : &quot;https://mblogthumb-phinf.pstatic.net/MjAyNjA5MjZfNDgg/MDAxNzkwMzk3MzM0Mzc2.hf0lkfTykFKoFsmFtQD4bBb6O3pqg6OlssIoSNpB2T4g._Hrrp_pPdpL5NMNa-2T8SzjcXoBbn6Irebr_3zNNiC8g.PNG/image.png&quot;, &quot;originalWidth&quot; : &quot;1548&quot;, &quot;originalHeight&quot; : &quot;1380&quot;, &quot;linkUse&quot; : &quot;false&quot;, &quot;link&quot; : &quot;&quot;, &quot;fileSize&quot;: &quot;64563&quot; }">
                                        <img src="https://mblogthumb-phinf.pstatic.net/MjAyNjA5MjZfNDgg/MDAxNzkwMzk3MzM0Mzc2.hf0lkfTykFKoFsmFtQD4bBb6O3pqg6OlssIoSNpB2T4g._Hrrp_pPdpL5NMNa-2T8SzjcXoBbn6Irebr_3zNNiC8g.PNG/image.png?type=w800" data-width="693" data-height="617" alt="">
                                    </a>
                                </p>
                        </div>
                <div id="SE-1c75199a-3e38-4868-bab4-63a319e1c594">
                                    <p id="SE-e9217923-ce3f-4039-b2f2-a6125f15f712"><span id="SE-3abea742-8865-4553-a853-a661eab3c991">그리고 llama.cpp는 --split-mode를 통해 layer(레이어/파이프라인), row(텐서/로우)를 분할할 수 있습니다. 이는 거대한 모델을 여러 개의 GPU에 남을때 사용합니다.</span></p>
<p id="SE-82770b44-9615-48b8-9eaf-ba69fa40448b"><span id="SE-6fdcb1a8-c2d5-4157-b2df-ecd58f260176">​</span></p>
<p id="SE-5d73dab8-1b78-43fb-82ce-e6688e1eafeb"><span id="SE-2cb33173-1ee3-4d28-94a5-15e2f2aae133">또한 동시 요청 처리를 위해 -np(number of parallel slots)을 지정할 수 있습니다.</span></p>

                            </div>                <div id="SE-7233ff3e-71dc-4eb9-bb33-f3ec2b1b513c">
                                    <p>$ llama server -m model.gguf --split-mode row -ngpu 2

$ llama server m model.gguf -c 8192 -np 4</p>
                                </div>                <div id="SE-1623e36b-e1e0-43a7-83d1-2b30617b9a4b">
                                    <p id="SE-3f77a1ef-2c79-4eac-aabc-ec1b171f55ff"><span id="SE-e6760bc2-e08f-417c-87bc-ffbbd3d49140">​</span></p>
<p id="SE-315b67e1-e1b1-4ca7-9c77-1da382999bd9"><span id="SE-e35d6e24-d3cd-4e61-a373-4cb5e535fb57"><b>· 모델확인</b></span></p>
<p id="SE-459faae8-ac53-4808-b13d-e88b629d4587"><span id="SE-4f9e15ef-7e77-4914-8653-b96449ce6ad0">llama.cpp는 ~/.cache/huggingface/hub에서 다운받은 모델을 저장합니다.</span></p>

                            </div>                <div id="SE-29cd972f-eaed-47c0-96b4-014a0e0cf860">
                                    <p>$ ls -la ~/.cache/huggingface/hub/</p>
                                </div>                <div id="SE-6afa4930-18c6-4d11-8587-29d4b430a809">
                                    <p id="SE-6fdb54a6-c4b0-48ac-82d9-d8df15616f44"><span id="SE-ebd1a4d0-855b-4904-978e-dbf0416b3764"><b>· 임베딩</b></span></p>
<p id="SE-69f4cdd8-a977-402e-a022-57516194d584"><span id="SE-b75a62b0-d7e1-4e56-b2d6-c2d8bd3846b8">llama serve를 실행할 때 embedding 옵션을 추가하면 임베딩이 가능하도록 서버를 실행할 수 있습니다.</span></p>

                            </div>                <div id="SE-017586f9-b5cf-425d-8caa-bc4a98b39852">
                                    <p>$ llama server -hf nomic-ai/nomic-embed-text-v1.5-GGUF \
  --embedding \
  -c 8192 \
  --port 8080 \
  --pooling mean</p>
                                </div>                <div id="SE-502c078f-f676-47c2-a7bf-36ce41cc221d">
                                    <p>$ curl http://127.0.0.1:8080/embedding \
  -H "Content-Type: application/json" \
  -d '{"content": "임베딩 테스트 문장입니다."}'</p>
                                </div>                <div id="SE-c0825366-1dbe-4e15-8b26-41e391b4e943">
                                    <p>[{"index":0,"embedding":[[-0.000252246274612844, ... 생략 ...]]}]</p>
                                </div>                    </div></div>

삽질하는 멍개씨

안녕하세요 멍개입니다. 전에 lm studio와 ollama를 이용하여 로컬에서 LLM을 구동하는 방법을 다루어 보았습니다. https://blog.naver.com/pjt3591oo/223939267795 https://blog.naver.com/pjt3591oo/223805688524 이번 포스트에서는 두 도구가 내부적으로 사용하는 llama.cpp를 다뤄봅니다. llama.cpp는 gguf를 서빙하는 도구입니다. llama.cpp는 GGUF 포맷 모델을 구동하는 핵심 추론 엔진이며, Ollama와 LM Studio(LM Studio는 GGUF 이외에 MLX 포맷도 지원함.) 같은 툴들은 이 엔진을 내부적으로 감싸서 모델 관리·API 서버·UI 같은 편의 기능을 얹은 것 입니다. https://blog.naver.com/pjt3591oo/224410064127 · 설치 · cli 구동 · 서버 구.......

[LLM] llama.cpp를 이용하여 로컬에서 LLM을 모델(gguf)을 구동해보자

댓글