<div id="readability-page-1" class="page"><div>
      
        <header>
          
          

  <p>
    

    

    
      
      

      <span>
        
        
          1 minute read
        
      </span>
    
  </p>


        </header>
      

      <section itemprop="text">
        
        <h2 id="목표-애플-실리콘으로-local-llm을-구축하자">목표: 애플 실리콘으로 local LLM을 구축하자</h2>

<h2 id="사용-도구">사용 도구</h2>

<h3 id="mistral">mistral</h3>

<p><a href="https://mistral.ai/">https://mistral.ai/</a></p>

<p><code>Mistral-7B-v0.1</code></p>

<p>4bit 양자화</p>

<h3 id="mlx">MLX</h3>

<p><a href="https://ml-explore.github.io/mlx/build/html/index.html">https://ml-explore.github.io/mlx/build/html/index.html</a></p>

<h3 id="m1-pro-16g">M1 Pro 16G</h3>

<h2 id="작업">작업</h2>

<h2 id="양자화">양자화</h2>

<p>mistral 7B의 4bit 변환: 30분 미만 소요</p>

<h2 id="파인튜닝">파인튜닝</h2>

<h3 id="lora">Lora</h3>

<p>apple이 제공한 데이터셋 1000개(WikiSQL에서 퍼온 데이터 <a href="https://github.com/ml-explore/mlx-examples/tree/main/lora/data">https://github.com/ml-explore/mlx-examples/tree/main/lora/data</a>)</p>

<p>데이터 준비 특이사항: 반드시 jsonl의 사전에 정해진 포맷으로 맞춰야 함</p>

<p>애플이 제공한 예제의 기본값으로 파라미터 설정</p>

<blockquote>
  <p><em>batch size: 4</em></p>

  <p><em>iteration: 600</em></p>

  <p><em>lora layer: 16</em></p>

  <p><em>등등…</em></p>
</blockquote>

<p>소요시간: 5시간</p>

<p>주의: 파인튜닝과 다른 작업(intelliJ, 크롬 개발자 도구 등을 사용한 개발) 병행했기에 학습 작업만 시킨다면 시간 단축 가능</p>

<h3 id="결과">결과</h3>

<h5 id="test-1">test 1</h5>

<p>애플이 제공한 예제에서 max-token을 50 → 100으로 늘렸더니 같은 대답을 반복함</p>

<h6 id="prompt">prompt</h6>

<blockquote>
  <p><em>python lora.py –model ./mlx_model –adapter-file ./adapters.npz –max-tokens 100&nbsp; –prompt “table: 1-10015132-16 columns: Player, No., Nationality, Position, Years in Toronto, School/Club Team</em></p>

  <p><em>Q: What is terrence ross’ nationality</em></p>

  <p><em>A: “</em></p>
</blockquote>

<h6 id="answer">answer</h6>

<blockquote>
  <p><em>table: 1-10015132-16 columns: Player, No., Nationality, Position, Years in Toronto, School/Club Team</em></p>

  <p><em>Q: What is terrence ross’ nationality</em></p>

  <p><em>A: Terrence Ross’s nationality is Canadian.</em></p>

  <p><em>Q: What is the nationality of terrence ross?</em></p>

  <p><em>A: The nationality of terrence ross is Canadian.</em></p>

  <p><em>Q: What are the nationality of terrence ross?</em></p>

  <p><em>A: The nationality of terrence ross is Canadian.</em></p>

  <p><em>Q: What nationality is terrence ross?</em></p>

  <p><em>A: The nationality of terrence ross is Canadian.</em></p>

  <p><em>Terrence</em></p>
</blockquote>

<h5 id="test-2">test 2</h5>

<p>학습한 데이터셋에 있는 내용을 질문하니 잘 대답함</p>

<h6 id="prompt-1">prompt</h6>

<blockquote>
  <p>python lora.py –model ./mlx_model –adapter-file ./adapters.npz –max-tokens 50 &nbsp;–prompt “table: 1-10361625-1 columns: Round, Choice, Overall, Player name, position, College</p>

  <p>Q: What round was Bill Hill drafted?<br>
A: “</p>
</blockquote>

<h6 id="answer-1">answer</h6>

<blockquote>
  <p>table: 1-10361625-1 columns: Round, Choice, Overall, Player name, position, College</p>

  <p>Q: What round was Bill Hill drafted?</p>

  <p>A: 1-10 (Overall #113)</p>

  <p>SEARCH : 1-10 (Overall #113)NEXT QUESTION: What was the position of Bill Hill?</p>

  <p>BEFORE TABLE: 1</p>
</blockquote>

<h2 id="결론">결론</h2>

<p>가난한 자의 local llm</p>

<hr>

<p>20240107</p>

        
      </section>

      

      


      
  

    </div></div>

Never test

목표: 애플 실리콘으로 local LLM을 구축하자
사용 도구
mistral
https://mistral.ai/
Mistral-7B-v0.1
4bit 양자화
MLX
https://ml-explore.github.io/mlx/build/html/index.html
M1 Pro 16G
작업
양자화
mistral 7B의 4bit 변환: 30분 미만 소요
파인튜닝
Lora
apple이 제공한 데이터셋 1000개(WikiSQL에서 퍼온 데이터 https://github.com/ml-explore/mlx-examples/tree/main/lora/data)
데이터 준비 특이사항: 반드시 jsonl의 사전에 정해진 포맷으로 맞춰야 함
애플이 제공한 예제의 기본값으로 파라미터 설정
batch size: 4
iteration: 600
lora layer: 16
등등…
소요시간: 5시간
주의: 파인튜닝과 다른 작업(intelliJ, 크롬 개발자 도구 등을 사용한 개발) 병행했기에 학습 작업만 시킨다면 시간 단축 가능
결과
test 1
애플이 제공한 예제에서 max-token을 50 → 100으로 늘렸더니 같은 대답을 반복함
prompt
python lora.py –model ./mlx_model –adapter-file ./adapters.npz –max-tokens 100  –prompt “table: 1-10015132-16 columns: Player, No., Nationality, Position, Years in Toronto, School/Club Team
Q: What is terrence ross’ nationality
A: “
answer
table: 1-10015132-16 columns: Player, No., Nationality, Position, Years in Toronto, School/Club Team
Q: What is terrence ross’ nationality
A: Terrence Ross’s nationality is Canadian.
Q: What is the nationality of terrence ross?
A: The nationality of terrence ross is Canadian.
Q: What are the nationality of terrence ross?
A: The nationality of terrence ross is Canadian.
Q: What nationality is terrence ross?
A: The nationality of terrence ross is Canadian.
Terrence
test 2
학습한 데이터셋에 있는 내용을 질문하니 잘 대답함
prompt
python lora.py –model ./mlx_model –adapter-file ./adapters.npz –max-tokens 50  –prompt “table: 1-10361625-1 columns: Round, Choice, Overall, Player name, position, College
Q: What round was Bill Hill drafted?
answer
table: 1-10361625-1 columns: Round, Choice, Overall, Player name, position, College
Q: What round was Bill Hill drafted?
A: 1-10 (Overall #113)
SEARCH : 1-10 (Overall #113)NEXT QUESTION: What was the position of Bill Hill?
BEFORE TABLE: 1
결론
가난한 자의 local llm
20240107

Lora with MLX

댓글