<div id="readability-page-1" class="page"><div>
		
<p>Scaling Language Modeling with Pathways</p>



<p>PaLM논문을 살펴보게 된 배경은 대용량 언어모델의 서비스 상용화가 ChatGPT의 출현으로 실현되었고, <br>이런 대용량 언어모델에 대한 서비스 방향성과 추론 및 학습을 위한 인프라 환경에 대한 이해를 하기 위하여<br>2022년 4월에 구글에서 발표한 PaLM논문을 리뷰해 보기로 한다.</p>



<figure><img decoding="async" width="958" height="376" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.10.34.png?resize=958%2C376&amp;ssl=1" alt="" sizes="(max-width: 958px) 100vw, 958px" data-recalc-dims="1"></figure>











<h2><strong>사전 지식 공유</strong></h2>



<figure><img decoding="async" loading="lazy" width="641" height="513" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.11.11.png?resize=641%2C513&amp;ssl=1" alt="" sizes="(max-width: 641px) 100vw, 641px" data-recalc-dims="1"></figure>











<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.12.06.png?resize=914%2C440&amp;ssl=1" alt="" width="914" height="440" sizes="(max-width: 914px) 100vw, 914px" data-recalc-dims="1"></figure>











<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.12.30.png?resize=904%2C508&amp;ssl=1" alt="" width="904" height="508" sizes="(max-width: 904px) 100vw, 904px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.12.51.png?resize=901%2C488&amp;ssl=1" alt="" width="901" height="488" sizes="(max-width: 901px) 100vw, 901px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" width="980" height="558" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.13.13.png?resize=980%2C558&amp;ssl=1" alt="" sizes="(max-width: 980px) 100vw, 980px" data-recalc-dims="1"></figure>











<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.13.33.png?resize=923%2C449&amp;ssl=1" alt="" width="923" height="449" sizes="(max-width: 923px) 100vw, 923px" data-recalc-dims="1"></figure>











<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.13.58.png?resize=943%2C533&amp;ssl=1" alt="" width="943" height="533" sizes="(max-width: 943px) 100vw, 943px" data-recalc-dims="1"></figure>











<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.14.15.png?resize=944%2C482&amp;ssl=1" alt="" width="944" height="482" sizes="(max-width: 944px) 100vw, 944px" data-recalc-dims="1"></figure>











<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.14.39.png?resize=937%2C489&amp;ssl=1" alt="" width="937" height="489" sizes="(max-width: 937px) 100vw, 937px" data-recalc-dims="1"></figure>











<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.15.00.png?resize=937%2C463&amp;ssl=1" alt="" width="937" height="463" sizes="(max-width: 937px) 100vw, 937px" data-recalc-dims="1"></figure>











<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.15.20.png?resize=931%2C447&amp;ssl=1" alt="" width="931" height="447" sizes="(max-width: 931px) 100vw, 931px" data-recalc-dims="1"></figure>











<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.15.41.png?resize=927%2C459&amp;ssl=1" alt="" width="927" height="459" sizes="(max-width: 927px) 100vw, 927px" data-recalc-dims="1"></figure>



















<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.17.01.png?resize=936%2C488&amp;ssl=1" alt="" width="936" height="488" sizes="(max-width: 936px) 100vw, 936px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.17.20.png?resize=919%2C496&amp;ssl=1" alt="" width="919" height="496" sizes="(max-width: 919px) 100vw, 919px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.17.44.png?resize=927%2C491&amp;ssl=1" alt="" width="927" height="491" sizes="(max-width: 927px) 100vw, 927px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.18.05.png?resize=935%2C437&amp;ssl=1" alt="" width="935" height="437" sizes="(max-width: 935px) 100vw, 935px" data-recalc-dims="1"></figure>















<h2><strong>2&gt; Reasoning</strong></h2>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.21.08.png?resize=932%2C515&amp;ssl=1" alt="" width="932" height="515" sizes="(max-width: 932px) 100vw, 932px" data-recalc-dims="1"></figure>















<h2><strong>LLMs are Zero-Shot Reasoners</strong></h2>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.21.56.png?resize=941%2C510&amp;ssl=1" alt="" width="941" height="510" sizes="(max-width: 941px) 100vw, 941px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.22.31.png?resize=945%2C479&amp;ssl=1" alt="" width="945" height="479" sizes="(max-width: 945px) 100vw, 945px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.22.49.png?resize=942%2C447&amp;ssl=1" alt="" width="942" height="447" sizes="(max-width: 942px) 100vw, 942px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.23.08.png?resize=943%2C483&amp;ssl=1" alt="" width="943" height="483" sizes="(max-width: 943px) 100vw, 943px" data-recalc-dims="1"></figure>















<h2><strong>Zero-Shot Learning</strong></h2>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.24.04.png?resize=942%2C452&amp;ssl=1" alt="" width="942" height="452" sizes="(max-width: 942px) 100vw, 942px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.24.27.png?resize=943%2C460&amp;ssl=1" alt="" width="943" height="460" sizes="(max-width: 943px) 100vw, 943px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.24.46.png?resize=943%2C497&amp;ssl=1" alt="" width="943" height="497" sizes="(max-width: 943px) 100vw, 943px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.27.11.png?resize=947%2C507&amp;ssl=1" alt="" width="947" height="507" sizes="(max-width: 947px) 100vw, 947px" data-recalc-dims="1"></figure>



<figure><img decoding="async" loading="lazy" src="https://i0.wp.com/hugrypiggykim.com/wp-content/uploads/2023/03/%EC%8A%A4%ED%81%AC%EB%A6%B0%EC%83%B7-2023-03-09-%EC%98%A4%ED%9B%84-3.27.43.png?resize=949%2C481&amp;ssl=1" alt="" width="949" height="481" sizes="(max-width: 949px) 100vw, 949px" data-recalc-dims="1"></figure>



<h2><strong>참고 링크</strong></h2>



<p>Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (<a href="https://arxiv.org/abs/2201.11903">https://arxiv.org/abs/2201.11903</a>)</p>



<p><a href="https://jiho-ml.com/weekly-nlp-54/">https://jiho-ml.com/weekly-nlp-54/</a></p>



<p><a href="https://wikidocs.net/31379">https://wikidocs.net/31379</a></p>



<p><a href="https://velog.io/@tobigs-nlp/PaLM-Scaling-Language-Modeling-with-Pathways-2">https://velog.io/@tobigs-nlp/PaLM-Scaling-Language-Modeling-with-Pathways-2</a></p>



<p>Large Language Models are Zero-Shot Reasoners<strong> (</strong><a href="https://arxiv.org/abs/2205.11916">https://arxiv.org/abs/2205.11916</a>)</p>



<p><a href="https://velog.io/@nochesita/%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0-Large-Language-Models-are-Zero-Shot-Reasoners">https://velog.io/@nochesita/%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0-Large-Language-Models-are-Zero-Shot-Reasoners</a></p>



<p><a href="https://deep-learning-study.tistory.com/873">https://deep-learning-study.tistory.com/873</a></p>



<p>A Review of Generalized Zero-Shot Learning Methods (<a href="https://arxiv.org/abs/2011.08641">https://arxiv.org/abs/2011.08641</a>)</p>



<p><a href="https://www.blossominkyung.com/deeplearning/transformer-mha">https://www.blossominkyung.com/deeplearning/transformer-mha</a></p>
		</div></div>

TensorMSA

Scaling Language Modeling with Pathways
PaLM논문을 살펴보게 된 배경은 대용량 언어모델의 서비스 상용화가 ChatGPT의 출현으로 실현되었고, 
이런 대용량 언어모델에 대한 서비스 방향성과 추론 및 학습을 위한 인프라 환경에 대한 이해를 하기 위하여
2022년 4월에 구글에서 발표한 PaLM논문을 리뷰해 보기로 한다.
사전 지식 공유

1> Pathways

2> Reasoning

LLMs are Zero-Shot Reasoners

Zero-Shot Learning
참고 링크
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (https://arxiv.org/abs/2201.11903)
https://jiho-ml.com/weekly-nlp-54/
https://wikidocs.net/31379
https://velog.io/@tobigs-nlp/PaLM-Scaling-Language-Modeling-with-Pathways-2
Large Language Models are Zero-Shot Reasoners (https://arxiv.org/abs/2205.11916)
https://velog.io/@nochesita/%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0-Large-Language-Models-are-Zero-Shot-Reasoners
https://deep-learning-study.tistory.com/873
A Review of Generalized Zero-Shot Learning Methods (https://arxiv.org/abs/2011.08641)
https://www.blossominkyung.com/deeplearning/transformer-mha

PaLM (Pathways Language Model)

댓글