<div id="readability-page-1" class="page"><div>
                <div id="SE-573abf4c-9368-445d-af7a-736751b23b90">
                                    <p id="SE-bae8cabe-1707-4453-9a6c-82a5b2f0fb4f"><span id="SE-4eb6c1cd-ead3-40f3-9ae0-7c92a67cf3cc">안녕하세요 멍개입니다.</span></p><p id="SE-04fd991b-7d2f-4987-b282-3017db798fd8"><span id="SE-d5deafe8-524e-47c1-8c95-6a71773d8343">​</span></p><p id="SE-0fbaa6e3-0005-48a2-8818-8e31cb5d6b04"><span id="SE-6a747a2d-81a9-41e1-bde4-9423e838b49b"><b>· 게시글 목록 저장</b></span></p><p id="SE-88b62da8-6adc-4e88-af4d-9a33cb294d74"><span id="SE-5ce705af-c6c5-4a3a-a6ba-ab2be8332d07">게시글 목록을 가져온 후 blog-rag-data/post_list.json으로 저장합니다.</span></p>
                            </div>                <div id="SE-88126bf5-5c47-467a-991e-5be48ea51502">
                                    <p>import requests, json, time, random
from urllib import parse

countPerPage = 30
blogId = 'pjt3591oo'

def get_blog_list(page):
    url = f'https://blog.naver.com/PostTitleListAsync.naver?blogId={blogId}&amp;viewdate=&amp;currentPage={page}&amp;categoryNo=0&amp;parentCategoryNo=&amp;countPerPage={countPerPage}'
    response = requests.get(url)
    return json.loads(response.text.replace('\\', ''))['postList']

if __name__ == "__main__":
    page = 0
    post_count = 1
    post_list = []

    while True:
        posts = get_blog_list(page)
        
        for post in posts:
            url = f'https://blog.naver.com/PostView.naver?blogId={blogId}&amp;logNo={post["logNo"]}'
            
            print(f'post count: {post_count}, page: {page}, logNo: {post["logNo"]}, title: {parse.unquote(post["title"])}')

            post_count += 1
            post_list.append({
                'url': url,
                'logNo': post["logNo"],
                'title': parse.unquote(post["title"]),
            })


        if len(posts) != countPerPage:
            break
        
        page += 1

        time.sleep(random.uniform(1.5, 2.5))

    with open('blog-rag-data/post_list.json', 'w') as f:
        json.dump(post_list, f)</p>
                                </div>                <div id="SE-b90b217e-c870-4d2c-988c-cd85fc6e47cf">
                                    <p id="SE-23dee5a2-33c6-403e-b9d3-a37f7178c2b6"><span id="SE-b586fdc3-0576-4848-aeec-49436818c017"><b>· 포스트 데이터 가져오기</b></span></p><p id="SE-b1f8b084-a5ce-4a6b-88a8-f55066f445ae"><span id="SE-e0ca3d68-2d73-4033-9913-08d7aebfd79b">앞에서 저장한 blog-rag-data/post_list.json을 읽어서 각각의 포스트 정보를 가져옵니다. html에서 id가 post-view[blog_id]로 지정된 dom의 텍스트를 가져옵니다.</span></p>
                            </div>                <div id="SE-1bcd82f9-20f4-40cb-93ee-c2af84eb78ec">
                                    <p>import requests, json, time, random
from urllib import parse
from bs4 import BeautifulSoup

blogId = 'pjt3591oo'

def read_blog_postlist_from_file():
    with open('blog-rag-data/post_list.json', 'r') as f:
        return json.load(f)

if __name__ == "__main__":
    info = read_blog_postlist_from_file()
    
    current_post_search_index = info['current_post_search_index']
    post_list = info['post'][current_post_search_index:]

    for post_count, post in enumerate(post_list):
        url = f'https://blog.naver.com/PostView.naver?blogId={blogId}&amp;logNo={post["logNo"]}'
        response = requests.get(url)
        soup = BeautifulSoup(response.text, 'lxml')

        content = soup.find('div', id=f'post-view{post["logNo"]}')
        content = content.text
        title = parse.unquote(post["title"])

        print(f'post count: {post_count}, logNo: {post["logNo"]}, url: {url}, title: {title}')
        
        with open(f'blog-rag-data/posts/{post["logNo"]}.json', 'w') as f:
            f.write(json.dumps({
                'content': content,
                'url': url,
                'title': title,
                'logNo': post["logNo"],
            }))
            f.close()
            print('save done')
            
        try:
            print(f"last document content: {content[- 100:].replace('\n', '')}")
        except Exception as err:
            print('***************************')
            print(err)
            # print(documents)
            print(f'post count: {post_count}, logNo: {post["logNo"]}, title: {title}')
            print('exception')
            break
            
        print(f'=================== document size: {len(content)} save done ==================')
        time.sleep(random.uniform(1.5, 2.5))
</p>
                                </div>                    </div></div>

삽질하는 멍개씨

안녕하세요 멍개입니다. · 게시글 목록 저장 게시글 목록을 가져온 후 blog-rag-data/post_list.json으로 저장합니다. · 포스트 데이터 가져오기 앞에서 저장한 blog-rag-data/post_list.json을 읽어서 각각의 포스트 정보를 가져옵니다. html에서 id가 post-view[blog_id]로 지정된 dom의 텍스트를 가져옵니다.

[Crawler] 네이버 블로그 크롤러

댓글