python如何爬取淘寶某關鍵詞的所有商品及其信息

這篇文章將為大家詳細講解有關python如何爬取淘寶某關鍵詞的所有商品及其信息，小編覺得挺實用的，因此分享給大家做個參考，希望大家閱讀完這篇文章后可以有所收獲。

源代碼：https://github.com/Ckend/taobao_scrapy

源代碼包含兩個Python文件，get_items.py 用于通過關鍵詞獲得商品基本信息，包括商品名稱、店鋪、銷量、評論數、圖片、優惠、出貨地、原價等等。輸出格式為.csv文件。

另一個Python文件: get_review_cloud.py 用于通過userID和itemID (從get_items.py中獲得這兩個信息) 獲得某商品的評論云。輸出.csv文件。

源代碼很簡單，十幾行，邏輯如下：

1. 訪問接口(f=urllib.request.urlopen(url)) 

2. 讀取源代碼并解碼為utf-8然后轉化為json 

3. 寫入至.csv文件。

以下給出get_items.py的內容

get_items.py:

import urllib.request

import urllib.parse

import json

import csv

def write_csv(data, filename):

    count = 0

    with open(filename, 'a') as outf:

        dw = csv.DictWriter(outf, fieldnames=["title","sold","commentCount","item_id","shipping","fastPostFee","userId","nick","userType","isB2c","location","sellerLoc","pic_path","type","tItemType","zkType","zkGroup","priceColor","priceWithRate","auctionURL","isP4p","itemNumId","originalPrice","freight","act","coinLimit","priceWap","price","category","auctionType","url","img2","wwimUrl","previewUrl","favoriteUrl","isMobileEcard","iswebp","name","iconList","icons","area"])

        if count == 0:

            # 第一行才寫入頭部

            dw.writeheader()

        count += 1

        for row in data:

            dw.writerow(row)

def get_items(searchWords, page):

    url = "https://s.m.taobao.com/search?event_submit_do_new_search_auction=1&_input_charset=utf-8&topSearch=1&atype=b&searchfrom=1&action=home%3Aredirect_app_action&from=1&q="+ str(urllib.parse.quote(searchWords)) +"&sst=1&n=20&buying=buyitnow&m=api4h6&token4h6=&abtest=20&wlsort=20&page="+str(page)

    f = urllib.request.urlopen(url)

    result = json.loads(f.read().decode('utf-8'))

    return result['listItem']

def get_items_by_keywords(searchWords, page):

    for i in range(1,page):

        result = get_items(searchWords ,i)

    try:

        write_csv(result,"./result/"+str(searchWords)+"_result.csv")

    except INdexError:

        # 關鍵詞產品已抓取完

        print("該關鍵詞產品已全部抓取完畢")

get_items_by_keywords("空氣清新器", 100)

關于“python如何爬取淘寶某關鍵詞的所有商品及其信息”這篇文章就分享到這里了，希望以上內容可以對大家有一定的幫助，使各位可以學到更多知識，如果覺得文章不錯，請把它分享出去讓更多的人看到。