问题:BeautifulSoup Amazon Web Scraping - NoneType 错误

我正在尝试抓取亚马逊网页,但是当我尝试抓取产品标题时,我不断收到以下错误:

'NoneType' 对象不可下标

我使用相同的模板来拉价格并且效果很好,所以我不明白为什么它也不适用于产品名称。此外,当我在亚马逊页面上检查元素时,我可以看到 div 明显存在。

这就是我要捕获的 div 的样子:

<div class="p13n-sc-truncated" aria-hidden="true" data-rows="2" title="Fire TV Stick (3rd Gen) with Alexa Voice Remote (includes TV controls) | HD streaming device | 2021 release">Fire TV Stick (3rd Gen) with Alexa Voice Remote (includes TV controls) | HD streaming device |…</div>

错误发生在我的代码中 for 循环的“名称”部分。这是我的代码:

import bs4
import requests

URL = 'https://www.amazon.ca/Best-Sellers-Electronics/zgbs/electronics/ref=zg_bs_electronics_home_all?pf_rd_p=a46a1735-3543-4413-99b2-48b0691cac1a&pf_rd_s=center-6&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=A3DWYIK6Y9EEQB&pf_rd_r=K1YTKARBX7RJ24SX7FHY&pf_rd_r=K1YTKARBX7RJ24SX7FHY&pf_rd_p=a46a1735-3543-4413-99b2-48b0691cac1a'

headers = ({'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'}) 
html = requests.get(URL, headers=headers).text
soup = bs4.BeautifulSoup(html, 'lxml')
prod_table = soup.find('ol', {'class':'a-ordered-list a-vertical', 'role':'grid'})

for element in prod_table.find_all("li", {'class':'zg-item-immersion'}):
    price = element.find('span', {'class':'p13n-sc-price'}).text
    name = element.find('div', {'class': 'p13n-sc-truncated'})['title']  # THIS IS WHERE PROBLEM IS
    print(name)

解答

您可能已经收到验证码页面,因此请尝试添加更多 HTTP 标头以防止这种情况发生。此外,我将课程从p13n-sc-truncated更改为p13n-sc-truncate:

import bs4
import requests

URL = "https://www.amazon.ca/Best-Sellers-Electronics/zgbs/electronics/ref=zg_bs_electronics_home_all?pf_rd_p=a46a1735-3543-4413-99b2-48b0691cac1a&pf_rd_s=center-6&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=A3DWYIK6Y9EEQB&pf_rd_r=K1YTKARBX7RJ24SX7FHY&pf_rd_r=K1YTKARBX7RJ24SX7FHY&pf_rd_p=a46a1735-3543-4413-99b2-48b0691cac1a"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.5",  # <-- add "Accept-Language" to try to prevent captcha
}
html = requests.get(URL, headers=headers).text
soup = bs4.BeautifulSoup(html, "lxml")

prod_table = soup.find(
    "ol", {"class": "a-ordered-list a-vertical", "role": "grid"}
)

for element in prod_table.find_all("li", {"class": "zg-item-immersion"}):
    price = element.find("span", {"class": "p13n-sc-price"}).text
    name = element.find("div", {"class": "p13n-sc-truncate"}).get_text(
        strip=True
    )
    print(name)

印刷:

Fire TV Stick 4K streaming device with Alexa built in, Ultra HD, Dolby Vision, includes the Alexa Voice Remote
Fire TV Stick (3rd Gen) with Alexa Voice Remote (includes TV controls) | HD streaming device | 2021 release
VicTsing FM Transmitter for Car Bluetooth 5.0, Siri Voice Assistant, QC3.0 Wireless in-Car FM Radio Adapter with LED Backlit, 2 USB Ports, Hi-Fi Music, Hands-free Calling, Support U Disk/Bluetooth Connection
Wyze Cam v3 1080p HD IndWyze Cam v3 1080p HD Indoor/Outdoor Video Camera with Color Night Vision, 2-Way Audio, Works with Alexa & The Google Assistant, and IFTTT
TP-Link AC1750 WiFi Extender RE450 - Covers up to 2,000 sq. ft. and 32 Devices, Dual Band Wireless Signal Booster & Repeater (up to 1750Mbps Speed), Compact Wall Plug Design
Seagate Portable 2TB External Hard Drive Portable HDD – USB 3.0 for PC Laptop and Mac (STGX2000400)
Fire TV Stick Lite with Alexa Voice Remote Lite (no TV controls) | HD streaming device | 2020 release
DYMO LetraTag LT-100H Plus Handheld Label Maker with 3 Bonus LetraTag Labeling Tapes (1955663)
Duracell - 2032 3V Lithium Coin Battery - long lasting battery - 1 count
ColorCoral Keyboard Cleaner Universal Cleaning Gel for PC Tablet Laptop Keyboards, Car Vents, Cameras, Printers, 160G
Ledger Nano S - The Best Crypto Hardware Wallet - Secure and Manage Your Bitcoin, Ethereum, ERC20 and Many Other Coins
Fire HD 8 tablet, 8" HD display, 32 GB, designed for portable entertainment, Black
UBeesize 10’’ Selfie Ring Light with Stand and Phone holder, LED Ring Light with 62’’ Stand and Bluetooth Shutter for Video Recording＆Live streaming(YouTube,Instagram , Facebook Stories), compatible with iPhone＆Android
Echo Dot (3rd gen) - Smart speaker with Alexa - Charcoal
AmazonBasics 6-Sheet Cross-Cut Paper and Credit Card Home Office Shredder
APEMAN Dash Cam 1080P FHD DVR Car Driving Recorder 3 Inch LCD Screen 170° Wide Angle, G-Sensor, WDR, Parking Monitor, Loop Recording, Motion Detection
Roku Express | HD Streaming Media Player with Simple Remote and Premium HDMI Cable
Selfie Ring Light with Tripod Stand & Cell Phone Holder for Live Stream/Makeup, UBeesize Mini Led Camera Ringlight for YouTube Video/Photography
Amazon Smart Plug, works with Alexa
Kindle Paperwhite – Now Waterproof with 2x the Storage
SanDisk Ultra Flair USB 3.0 128GB Flash Drive High Performance up to 150MB/s (SDCZ73-128G-G46)
Seagate BarraCuda 2TB Internal Hard Drive HDD 3.5 Inch SATA 6Gb/s 7200 RPM 256MB Cache 3.5-Inch Frustration Free Packaging
DYMO LabelManager 280 Rechargeable Hand-Held Label Maker (1815990)
Blink Outdoor – wireless, weather-resistant HD security camera with two-year battery life and motion detection – 2 camera kit
NETGEAR 5-Port Gigabit Ethernet Unmanaged Switch (GS305) - Desktop, Sturdy Metal Fanless Housing
Fire HD 10 Tablet (10.1" 1080p full HD display, 32 GB) – Black (9th Generation)
ColorCoral Cleaning Gel Universal Gel Cleaner for Car Vent Keyboard Auto Cleaning Putty Dashboard Dust Remover Putty Auto Duster Cleaning Kit 160G
Mpow Bluetooth 5.0 USB Adapter for PC, Bluetooth Dongle Supports Windows 7/8.1/10, for Desktop, Laptop, Mouse, Keyboard, Printers, Headsets, Speakers
Wyze Cam 1080p HD Indoor Wireless Smart Home Camera with Night Vision, 2-Way Audio, Works with Alexa (Pack of 2)
Indoor Security Camera, Littlelf 1080P Home Wifi Wireless Camera with 2-Way Audio Night Vision Motion Detection for Pet/Elder/Baby Monitor, Cloud Storage and MicroSD Support (Not Included)
TP-Link AC1200 WiFi Extender RE300 - Covers up to 1,500 sq.ft and 25 Devices, Dual Band Wireless Signal Booster & Repeater (up to 1200Mbps Speed), and Compact Wall Plug Design
Bike Phone Mount, Pobon Bicycle Phone Holder Adjustable Handlebar Motorcycle Phone Mount Compatible with iPhone 12 Pro Max, 12 Mini,11 Pro Max,XR,XS,X,8 Plus, Samsung Galaxy Note 20,S20, 4.7-7.5 Inch
Bluetooth 5.0 Transmitter/Receiver, TaoTronics 2-in-1 Wireless 3.5mm Audio Adapter (aptX Low Latency, 2 Devices Simultaneously, for TV/Home Sound System)
2021 Upgrade Projector, Mini Video Projector with 5500 LUX, 1080P Supported, Portable Outdoor Movie Projector, 176" Display Compatible with TV Stick, HDMI, USB, VGA, AV for Home Entertainment
Fire HD 8 Kids Edition tablet, 8" HD display, 32 GB, Kid-Proof Case, Blue
Wyze Cam v2 1080p HD Smart Home Camera with Night Vision, 2-Way Audio, Free Cloud, for iOS and Android (US Version)
Roku Premiere | HD/4K/HDR Streaming Media Player with Simple Remote and Premium HDMI Cable
Samsung 870 EVO 500GB SATA 2.5" Internal SSD (MZ-77E500B/AM) [Canada Version]
TCL 32S327-CA 1080p Smart LED Television (2019), 32"
Fire HD 8 tablet, 8" HD display, 32 GB, designed for portable entertainment, Twilight Blue
Amazon Ethernet Adapter for Amazon Fire TV Devices
Fire 7 Tablet (7" display, 16 GB) - Black
TP-Link WiFi Extender TL-WA850RE - Covers up to 800 sq. ft. with N300 Wireless Signal Booster & Repeater (Up to 300Mbps Speed), Single Band 2.4Ghz only, and Compact Wall Plug Design
TP-Link AC1300 USB 3.0 Mini WiFi Adapter, 2.4G/5G Dual Band Wireless Network Adapter for PC Desktop, MU-MIMO WiFi Dongle, Supports Windows 10, 8.1, 8, 7, XP/Mac OS X 10.9-10.14 (Archer T3U)
Vtech Dect 6.0 2-Handset Cordless Phone System with Caller ID, Backlit Keypad and Screen(CS6114), White
Seagate Portable 1TB External Hard Drive HDD – USB 3.0 for PC Laptop and Mac (STGX1000400)
TP-Link 8 Port Gigabit Ethernet Network Switch | Ethernet Splitter | Sturdy Metal w/ Shielded Ports | Plug-and-Play | Traffic Optimization | Unmanaged | Lifetime Warranty (TL-SG108)
UBeesize Phone tripod, UBeesize Portable and Adjustable Camera Stand Holder with Wireless Remote and Universal Clip, Compatible with iPhone, Android Phone, Camera, Sports Camera GoPro
Wireless Earbuds, Bluetooth 5.0 Graphene bass Hi-Fi Headphones with 1500mAh LED Charging Case, IPX7 Waterproof TWS in-Ear Headset 60H Playtime, Binaural Stereo Sound Earphones, Noise Canceling w/Mic
Roku Streaming Stick+ | HD/4K/HDR Streaming Device with Long-range Wireless and Voice Remote with TV Power and Volume