问题:使用 Python 计算网站中词频的代码未输出正确的频率

我想计算特定网站中单词列表的频率。但是,该代码不会返回手动“控制 F”命令所返回的确切字数。我究竟做错了什么?

这是我的代码:

import pandas as pd
from bs4 import BeautifulSoup
import requests
from selenium import webdriver
import re

url='https://www.gov.uk/government/publications/specialist-quality-mark-tender-2016'
fr=[] 
wanted = ['tender','2020','date']    
for word in wanted:
    a=requests.get(url).text.count(word)
    dic={'phrase':word,
          'frequency':a,              
            }          
    fr.append(dic)  
    print('Frequency of',word, 'is:',a)
data=pd.DataFrame(fr)    

解答

请参阅您问题中的评论,以了解为什么使用requests来计算网页“可见光谱”中单词的频率(您在浏览器中实际看到的)可能不是一个好主意。

如果你想用selenium解决这个问题,你可以尝试:

from selenium import webdriver

url = 'https://www.gov.uk/government/publications/specialist-quality-mark-tender-2016'

driver = webdriver.Chrome(chromedriver_location)
driver.get(url)
body = driver.find_element_by_tag_name('body')

fr = [] 
wanted = ['tender', '2020', 'date']    
for word in wanted:
    freq = body.text.lower().count(word) # .lower() to account for count's case sensitive behaviour
    dic = {'phrase': word, 'frequency': freq}          
    fr.append(dic)  
    print('Frequency of', word, 'is:', freq)

这给了我与CTRL + F相同的结果。

您可以通过稍微修改代码来测试BeautifulSoup到(顺便导入):

import requests
from bs4 import BeautifulSoup

url = 'https://www.gov.uk/government/publications/specialist-quality-mark-tender-2016'
fr = [] 
wanted = ['tender','2020','date']    
a = requests.get(url).text
soup = BeautifulSoup(a, 'html.parser')
for word in wanted:
    freq = soup.get_text().lower().count(word)
    dic = {'phrase': word, 'frequency': freq}          
    fr.append(dic)  
    print('Frequency of', word, 'is:', freq)

这给了我相同的结果,除了tender这个词,根据BeautifulSoup出现 12 次,而不是 11 次。你自己测试一下,看看什么适合你。