Answer a question

I'm new to web scraping and have been using BeautifulSoup to scrape daily mortgage rates. However, a lot of the servicer sites that I am trying to scrape return 'none' or an empty list. The tags that I'm trying to scrape definitely exist and I just don't understand why this is happening. One example is below.

html=urlopen('https://www.popular.com/en/mortgages/')
bs0bj=BeautifulSoup(html, 'html.parser');
rate=bs0bj.find('span', {'class':'text-md text-popular-medium-blue'}).div
print(rate)

Answers

To get the data you are after you can use selenium in combination with python something like below:

from bs4 import BeautifulSoup
from selenium  import webdriver

driver = webdriver.Chrome()
driver.get('https://www.popular.com/en/mortgages/')

soup = BeautifulSoup(driver.page_source,"lxml")
item = soup.select('.table-responsive')[0].select("span div")[0].text
print(item)
driver.quit()

Result:

2.75%