Answer a question

I've been playing around with scraping data tables using pandas. It's super easy with pd.read_html(), but one of the urls I'm trying it with just won't work. Here's my code:

import pandas as pd
import requests

base_site = 'https://stats.ncaa.org/team/376/stats/15061'

r = requests.get(base_site)
r.status_code

tables = pd.read_html(base_site)

I imported requests to check the status code, which outputs a 200 which is good.

Here is the output for the pd.read_html():

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-4-398e418f089e> in <module>()
----> 1 tables = pd.read_html(base_site, attrs = {'class' : 'dataTable', 'id' : 'statgrid'})

11 frames
/usr/lib/python3.6/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    648 class HTTPDefaultErrorHandler(BaseHandler):
    649     def http_error_default(self, req, fp, code, msg, hdrs):
--> 650         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    651 
    652 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden

I tried adding parameters after looking through the html to make the call more specific, but received the same error to no avail.

tables = pd.read_html(base_site, attrs = {'class' : 'dataTable', 'id' : 'statgrid'})

Am I missing something obvious? Other sites would return a list to the tables object I specified, but I can't figure out why that won't work for the site I actually want data from.

Answers

you might want to check out bs4 or selenium for web scraping.

For you question, try to replace

pd.read_html(base_site)

to

pd.read_html(r.text)