Answer a question

The link below has the table I want to scrape. I have tried in the past, but it has a strange format and I am not sure if they are preventing people from scraping.

This is what I have tried

req_url = "https://fantasy.espn.com/basketball/players/add?leagueId=133998"
req = requests.get(req_url)
soup = BeautifulSoup(req.content, 'html.parser')
table = soup.find('table', {'id':'fitt-analytics'})
df = pd.read_html(str(table))[0]
df.head()

But it has given me an error stating no tables were found.

https://fantasy.espn.com/basketball/players/add?leagueId=133998

Answers

Well... you could either explore selenium because you won't get that with BeautifulSoup (the table is dynamically rendered by JS) or try to rebuild the table from the API.

Here's a working example that fetches players' names and their fantasy PTS total & average.

import json

import requests
from tabulate import tabulate

filter_string = {
  "players": {
    "filterStatus": {
      "value": [
        "FREEAGENT",
        "WAIVERS"
      ]
    },
    "filterSlotIds": {
      "value": [
        0,
        1,
        2,
        3,
        4,
        5,
        6,
        7,
        8,
        9,
        10,
        11
      ]
    },
    "filterRanksForScoringPeriodIds": {
      "value": [
        1
      ]
    },
    "sortPercOwned": {
      "sortPriority": 2,
      "sortAsc": False
    },
    "sortDraftRanks": {
      "sortPriority": 100,
      "sortAsc": True,
      "value": "STANDARD"
    },
    "limit": 50,
    "offset": 0,
    "filterStatsForTopScoringPeriodIds": {
      "value": 5,
      "additionalValue": [
        "002021",
        "102021",
        "002020",
        "012021",
        "022021",
        "032021",
        "042021"
      ]
    }
  }
}

headers = {
    "x-fantasy-filter": json.dumps(filter_string)
}

api_url = "https://fantasy.espn.com/apis/v3/games/fba/seasons/2021/segments/0/leagues/133998?view=kona_player_info"
response = requests.get(api_url, headers=headers).json()

sample_table = []
for p in response["players"]:
    try:
        sample_table.append(
            [
                p['player']['fullName'],
                p['player']['stats'][9]['appliedTotal'],
                round(p['player']['stats'][9]['appliedAverage'], 2),
            ]
        )
    except (KeyError, IndexError):
        continue

print(tabulate(sample_table, headers=["Player", "Total", "Average"]))

Output:

Player                     Total    Average
-----------------------  -------  ---------
Giannis Antetokounmpo       4173      62.28
Anthony Davis               3518      57.67
Nikola Jokic                3728      54.82
Luka Doncic                 4172      63.21
LeBron James                3474      55.14
James Harden                4078      60.87
Stephen Curry               3506      56.55
Devin Booker                3491      52.89
Damian Lillard              3892      59.88
Trae Young                  3575      55.86
Kevin Durant                3339      52.17
Joel Embiid                 3375      54.44
Kawhi Leonard               3131      53.98
Jayson Tatum                3559      55.61
Jimmy Butler                2906      44.03
Russell Westbrook           3324      52.76
Kyrie Irving                3093      51.55
Donovan Mitchell            3383      49.75
Karl-Anthony Towns          3888      58.03
Pascal Siakam               3418      51.79
Bradley Beal                3571      53.3
Brandon Ingram              3306      50.86
Zion Williamson             3164      51.87
Bam Adebayo                 2647      38.93
Paul George                 3281      51.27
Ja Morant                   2946      45.32
Jamal Murray                3084      47.45
Andre Drummond              2949      44.01
DeMar DeRozan               2876      42.29
Khris Middleton             3064      47.88
Ben Simmons                 2825      44.14
Rudy Gobert                 2554      39.91
Jrue Holiday                2931      45.09
Deandre Ayton               2944      46
Chris Paul                  2561      41.31
Nikola Vucevic              2930      47.26
Shai Gilgeous-Alexander     3081      48.14
D'Angelo Russell            3127      50.44
Zach LaVine                 2944      49.07
Domantas Sabonis            2593      43.22
Tobias Harris               3213      46.57
John Collins                2584      41.68
Kyle Lowry                  2619      40.29
CJ McCollum                 2906      44.03
De'Aaron Fox                2765      43.89
Fred VanVleet               2657      40.88
T.J. Warren                 2838      44.34
LaMarcus Aldridge           2404      39.41
Kristaps Porzingis          2694      49.89
Kemba Walker                2398      42.82

Bonus:

If you want to "paginate" the API just change the offset value to 50. That's your page 2.