Answer a question

I am trying to programmatically fetch the list of companies included in NASDAQ-100. I tried scraping Nasdaq-100-Index-Components using Beautiful Soup - bs4, but so far without much success.

How can I get this list (tickers and company names)?

s = requests.Session()
s.headers.update(
    {
        "Accept-Language":"en-US,en;q=0.9", 
    "Accept-Encoding":"gzip, deflate, br",
    "User-Agent":"Java-http-client/"
    }
)
r = s.get("https://www.nasdaq.com/market-activity/quotes/nasdaq-ndx-index")
soup = BeautifulSoup(r.content, "html.parser")
res = json.loads([x for x in soup.find("script", {"type": "application/json"})][0])

This only returns a very limited list and I suspect that this naive scraping doesn't really get all the data.

Answers

As data is dynamic generated go to chrome developer mode to Network tab and find data by searching in box and refresh website now you can find link which content company list data as json data

import requests
headers={"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"}
res=requests.get("https://api.nasdaq.com/api/quote/list-type/nasdaq100",headers=headers)
main_data=res.json()['data']['data']['rows']


for i in range(len(main_data)):
    print(main_data[i]['companyName'])

Output:

Activision Blizzard, Inc. Common Stock
Adobe Inc. Common Stock
Advanced Micro Devices, Inc. Common Stock
Align Technology, Inc. Common Stock
..

Image