Answer a question

I tried beautifulsoup to get a list of elements and then I want the xpath so that I can click it, the button that I want to click is arbitrary it can be anywhere depending on the data.

Eg.

from bs4 import BeautifulSoup as bs

soup=bs(driver.page_source,'html.parser') #used selenium
soup.find('div', class_='dataTable').find_all('span',class_='blue-icon')

Class name can be anything I just made up one but I want to know the main thing how can I get the xpath. It gave me a list of two span tags actually these span tags are a button that can be anywhere in the table and sometimes they won't so I want to if I can use it to find it's xpath so that I can use selenium to click the button whenever I see this span tag. If you want to have a look at the website: https://www.screener.in/company/GRANULES/consolidated/ Now there are some tables that have plus (+) symbol after clicking it it gives extra info so I want to scrape that too, but without clicking it I can't scrape it, so this is the method I thought of but I could not get to locate the symbol. If anyone can help me with this. There can be any other methods to do this that would help too.

Thanks

Answers

Clicking buttons:

To interact with the buttons you need to automate a browser e.g. with selenium.

Targeting a specific button:

You will notice the expand + buttons have an onclick attribute which triggers a function call with various arguments. It may be possible to replicate this with requests; However, I am going to focus on the fact you can target the particular + button by that attribute, and its value, based upon the unique arguments, passed to the function, using a css attribute = value selector with * contains operator.

You can see here the function that is called with Expenses and Quarters arguments in line with the part of the table to be expanded:

I target that onclick attribute by the argument values, escaping the ' as I wrapping with outer '. I am looking for the onclick containing those values.

Expand all +

To find all + buttons, rather than target a specific one, change the method call to find_elements_by_css_selector and change the css selector to instead look at the function rather than the arguments i.e. [onclick*="Company.showSchedule"].

You will need to loop the matched webElements collection (list) to click individual elements within the loop.

Py:

from selenium import webdriver

d = webdriver.Chrome()
d.get('https://www.screener.in/company/GRANULES/consolidated/')
d.find_element_by_css_selector('[onclick*="\'Expenses\', \'quarters\'"]').click()