Answer a question

Currently I have Selenium hooked up to python to scrape a webpage. I found out that the page actually pulls data from a JSON API, and I can get a JSON response as long as I'm logged in to the page.

However, my approach of getting that response into python seems a bit junky; I select text enclosed in <pre> tags and use python's json package to parse the data like so:

import json
from selenium import webdriver

url = 'http://jsonplaceholder.typicode.com/posts/1'
driver = webdriver.Chrome()
driver.get(url)
json_text = driver.find_element_by_css_selector('pre').get_attribute('innerText')
json_response = json.loads(json_text)

The only reason I need to select within <pre> tags at all is because when JSON appears in Chrome, it comes formatted like this:

<html>
<head></head>
<body>
<pre style="word-wrap: break-word; white-space: pre-wrap;">{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}</pre>
</body>
</html>

And the only reason I need to do this inside selenium at all is because I need to be logged into the website in order to get a response. Otherwise I get a 401 and no data.

Answers

You can find the pre element and get it's text, then load it via json.loads():

import json 

pre = driver.find_element_by_tag_name("pre").text
data = json.loads(pre)
print(data)

Also, if this does not work as-is, and, as suggested by @Skandix in comments, prepend view-source: to your url.

Also, you may avoid using selenium to get the desired JSON data and transfer the cookies from selenium to requests to keep "staying logged in", see:

  • How do I load session and cookies from Selenium browser to requests library in Python?
# python
Logo
Python

Python社区为您提供最前沿的新闻资讯和知识内容

更多推荐

cover

求助！为什么用InsCode部署会出现无限重定向？

avatar Python

如何重塑熊猫。系列

问题:如何重塑熊猫。系列 在我看来,它就像 pandas.Series 中的一个错误。 a = pd.Series([1,2,3,4]) b = a.reshape(2,2) b b 有类型 Series 但无法显示,最后一条语句给出异常,非常冗长,最后一行是“TypeError: %d format: a number is required, not numpy.ndarray”。 b.sha

avatar Python

在哪里可以找到有关 Keras 中默认权重初始化器的文档? [复制]

问题:在哪里可以找到有关 Keras 中默认权重初始化器的文档? [复制] 我刚刚在这里](https://keras.io/initializers/)中阅读了有关[中的 Keras 权重初始化器的信息。在文档中,只介绍了不同的初始化程序。如: model.add(Dense(64, kernel_initializer='random_normal')) 当我没有指定kernel_initia

avatar Python