Answer a question

What I'm trying to do is acquire a product ID from a script tag inside an HTML document. Unfortunately, StockX doesn't offer a public API, so I have to scrape the data from an HTML document. Here are my attempts at it (both work):

Attempt 1

import requests

PRODUCT_URL = 'https://stockx.com/supreme-steiff-bear-heather-grey'
HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}

response = requests.get(url=PRODUCT_URL, headers=HEADERS).text
PRODUCT_ID = response[response.find('"product":{"id":"')+17:].partition('"')[0]
PRODUCT_NAME = response[response.find('<title>')+7:].partition('<')[0]

Attempt 2

from bs4 import BeautifulSoup
import requests

# Gets HTML document
PRODUCT_URL = 'https://stockx.com/supreme-steiff-bear-heather-grey'
HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}
html_content = requests.get(url=PRODUCT_URL, headers=HEADERS)

# Make BeautifulSoup parser from HTML document
soup = BeautifulSoup(html_content.text, 'html.parser')

# Get product name
PRODUCT_NAME = soup.title.text

# Get script tag data with product ID
js_content = soup.find_all('script', type='text/javascript')[9].text
PRODUCT_ID = js_content[50:86]

print(PRODUCT_ID)

Output: 884861d2-abe6-4b09-90ff-c8ad1967ac8c

However, I feel like there is a better approach to this problem instead of just "hard-coding" in where to find the ID.

If you view the page source of the product URL and do a search for "product":{"id":, you will find that the ID is inside a nested dictionary that is assigned to an object and inside a tag.

Is there any better way to go about obtaining the product ID from an HTML document?

EDIT: Here is the content of html_content: https://gist.github.com/leecharles50/9b6b11fb458767cabcfc0ed4f961984d

Answers

My first idea was to parse the JavaScript inside the tag. There is a package called slimit that can do this. See for example this answer.

However, in your case there is an even easier solution. I searched the DOM for the id you gave (884861d2-abe6-4b09-90ff-c8ad1967ac8) and found an occurrence inside the following tag:

<script type="application/ld+json">
    {
        [...]
        "sku" : "884861d2-abe6-4b09-90ff-c8ad1967ac8c",
        [...]
    }
</script>

which contains valid JSON. Simply find the tag with BeautifulSoup:

tag = soup('script', {'type': 'application/ld+json'})[-1]

and decode the JSON within:

import json
product_id = json.loads(tag.text)['sku']