LaVOZs

The World’s Largest Online Community for Developers

'; python - How can I scrape hidden elements without using selenium? - LavOzs.Com

I am trying to create a web scraper to gather data for a science fair project. I am creating a map of air quality for Calgary. I am getting my data off of CRAZ website When I inspect the code in Chrome it shows the data I want.

My code is below the image:

from bs4 import BeautifulSoup as Bsp
import requests as r

page_ce = r.get('https://craz.ca/monitoring/calgary-central/')
soup = content = Bsp(page_ce.content, 'html.parser')
ce_d = soup.find(id='mainTable')


print(ce_d)

You can use selenium without opening a window. Selenium executes javascript so you will be able to scrape.

To do this, add the "--headless" option. If your server is windows, add "--disable-gpu" as well. If your server is linux, add "--no-sandbox".

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
# chrome_options.add_argument("--disable-gpu") # windows only
# chrome_options.add_argument("--no-sandbox) # linux only
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(options=chrome_options)

driver.get("https://craz.ca/monitoring/calgary-central/")

Then you can scrape with selenium.

Related
How do I check whether a file exists without exceptions?
How can I safely create a nested directory?
How to print without newline or space?
How can I know which radio button is selected via jQuery?
How do I modify the URL without reloading the page?
How to move an element into another element?
How do I get the number of elements in a list?
How can I set the default value for an HTML <select> element?