The World’s Largest Online Community for Developers
I am trying to create a web scraper to gather data for a science fair project. I am creating a map of air quality for Calgary. I am getting my data off of CRAZ website When I inspect the code in Chrome it shows the data I want.
My code is below the image:
from bs4 import BeautifulSoup as Bsp import requests as r page_ce = r.get('https://craz.ca/monitoring/calgary-central/') soup = content = Bsp(page_ce.content, 'html.parser') ce_d = soup.find(id='mainTable') print(ce_d)
To do this, add the "--headless" option. If your server is windows, add "--disable-gpu" as well. If your server is linux, add "--no-sandbox".
from selenium import webdriver from selenium.webdriver.chrome.options import Options chrome_options = Options() # chrome_options.add_argument("--disable-gpu") # windows only # chrome_options.add_argument("--no-sandbox) # linux only chrome_options.add_argument("--headless") driver = webdriver.Chrome(options=chrome_options) driver.get("https://craz.ca/monitoring/calgary-central/")
Then you can scrape with selenium.