LaVOZs

The World’s Largest Online Community for Developers

'; python - How to scrape non HTML tags using BeautifulSoup - LavOzs.Com

I'm trying to scarpe data from a website that has a tag as <a&#32;href="https: evisa.mfa.am "> for example, look at this website

Is there any way in BeautifulSoup to extract data from non html tags?

here's a snippet of the whole html page from the above link

<br/>2.&#32;Airlines&#32;must&#32;provide&#32;advance&#32;passenger&#32;information&#32;of&#32;scheduled&#32;arrival&#32;of&#32;nationals&#32;of&#32;Antigua&#32;and&#32;Barbuda&#32;and&#32;resident&#32;diplomats.&#32;<br/><br/><b>ARGENTINA</b>&#32;-&#32;published&#32;02.04.2020&#32;<br/>Passengers&#32;are&#32;not&#32;allowed&#32;to&#32;enter&#32;Argentina&#32;until&#32;12&#32;April&#32;2020.<br/><br/><b>ARMENIA</b>&#32;-&#32;published&#32;22.03.2020&#32;<br/>1.&#32;Nationals&#32;of&#32;China&#32;(People's&#32;Rep.)&#32;with&#32;a&#32;normal&#32;passport&#32;are&#32;no&#32;longer&#32;visa&#32;exempt.&#32;<br/>2.&#32;Nationals&#32;of&#32;Iran&#32;can&#32;no&#32;longer&#32;obtain&#32;a&#32;visa&#32;on&#32;arrival.&#32;They&#32;must&#32;obtain&#32;a&#32;visa&#32;or&#32;an&#32;e-visa&#32;prior&#32;to&#32;their&#32;arrival&#32;in&#32;Armenia.&#32;The&#32;e-visa&#32;can&#32;be&#32;obtained&#32;at&#32;<a&#32;href="https://evisa.mfa.am/">https://evisa.mfa.am/</a>&#32;<br/>3.&#32;Passengers&#32;who&#32;have&#32;been&#32;in&#32;Austria,&#32;Belgium,&#32;China&#32;(People's&#32;Rep.),&#32;Denmark,&#32;France,&#32;Germany,&#32;Iran,&#32;Italy,&#32;Japan,&#32;Korea&#32;(Rep.),&#32;Netherlands,&#32;Norway,&#32;Spain,&#32;Sweden,&#32;Switzerland&#32;or&#32;United&#32;Kingdom&#32;in&#32;the&#32;past&#32;14&#32;days&#32;are&#32;not&#32;allowed&#32;to&#32;enter&#32;Armenia.<br/>-&#32;This&#32;does&#32;not&#32;apply&#32;to&#32;nationals&#32;or&#32;residents&#32;of&#32;Armenia.<br/>-&#32;This&#32;does&#32;not&#32;apply&#32;to&#32;spouses&#32;or&#32;children&#32;of&#32;nationals&#32;of&#32;Armenia.<br/>-&#32;This&#32;does&#32;not&#32;apply&#32;to&#32;employees&#32;of&#32;foreign&#32;diplomatic&#32;missions&#32;and&#32;consular&#32;institutions.<br/>-&#32;This&#32;does&#32;not&#32;apply&#32;to&#32;representations&#32;of&#32;official&#32;international&#32;missions&#32;or&#32;organizations.<br/>4.&#32;Nationals&#32;of&#32;Armenia&#32;who&#32;have&#32;been&#32;in&#32;Austria,&#32;Belgium,&#32;China&#32;(People's&#32;Rep.),&#32;Denmark,&#32;France,&#32;Germany,&#32;Iran,&#32;Italy,&#32;Japan,&#32;Korea&#32;(Rep.),&#32;Netherlands,&#32;Norway,&#32;Spain,&#32;Sweden,&#32;Switzerland&#32;or&#32;United&#32;Kingdom&#32;in&#32;the&#32;past&#32;14&#32;days&#32;must&#32;undergo&#32;14-days&#32;of&#32;quarantine&#32;or&#32;self-isolation&#32;regime.

That's called AMP chars, you can have a look here to understand what it is.

Don't use html.parser. Just use a real parser such as lxml or html5lib

from bs4 import BeautifulSoup
import requests

r = requests.get(
    "https://www.iatatravelcentre.com/international-travel-document-news/1580226297.htm")
soup = BeautifulSoup(r.content, 'html5lib')


print(soup.prettify())

If you parse the webpage using requests remove the part of the tag that is wrong, you can pass that to BeautifulSoup.

In the following I'm replacing &#32; because it is just a HTML representation of a space.

import requests
url = 'https://www.iatatravelcentre.com/international-travel-document-news/1580226297.htm'

response = requests.get(url)
content = response.text.replace('&#32;',' ')

from bs4 import BeautifulSoup
soup = BeautifulSoup(content, 'html.parser')

now you can use BeautifulSoup as you use too.

You have to analyse your html code before post questions .

Now Try to get your URL

from bs4 import BeautifulSoup

with open("test.html","r") as f:
    page = f.read()
    soup = BeautifulSoup(page, 'html.parser')
    url = soup.findAll("a&#32;href=\"https:")
    print(url)
Related
How do I merge two dictionaries in a single expression in Python?
How do I check if a list is empty?
How do I check whether a file exists without exceptions?
How can I safely create a nested directory?
How to get the current time in Python
How do I sort a dictionary by value?
How to make a chain of function decorators?
How to make a flat list out of list of lists?
How do I list all files of a directory?
Cannot display HTML string