Data is missing while scraping using beautifulsoup4

Question

Actually I'm a newbie to the parsing stuff with Python Beautifulsoup4. I was scraping this website. I need Current Price Per Mil on the front page. I already spent 3 hours with this. While looking for the solution on internet. I got to know that there is a library PyQT4 that can mimic like a web browser and load the content and then once it's done with loading you can extract ur required data. But I got crashed. Used this approach to collect the data in raw text format. I tried other approaches too. def parseMe(url): soup = getContent(url) source_code = requests.get(url) plaint_text = source_code.text soup = BeautifulSoup(plaint_text, 'html.parser') osrs_text = soup.find('div', class_='col-md-12 text-center') print(osrs_text.encode('utf-8')) Please have a look on this image. I think the problem is with ::before and ::after tags. They appear once the page get loaded. Any help will be highly appreciated. JavaScript questions and answers, JavaScript questions pdf, JavaScript question bank, JavaScript questions and answers pdf, mcq on JavaScript pdf, JavaScript questions and solutions, JavaScript mcq Test , Interview JavaScript questions, JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)

Answer 1

You should use selenium instead of `requests: from selenium import webdriver from bs4 import BeautifulSoup def parse(url): driver = webdriver.Chrome('D:\Programming\utilities\chromedriver.exe') driver.get('https://boglagold.com/buy-runescape-gold/') soup = BeautifulSoup(driver.page_source) return soup.find('h4', {'id': 'curr-price-per-mil-text'}).text parse() Output: 'Current Price Per Mil: 0.80USD' The reason is that the value of that element is obtained through JavaScript, which requests can't handle. This particular snippet of code uses the Chrome driver; if you prefer, you can use the Firefox/some other browser equivalent (you will need to install the selenium library and look for the Chrome driver yourself).