Skip to main content

Scraping HTML Content using Python

To Scrape Data using python we are using BeautifulSoup python Package

!pip install beautifulsoup4

As a first step we have to import the packages and html page that we need to scrape. In here I have used some static HTML content which was customized to scrape the data.
#imports

import requests
from bs4 import BeautifulSoup



#html HTML Sample Doing Data Science with Python

Doing Data Science with Python

Author: Eranda Kodagoda

This will help to perform various data science activitied using python

Modules

Title Duration in minutes
Getting Started 20
Setting Up Environment 40
Extracting Data 30
Exploring and Processing Data 45
Building Productive Model 45
To View the HTML using beautifulsoup we can use below code-lines and execute on python executor

from IPython.core.display import display, HTML
display(HTML(html_string))

To Print the HTML using beautifulsoup we can use below code-lines and execute on python executor

ps=BeautifulSoup(html_string,"lxml")
print(ps)

Find and extract content by HTML tags

#use Parameter name to select by tag name

body=ps.find(name="body")
print(body)

Extract the value referred in the HTML tag


# use text attribute to get the content of the tag

print(body.find(name="h1").text)

# find first element by using .text its restricting the HTML tag

print(body.find(name="p").text)

# find all elements

print(body.findAll(name="p"))

# get only the contents of "p" elements Loop through each element

for p in body.findAll(name="p"):
print(p.text)

# add attributes in selection process

print(body.find(name="p", attrs={"id":"description"}))

#get the data contain in the table

body=ps.find(name="body")
module_table=body.find(name="table",attrs={"id":"module"})
for row in module_table.findAll(name="tr") [1:]:
title=row.findAll(name="td")[0].text
duration=row.findAll(name="td") [1].text
print (title,duration)

Comments

Popular posts from this blog

SQL Blocking query

SELECT s.session_id, r.status, r.blocking_session_id 'Blk by' , r.wait_type, wait_resource, r.wait_time / ( 1000.0 ) 'Wait Sec' , r.cpu_time, r.logical_reads, r.reads, r.writes, r.total_elapsed_time / ( 1000.0 ) 'Elaps Sec' , Substring (st. TEXT ,(r.statement_start_offset / 2 ) + 1 , (( CASE r.statement_end_offset WHEN - 1 THEN Datalength (st. TEXT ) ELSE r.statement_end_offset END - r.statement_start_offset) / 2 ) + 1 ) AS statement_text, Coalesce ( Quotename ( Db_name (st.dbid)) + N '.' + Quotename (Object_sch...

Identify Missing PrimaryKeys in tables

-- DEFINE &TableOwnerPattern = &1 -- DEFINE &TableNamePattern = &2 -- COLUMN owner FORMAT A25 HEADING 'Owner' -- COLUMN table_name FORMAT A30 HEADING 'Table' -- COLUMN constraint_name FORMAT A30 HEADING 'FK constraint' -- COLUMN column_list FORMAT A62 HEADING 'FK columns(s)' WORD WRAP WITH constr AS ( SELECT c.owner, c.table_name, c.constraint_name, listagg(col.column_name, ', ' ) within group ( ORDER BY col.column_name) column_list_alphabetic, listagg(col.column_name, ', ' ) within group ( ORDER BY col.position) column_list FROM dba_constraints c JOIN dba_cons_columns col ON ( col.owner = c.owner AND col.constraint_name = c.constraint_name AND col.table_name = c.table_name ) WHERE c.constraint_type = 'R' AND c.owner like '&TableOwnerPattern' AND c.table_name like '&TableNamePatte...