nodes/EricRollei-Download_Tools
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
main
EricRollei-Download_Tools/Docs/scrapling_docs.md
T

4341 lines
190 KiB
Markdown
Raw Permalink Blame History

Key Features¶
Fetch websites as you prefer with async support¶
HTTP Requests: Fast and stealthy HTTP requests with the Fetcher class.
Dynamic Loading & Automation: Fetch dynamic websites with the PlayWrightFetcher class through your real browser, Scrapling's stealth mode, Playwright's Chromium browser, or NSTbrowser's browserless!
Anti-bot Protections Bypass: Easily bypass protections with the StealthyFetcher and PlayWrightFetcher classes.
Easy Scraping¶
Smart Element Tracking: Relocate elements after website changes using an intelligent similarity system and integrated storage.
Flexible Selection: CSS selectors, XPath selectors, filters-based search, text search, regex search, and more.
Find Similar Elements: Automatically locate elements similar to the element you found!
Smart Content Scraping: Extract data from multiple websites without specific selectors using Scrapling powerful features.
High Performance¶
Lightning Fast: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries.
Memory Efficient: Optimized data structures for minimal memory footprint.
Fast JSON serialization: 10x faster than standard library.
Developer Friendly¶
Powerful Navigation API: Easy DOM traversal in all directions.
Rich Text Processing: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries that use less memory than standard dictionaries with added methods.
Auto Selectors Generation: Generate robust short and full CSS/XPath selectors for any element.
Familiar API: Similar to Scrapy/BeautifulSoup and the same CSS pseudo-elements used in Scrapy.
Type hints: Complete type/doc-strings coverage for future-proofing and best autocompletion support.
Overview
We will start by quickly reviewing the parsing capabilities. Then, we will fetch websites with custom browsers, make requests, and parse the response.
Here's an HTML document generated by ChatGPT we will be using as an example throughout this page:
<html>
<head>
<title>Complex Web Page</title>
<style>
.hidden { display: none; }
</style>
</head>
<body>
<header>
<nav>
<ul>
<li> <a href="#home">Home</a> </li>
<li> <a href="#about">About</a> </li>
<li> <a href="#contact">Contact</a> </li>
</ul>
</nav>
</header>
<main>
<section id="products" schema='{"jsonable": "data"}'>
<h2>Products</h2>
<div class="product-list">
<article class="product" data-id="1">
<h3>Product 1</h3>
<p class="description">This is product 1</p>
<span class="price">$10.99</span>
<div class="hidden stock">In stock: 5</div>
</article>
<article class="product" data-id="2">
<h3>Product 2</h3>
<p class="description">This is product 2</p>
<span class="price">$20.99</span>
<div class="hidden stock">In stock: 3</div>
</article>
<article class="product" data-id="3">
<h3>Product 3</h3>
<p class="description">This is product 3</p>
<span class="price">$15.99</span>
<div class="hidden stock">Out of stock</div>
</article>
</div>
</section>
<section id="reviews">
<h2>Customer Reviews</h2>
<div class="review-list">
<div class="review" data-rating="5">
<p class="review-text">Great product!</p>
<span class="reviewer">John Doe</span>
</div>
<div class="review" data-rating="4">
<p class="review-text">Good value for money.</p>
<span class="reviewer">Jane Smith</span>
</div>
</div>
</section>
</main>
<script id="page-data" type="application/json">
{
"lastUpdated": "2024-09-22T10:30:00Z",
"totalProducts": 3
}
</script>
</body>
</html>
Starting with loading raw HTML above like this
from scrapling.parser import Adaptor
page = Adaptor(html_doc)
page # <data='<html><head><title>Complex Web Page</tit...'>
Get all text content on the page recursively
page.get_all_text(ignore_tags=('script', 'style'))
# 'Complex Web Page\nHome\nAbout\nContact\nProducts\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock\nCustomer Reviews\nGreat product!\nJohn Doe\nGood value for money.\nJane Smith'
Finding elements¶
If there's an element you want to find on the page, you will! Your creativity level is the only limitation!
Finding the first HTML section element
section_element = page.find('section')
# <data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>
Find all section elements
section_elements = page.find_all('section')
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
Find all section elements whose id attribute value is products
section_elements = page.find_all('section', {'id':"products"})
# Same as
section_elements = page.find_all('section', id="products")
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>]
Find all section elements that its id attribute value contains product
section_elements = page.find_all('section', {'id*':"product"})
Find all h3 elements whose text content matches this regex Product \d
page.find_all('h3', re.compile(r'Product \d'))
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
Find all h3 and h2 elements whose text content matches regex Product only
page.find_all(['h3', 'h2'], re.compile(r'Product'))
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>, <data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]
Find all elements that its text content matches exactly Products (Whitespaces are not taken into consideration)
page.find_by_text('Products', first_match=False)
# [<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]
Or find all elements whose text content matches regex Product \d
page.find_by_regex(r'Product \d', first_match=False)
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
Find all elements that are similar to the element you want
target_element = page.find_by_regex(r'Product \d', first_match=True)
# <data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>
target_element.find_similar()
# [<data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
Find the first element that matches a CSS selector
page.css_first('.product-list [data-id="1"]')
# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
Find all elements that match a CSS selector
page.css('.product-list article')
# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
Find the first element that matches an XPath selector
page.xpath_first("//*[@id='products']/div/article")
# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
Find all elements that match an XPath selector
page.xpath("//*[@id='products']/div/article")
# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
With this, we just scratched the surface of these functions; more advanced options with these selection methods are shown later.
Accessing elements' data¶
It's as simple as
>>> section_element.tag
'section'
>>> print(section_element.attrib)
{'id': 'products', 'schema': '{"jsonable": "data"}'}
>>> section_element.attrib['schema'].json() # If an attribute value can be converted to json, then use `.json()` to convert it
{'jsonable': 'data'}
>>> section_element.text # Direct text content
''
>>> section_element.get_all_text() # All text content recursively
'Products\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'
>>> section_element.html_content # The HTML content of the element
'<section id="products" schema=\'{"jsonable": "data"}\'><h2>Products</h2>\n <div class="product-list">\n <article class="product" data-id="1"><h3>Product 1</h3>\n <p class="description">This is product 1</p>\n <span class="price">$10.99</span>\n <div class="hidden stock">In stock: 5</div>\n </article><article class="product" data-id="2"><h3>Product 2</h3>\n <p class="description">This is product 2</p>\n <span class="price">$20.99</span>\n <div class="hidden stock">In stock: 3</div>\n </article><article class="product" data-id="3"><h3>Product 3</h3>\n <p class="description">This is product 3</p>\n <span class="price">$15.99</span>\n <div class="hidden stock">Out of stock</div>\n </article></div>\n </section>'
>>> print(section_element.prettify()) # The prettified version
'''
<section id="products" schema='{"jsonable": "data"}'><h2>Products</h2>
<div class="product-list">
<article class="product" data-id="1"><h3>Product 1</h3>
<p class="description">This is product 1</p>
<span class="price">$10.99</span>
<div class="hidden stock">In stock: 5</div>
</article><article class="product" data-id="2"><h3>Product 2</h3>
<p class="description">This is product 2</p>
<span class="price">$20.99</span>
<div class="hidden stock">In stock: 3</div>
</article><article class="product" data-id="3"><h3>Product 3</h3>
<p class="description">This is product 3</p>
<span class="price">$15.99</span>
<div class="hidden stock">Out of stock</div>
</article>
</div>
</section>
'''
>>> section_element.path # All the ancestors in the DOM tree of this element
[<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>,
<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>,
<data='<html><head><title>Complex Web Page</tit...'>]
>>> section_element.generate_css_selector
'#products'
>>> section_element.generate_full_css_selector
'body > main > #products > #products'
>>> section_element.generate_xpath_selector
"//*[@id='products']"
>>> section_element.generate_full_xpath_selector
"//body/main/*[@id='products']"
Navigation¶
Using the elements we found above
>>> section_element.parent
<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>
>>> section_element.parent.tag
'main'
>>> section_element.parent.parent.tag
'body'
>>> section_element.children
[<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>,
<data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>]
>>> section_element.siblings
[<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
>>> section_element.next # gets the next element, the same logic applies to `quote.previous`
<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>
>>> section_element.children.css('h2::text')
['Products']
>>> page.css_first('[data-id="1"]').has_class('product')
True
If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element like the one below
for ancestor in quote.iterancestors():
# do something with it...
You can search for a specific ancestor of an element that satisfies a function; all you need to do is to pass a function that takes an Adaptor object as an argument and return True if the condition satisfies or False otherwise like below:
>>> section_element.find_ancestor(lambda ancestor: ancestor.css('nav'))
<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>
Fetching websites¶
Instead of passing the raw HTML to Scrapling, you can get a website's response directly through HTTP requests or by fetching it from browsers.
A fetcher is made for every use case.
HTTP Requests¶
For simple HTTP requests, there's a Fetcher class that can be imported as below:
from scrapling.fetchers import Fetcher
But that's class, so you will need to create an instance of the Fetcher first like this:
from scrapling.fetchers import Fetcher
fetcher = Fetcher()
page = fetcher.get('https://httpbin.org/get')
This is intended, and you will find it with all fetchers because there are settings you can pass to Fetcher() initialization, but more on this later.
If you are going to use the default settings anyway, you can do this instead for a cleaner approach:
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://httpbin.org/get')
With that out of the way, here's how to do all HTTP methods:
>>> from scrapling.fetchers import Fetcher
>>> page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
>>> page = Fetcher.put('https://httpbin.org/put', data={'key': 'value'})
>>> page = Fetcher.delete('https://httpbin.org/delete')
For Async requests, you will just replace the import like below:
>>> from scrapling.fetchers import AsyncFetcher
>>> page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
>>> page = await AsyncFetcher.put('https://httpbin.org/put', data={'key': 'value'})
>>> page = await AsyncFetcher.delete('https://httpbin.org/delete')
Note: You have the stealthy_headers argument, which, when enabled, makes requests to generate real browser headers and use them, including a referer header, as if this request came from Google's search of this URL's domain. It's enabled by default.
This is just the tip of this fetcher; check the full page from here
Dynamic loading¶
We have you covered if you deal with dynamic websites like most today!
The PlayWrightFetcher class provides many options to fetch/load websites' pages through browsers.
>>> from scrapling.fetchers import PlayWrightFetcher
>>> page = PlayWrightFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option
>>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'
>>> # The async version of fetch
>>> page = await PlayWrightFetcher.async_fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True)
>>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'
It's named like that because it's built on top of Playwright, and it currently provides 4 main run options that can be mixed as you want:
Vanilla Playwright without any modifications other than the ones you chose.
Stealthy Playwright with custom stealth mode explicitly written for it. It's not top-tier stealth mode but bypasses many online tests like Sannysoft's. Check out the StealthyFetcher class below for more advanced stealth mode.
Real browsers by passing the real_chrome argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
NSTBrowser's docker browserless option by passing the CDP URL and enabling nstbrowser_mode option.
Note: All requests done by this fetcher are waited by default for all javascript to be fully loaded and executed. In detail, it waits for the load and domcontentloaded load states to be reached; you can make it wait for the networkidle load state by passing 'network_idle=True', as you will see later.
Again, this is just the tip of this fetcher. Check out the full page from here for all details and the complete list of arguments.
Dynamic anti-protection loading¶
We also have you covered if you deal with dynamic websites with annoying anti-protections!
The StealthyFetcher class uses a modified Firefox browser called Camoufox, bypassing most anti-bot protections by default. Scrapling adds extra layers of flavors and configurations to further increase performance and undetectability.
>>> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection') # Running headless by default
>>> page.status == 200
True
>>> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection', humanize=True, os_randomize=True) # and the rest of arguments...
>>> # The async version of fetch
>>> page = await StealthyFetcher().async_fetch('https://www.browserscan.net/bot-detection')
>>> page.status == 200
True
Note: All requests done by this fetcher are waited by default for all javascript to be fully loaded and executed. In detail, it waits for the load and domcontentloaded load states to be reached; you can make it wait for the networkidle load state by passing 'network_idle=True', as you will see later.
Again, this is just the tip of this fetcher. Check out the full page from here for all details and the complete list of arguments.
That's Scrapling at a glance. If you want to learn more about it, continue to the next section.
Querying elements
Introduction¶
Scrapling currently supports parsing HTML pages exclusively, so it doesn't support XML feeds. This decision was made because the automatch feature won't work with XML, but that might change soon, so stay tuned :)
In Scrapling, there are 5 main ways to find elements:
CSS3 Selectors
XPath Selectors
Finding elements based on filters/conditions.
Finding elements whose content contains specific text
Finding elements whose content matches specific regex
Of course, there are other indirect ways to find elements with Scrapling, but here we will discuss the main ways in detail. We will also bring up one of the most remarkable features of Scrapling: the ability to find elements that are similar to the element you have; you can jump to that section directly from here.
If you are new to Web Scraping, have little to no experience writing selectors, and want to start quickly, I recommend you jump directly to learning the find/find_all methods from here.
CSS/XPath selectors¶
What are CSS selectors?¶
CSS is a language for applying styles to HTML documents. It defines selectors to associate those styles with specific HTML elements.
Scrapling implements CSS3 selectors as described in the W3C specification. CSS selectors support comes from cssselect, so it's better to read about which selectors are supported from cssselect and pseudo-functions/elements.
Also, Scrapling implements some non-standard pseudo-elements like:
To select text nodes, use ::text
To select attribute values, use ::attr(name) where name is the name of the attribute that you want the value of
In short, if you come from Scrapy/Parsel, you will find the same logic for selectors here to make it easier. No need to implement a stranger logic to the one that most of us are used to :)
To select elements with CSS selectors, you have the css and css_first methods. The latter is useful when you are interested in the first element it finds only, or if it's one element, etc., and the first when it's more than one, as it returns Adaptors.
What are XPath selectors?¶
XPath is a language for selecting nodes in XML documents, which can also be used with HTML. This [cheatsheet] (https://devhints.io/xpath) is a good resource for learning about XPath. Scrapling adds XPath selectors directly through LXML.
In short, it is the same situation as CSS Selectors; if you come from Scrapy/Parsel, you will find the same logic for selectors here. BUT Scrapling doesn't implement the XPath extension function has-class as Scrapy/Parsel—instead, there's the has_class method that you can use on elements returned for the same purpose.
To select elements with XPath selectors, you have the xpath and xpath_first methods. Again, these methods follow the same logic as the CSS selectors methods above.
Note that each method of css, css_first, xpath, and xpath_first have additional arguments, but we didn't explain them here as they are all about the automatch feature. The automatch feature will have its page later to be described in detail.
Selectors examples¶
Let's see some shared examples of using CSS and XPath Selectors.
Select all elements with the class product
products = page.css('.product')
products = page.xpath('//*[@class="product"]')
Note: The XPath one won't be accurate if there's another class; better rely on CSS for selecting by class
Select the first element with the class product
product = page.css_first('.product')
product = page.xpath_first('//*[@class="product"]')
Which would be the same as doing
product = page.css('.product')[0]
product = page.xpath('//*[@class="product"]')[0]
Get the text of the first element with the h1 tag name
title = page.css_first('h1::text')
title = page.xpath_first('//h1//text()')
Which is again the same as doing
title = page.css_first('h1').text
title = page.xpath_first('//h1').text
Get the href attribute of the first element with a tag name
link = page.css_first('a::attr(href)')
link = page.xpath_first('//a/@href')
Select the text of the first element with the h1 tag name, which contains 'Phone' and under an element with class 'product'
title = page.css_first('.product h1:contains("Phone")::text')
title = page.page.xpath_first('//*[@class="product"]//h1[contains(text(),"Phone")]/text()')
You can nest and chain selectors as you want, given that it returns results
page.css_first('.product').css_first('h1:contains("Phone")::text')
page.xpath_first('//*[@class="product"]').xpath_first('//h1[contains(text(),"Phone")]/text()')
page.xpath_first('//*[@class="product"]').css_first('h1:contains("Phone")::text')
Another example
All links that have 'image' in their 'href' attribute
links = page.css('a[href*="image"]')
links = page.xpath('//a[contains(@href, "image")]')
for index, link in enumerate(links):
link_value = link.attrib['href'] # Cleaner than link.css('::attr(href)')
link_text = link.text
print(f'Link number {index} points to this url {link_value} with text content as "{link_text}"')
Text-content selection¶
Scrapling provides the ability to select elements based on their direct text content, and you have two ways to do this:
Elements whose direct text content contains given text with many options through the find_by_text method.
Elements whose direct text content matches the given regex pattern with many options through the find_by_regex method.
What you can do with find_by_text can be done with find_by_regex if you are good enough with regular expressions (regex), but we are providing more options to make them easier for all users to access.
With find_by_text, you will pass the text as the first argument; with the find_by_regex method, the regex pattern is the first. Both methods share the following arguments:
first_match: If True (the default), the method used will return the first result it finds.
case_sensitive: If True, the case of the letters will be considered.
clean_match: If True, all whitespaces and consecutive spaces will be ignored while matching.
By default, Scrapling search for exact matching for the text you pass to find_by_text, so the text content of the wanted element have to be ONLY the text you inputted, but that's why it also has one extra argument, which is:
partial: If enabled, find_by_text will return elements that contain the input text. So it's not an exact match anymore
Note: The method find_by_regex can accept both regular strings and a compiled regex pattern as its first argument, as you will see in the upcoming examples.
Finding Similar Elements¶
One of the most remarkable new features that Scrapling puts on the table is the feature that allows the user to tell Scrapling to find elements similar to the element at hand. This feature inspiration came from the AutoScraper library, but here, it can be used on elements found by any method. Most likely, most of its usage would be after finding elements through text content like how AutoScraper works, so it would also be convenient to explain it here.
So, how does it work?
Imagine a scenario where you found a product by its title, for example, and you want to extract other products listed in the same table/container. With the element you have, you can simply call the method .find_similar() on it, and Scrapling will:
Find all page elements with the same tree depth as this element.
All found elements will be checked, and those without the same tag name, parent tag name, and grandparent tag name will be dropped.
Now we are sure (like 99% sure) that these elements are the ones we want, but as a last check, Scrapling will use fuzzy matching to drop the elements whose attributes don't look like the attributes of our element. There's a percentage to control this step, and I recommend you not play with it unless the default settings don't get the elements you want.
That's a lot of talking, I know, but I had to go deep, I will give examples of using this method in the next section, but first, these are the arguments that can be passed to this method:
similarity_threshold: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the attributes' values of both elements should be at least 20% similar. If you want to turn off this check (Step 3, basically), you can set this attribute to 0, but I recommend you read what other arguments do first.
ignore_attributes: The attribute names passed will be ignored while matching the attributes in the last step. The default value is ('href', 'src',) because URLs can change a lot between elements, making them unreliable.
match_text: If True, the element's text content will be considered when matching. Using this in normal cases is not recommended, but it depends.
Now, let's check out the examples below.
Examples¶
Let's see some shared examples of finding elements with raw text and regex.
I will use the Fetcher to clarify these examples, but it will be explained in detail later.
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://books.toscrape.com/index.html')
Find the first element whose text fully matches this text
>>> page.find_by_text('Tipping the Velvet')
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>
Combining it with page.urljoin to return the full URL from the relative href
>>> page.find_by_text('Tipping the Velvet').attrib['href']
'catalogue/tipping-the-velvet_999/index.html'
>>> page.urljoin(page.find_by_text('Tipping the Velvet').attrib['href'])
'https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html'
Get all matches if there are more (hence, it returned a list)
>>> page.find_by_text('Tipping the Velvet', first_match=False)
[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]
Get all elements that contain the word the (Partial matching)
>>> results = page.find_by_text('the', partial=True, first_match=False)
>>> [i.text for i in results]
['A Light in the ...',
'Tipping the Velvet',
'The Requiem Red',
'The Dirty Little Secrets ...',
'The Coming Woman: A ...',
'The Boys in the ...',
'The Black Maria',
'Mesaerion: The Best Science ...',
"It's Only the Himalayas"]
The search is case insensitive, so those results have The, not only the lowercase one the; let's limit the search to the elements with the only.
>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
>>> [i.text for i in results]
['A Light in the ...',
'Tipping the Velvet',
'The Boys in the ...',
"It's Only the Himalayas"]
Get the first element that its text content matches my price regex
>>> page.find_by_regex(r'£[\d\.]+')
<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>
>>> page.find_by_regex(r'£[\d\.]+').text
'£51.77'
It's the same if you pass the compiled regex as well; Scrapling will detect the input type and act upon that:
>>> import re
>>> regex = re.compile(r'£[\d\.]+')
>>> page.find_by_regex(regex)
<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>
>>> page.find_by_regex(regex).text
'£51.77'
Get all elements that match the regex
>>> page.find_by_regex(r'£[\d\.]+', first_match=False)
[<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>,
<data='<p class="price_color">£53.74</p>' parent='<div class="product_price"> <p class="pr...'>,
<data='<p class="price_color">£50.10</p>' parent='<div class="product_price"> <p class="pr...'>,
<data='<p class="price_color">£47.82</p>' parent='<div class="product_price"> <p class="pr...'>,
...]
And so on...
Find all elements similar to the current element in location and attributes. For our case, ignore the 'title' attribute while matching
>>> element = page.find_by_text('Tipping the Velvet')
>>> element.find_similar(ignore_attributes=['title'])
[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
<data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
<data='<a href="catalogue/sharp-objects_997/ind...' parent='<h3><a href="catalogue/sharp-objects_997...'>,
...]
Notice that the number of elements is 19, not 20, because the current element is not included in the results.
>>> len(element.find_similar(ignore_attributes=['title']))
19
Get the href attribute from all similar elements
>>> [
element.attrib['href']
for element in element.find_similar(ignore_attributes=['title'])
]
['catalogue/a-light-in-the-attic_1000/index.html',
'catalogue/soumission_998/index.html',
'catalogue/sharp-objects_997/index.html',
...]
To increase the complexity a little bit, let's say we want to get all books' data using that element as a starting point for some reason
>>> for product in element.parent.parent.find_similar():
print({
"name": product.css_first('h3 a::text'),
"price": product.css_first('.price_color').re_first(r'[\d\.]+'),
"stock": product.css('.availability::text')[-1].clean()
})
{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
...
Advanced examples¶
See more advanced or real-world examples using the find_similar method.
E-commerce Product Extraction
def extract_product_grid(page):
# Find the first product card
first_product = page.find_by_text('Add to Cart').find_ancestor(
lambda e: e.has_class('product-card')
)
# Find similar product cards
products = first_product.find_similar()
return [
{
'name': p.css_first('h3::text'),
'price': p.css_first('.price::text').re_first(r'\d+\.\d{2}'),
'stock': 'In stock' in p.text,
'rating': p.css_first('.rating').attrib.get('data-rating')
}
for p in products
]
Table Row Extraction
def extract_table_data(page):
# Find the first data row
first_row = page.css_first('table tbody tr')
# Find similar rows
rows = first_row.find_similar()
return [
{
'column1': row.css_first('td:nth-child(1)::text'),
'column2': row.css_first('td:nth-child(2)::text'),
'column3': row.css_first('td:nth-child(3)::text')
}
for row in rows
]
Form Field Extraction
def extract_form_fields(page):
# Find first form field container
first_field = page.css_first('input').find_ancestor(
lambda e: e.has_class('form-field')
)
# Find similar field containers
fields = first_field.find_similar()
return [
{
'label': f.css_first('label::text'),
'type': f.css_first('input').attrib.get('type'),
'required': 'required' in f.css_first('input').attrib
}
for f in fields
]
Extracting reviews from a website
def extract_reviews(page):
# Find first review
first_review = page.find_by_text('Great product!')
review_container = first_review.find_ancestor(
lambda e: e.has_class('review')
)
# Find similar reviews
all_reviews = review_container.find_similar()
return [
{
'text': r.css_first('.review-text::text'),
'rating': r.attrib.get('data-rating'),
'author': r.css_first('.reviewer::text')
}
for r in all_reviews
]
Filters-based searching¶
This search method might be arguably the best way to find elements in Scrapling because it is powerful and easier to learn for newcomers to Web Scraping than learning to write selectors.
Inspired by BeautifulSoup's find_all function, you can find elements using the find_all and find methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.
To be more specific:
Any string passed is considered a tag name.
Any iterable passed like List/Tuple/Set is considered an iterable of tag names.
Any dictionary is considered a mapping of HTML element(s) attribute names and attribute values.
Any regex patterns passed are used to filter elements by content like the find_by_regex method
Any functions passed are used to filter elements
Any keyword argument passed is considered as an HTML element attribute with its value.
It collects all passed arguments and keywords, and each filter passes its results to the following filter in a waterfall-like filtering system.
It filters all elements in the current page/element in the following order:
All elements with the passed tag name(s) get collected.
All elements that match all passed attribute(s) are collected; if a previous filter is used, then previously collected elements are filtered.
All elements that match all passed regex patterns are collected, or if previous filter(s) are used, then previously collected elements are filtered.
All elements that fulfill all passed function(s) are collected; if a previous filter(s) is used, then previously collected elements are filtered.
Notes:
As you probably understood, the filtering process always starts from the first filter it finds in the filtering order above. So, if no tag name(s) are passed but attributes are passed, the process starts from that layer, and so on.
The order in which you pass the arguments doesn't matter. The only order that's taken into consideration is the order explained above.
Check examples to clear any confusion :)
Examples¶
>>> from scrapling.fetchers import Fetcher
>>> page = Fetcher.get('https://quotes.toscrape.com/')
Find all elements with the tag name div.
>>> page.find_all('div')
[<data='<div class="container"> <div class="row...' parent='<body> <div class="container"> <div clas...'>,
<data='<div class="row header-box"> <div class=...' parent='<div class="container"> <div class="row...'>,
...]
Find all div elements with a class that equals quote.
>>> page.find_all('div', class_='quote')
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
Same as above.
>>> page.find_all('div', {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
Find all elements with a class that equals quote.
>>> page.find_all({'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
Find all div elements with a class that equals quote and contains the element .text, which contains the word 'world' in its content.
>>> page.find_all('div', {'class': 'quote'}, lambda e: "world" in e.css_first('.text::text'))
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>]
Find all elements that don't have children.
>>> page.find_all(lambda element: len(element.children) > 0)
[<data='<html lang="en"><head><meta charset="UTF...'>,
<data='<head><meta charset="UTF-8"><title>Quote...' parent='<html lang="en"><head><meta charset="UTF...'>,
<data='<body> <div class="container"> <div clas...' parent='<html lang="en"><head><meta charset="UTF...'>,
...]
Find all elements that contain the word 'world' in its content.
>>> page.find_all(lambda element: "world" in element.text)
[<data='<span class="text" itemprop="text">“The...' parent='<div class="quote" itemscope itemtype="h...'>,
<data='<a class="tag" href="/tag/world/page/1/"...' parent='<div class="tags"> Tags: <meta class="ke...'>]
Find all span elements that match the given regex
>>> page.find_all('span', re.compile(r'world'))
[<data='<span class="text" itemprop="text">“The...' parent='<div class="quote" itemscope itemtype="h...'>]
Find all div and span elements with class 'quote' (No span elements like that, so only div returned)
>>> page.find_all(['div', 'span'], {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
Mix things up
>>> page.find_all({'itemtype':"http://schema.org/CreativeWork"}, 'div').css('.author::text')
['Albert Einstein',
'J.K. Rowling',
...]
A bonus pro tip: Find all elements whose href attribute's value ends with the word 'Einstein'.
>>> page.find_all({'href$': 'Einstein'})
[<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>]
Another pro tip: Find all elements that its href attribute's value has '/author/' in it
>>> page.find_all({'href*': '/author/'})
[<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
<data='<a href="/author/J-K-Rowling">(about)</a...' parent='<span>by <small class="author" itemprop=...'>,
<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
...]
And so on...
Generating selectors¶
You can always generate CSS/XPath selectors for any element that can be reused here or anywhere else, and the most remarkable thing is that it doesn't matter what method you used to find that element!
Generate a short CSS selector for the url_element element (if possible, create a short one; otherwise, it's a full selector)
>>> url_element = page.find({'href*': '/author/'})
>>> url_element.generate_css_selector
'body > div > div:nth-of-type(2) > div > div > span:nth-of-type(2) > a'
Generate a full CSS selector for the url_element element from the start of the page
>>> url_element.generate_full_css_selector
'body > div > div:nth-of-type(2) > div > div > span:nth-of-type(2) > a'
Generate a short XPath selector for the url_element element (if possible, create a short one; otherwise, it's a full selector)
>>> url_element.generate_xpath_selector
'//body/div/div[2]/div/div/span[2]/a'
Generate a full XPath selector for the url_element element from the start of the page
>>> url_element.generate_full_xpath_selector
'//body/div/div[2]/div/div/span[2]/a'
Note:
When you tell Scrapling to create a short selector, it tries to find a unique element to use in generation as a stop point, like an element with an id attribute, but in our case, there wasn't any so that's why the short and the full selector will be the same.
Using selectors with regular expressions¶
Like in parsel/scrapy, you have the methods re and re_first for extracting data using regular expressions. However, unlike the former, these methods are in nearly all classes like Adaptor/Adaptors/TextHandler and TextHandlers, which means you can use them directly on the element even if you didn't select a text node.
We will have a deep look at it while explaining the TextHandler class, but in general, it works like the below examples:
>>> page.css_first('.price_color').re_first(r'[\d\.]+')
'51.77'
>>> page.css('.price_color').re_first(r'[\d\.]+')
'51.77'
>>> page.css('.price_color').re(r'[\d\.]+')
['51.77',
'53.74',
'50.10',
'47.82',
'54.23',
...]
>>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
['a-light-in-the-attic_1000',
'tipping-the-velvet_999',
'soumission_998',
'sharp-objects_997',
...]
>>> filtering_function = lambda e: e.parent.tag == 'h3' and e.parent.parent.has_class('product_pod') # As above selector
>>> page.find('a', filtering_function).attrib['href'].re(r'catalogue/(.*)/index.html')
['a-light-in-the-attic_1000']
>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
['tipping-the-velvet_999']
And so on. You get the idea. We will explain this in more detail on the next page while explaining the TextHandler class.
Main classes
Introduction¶
After exploring the various ways to select elements with Scrapling and related features, Let's take a step back and examine the Adaptor class generally and other objects to better understand the parsing engine.
The Adaptor class is the core parsing engine in Scrapling that provides HTML parsing and element selection capabilities. You can always import it with any of the following imports
from scrapling import Adaptor
from scrapling.parser import Adaptor
then use it directly as you already learned in the overview page
adaptor = Adaptor(
text='<html>...</html>',
url='https://example.com'
)
# Then select elements as you like
elements = adaptor.css('.product')
In Scrapling, the main object you deal with after passing an HTML source or fetching a website is, of course, an Adaptor object. Any operation you do, like selection, navigation, etc., will return either an Adaptor object or an Adaptors object, given that the result is element/elements from the page, not text or similar.
In other words, the main page is a Adaptor object, and the elements within are Adaptor objects, and so on. Any text, such as the text content inside elements or the text inside element attributes, is a TextHandler object, and the attributes of each element are stored as AttributesHandler. We will return to both objects later, so let's focus on the Adaptor object.
Adaptor¶
Arguments explained¶
The most important ones are text and body. Both are used to pass the HTML code you want to parse, but the first one accepts str, and the latter accepts bytes like how you used to do with parsel :)
Otherwise, you have the arguments url, auto_match, storage, and storage_args. All these arguments are settings used with the auto_match feature, and they don't make a difference if you are not going to use that feature, so just ignore them for now, and we will explain them in the automatch feature page.
Then you have the arguments for adjustments for parsing or adjusting/manipulating the HTML while the library parsing it:
encoding: This is the encoding that will be used while parsing the HTML. The default is UTF-8.
keep_comments: This tells the library whether to keep HTML comments while parsing the page. It's disabled by default, as it can mess up your scraping in many ways.
keep_cdata: Same logic as the HTML comments. cdata is removed by default for cleaner HTML. This also means when you check for the raw html content, you will find it doesn't have the cdata.
I have intended to ignore the arguments huge_tree and root to avoid making this page more complicated than needed. You may notice that I'm doing that a lot, and that's because it's something you don't need to know to use the library. The development section will cover these missing parts if you are that interested.
After that, for the main page and elements within, most properties don't get initialized until you use it like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
Properties¶
You have already seen much of this on the overview page, but don't worry if you didn't. We will review it more thoroughly using more advanced methods/usages. For clarity, the properties for traversal are separated below in the traversal section.
Let's say we are parsing this HTML page for simplicity:
<html>
<head>
<title>Some page</title>
</head>
<body>
<div class="product-list">
<article class="product" data-id="1">
<h3>Product 1</h3>
<p class="description">This is product 1</p>
<span class="price">$10.99</span>
<div class="hidden stock">In stock: 5</div>
</article>
<article class="product" data-id="2">
<h3>Product 2</h3>
<p class="description">This is product 2</p>
<span class="price">$20.99</span>
<div class="hidden stock">In stock: 3</div>
</article>
<article class="product" data-id="3">
<h3>Product 3</h3>
<p class="description">This is product 3</p>
<span class="price">$15.99</span>
<div class="hidden stock">Out of stock</div>
</article>
</div>
<script id="page-data" type="application/json">
{
"lastUpdated": "2024-09-22T10:30:00Z",
"totalProducts": 3
}
</script>
</body>
</html>
Load the page directly as shown before:
from scrapling import Adaptor
page = Adaptor(html_doc)
Get all text content on the page recursively
>>> page.get_all_text()
'Some page\n\n \n\n \nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'
Get the first article as explained before; we will use it as an example
article = page.find('article')
With the same logic, get all text content on the element recursively
>>> article.get_all_text()
'Product 1\nThis is product 1\n$10.99\nIn stock: 5'
But if you try to get the direct text content, it will be empty; notice the logic difference
>>> article.text
''
The get_all_text method has the following optional arguments:
separator: All strings collected will be concatenated using this separator. The default is '\n'
strip: If enabled, strings will be stripped before concatenation. Disabled by default.
ignore_tags: A tuple of all tag names you want to ignore in the final results. The default is ('script', 'style',).
valid_values: If enabled, the method will only collect elements with real values, so all elements with empty text content or only whitespaces will be ignored. It's enabled by default
By the way, the text returned here is not a standard string but a TextHandler; we will get to this in detail later, so if the text content can be serialized to JSON, then use .json() on it
>>> script = page.find('script')
>>> script.json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
Let's continue to get the element tag
>>> article.tag
'article'
If you used it on the page directly, you will find you are operating on the root html element
>>> page.tag
'html'
Now, I think I hammered the (page/element) idea, so I won't return to it again.
Getting the attributes of the element
>>> print(article.attrib)
{'class': 'product', 'data-id': '1'}
Get the HTML content of the element
>>> article.html_content
'<article class="product" data-id="1"><h3>Product 1</h3>\n <p class="description">This is product 1</p>\n <span class="price">$10.99</span>\n <div class="hidden stock">In stock: 5</div>\n </article>'
It's the same if you used the .body property
>>> article.body
'<article class="product" data-id="1"><h3>Product 1</h3>\n <p class="description">This is product 1</p>\n <span class="price">$10.99</span>\n <div class="hidden stock">In stock: 5</div>\n </article>'
Get the prettified version of the HTML content of the element
>>> print(article.prettify())
<article class="product" data-id="1"><h3>Product 1</h3>
<p class="description">This is product 1</p>
<span class="price">$10.99</span>
<div class="hidden stock">In stock: 5</div>
</article>
To get all the ancestors in the DOM tree of this element
>>> article.path
[<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>,
<data='<body> <div class="product-list"> <artic...' parent='<html><head><title>Some page</title></he...'>,
<data='<html><head><title>Some page</title></he...'>]
Generate a CSS shortened selector if possible, or generate the full selector
>>> article.generate_css_selector
'body > div > article'
>>> article.generate_full_css_selector
'body > div > article'
Same case with XPath
>>> article.generate_xpath_selector
"//body/div/article"
>>> article.generate_full_xpath_selector
"//body/div/article"
Traversal¶
Using the elements we found above, we will go over the properties/methods for moving in the page in detail.
If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online for a better understanding.
If you are too lazy to search about it, here's a quick explanation to give you a good idea.
Simply put, the html element is the root of the website's tree, as every page starts with an html element.
This element will be directly above elements like head and body. These are considered "children" of the html element, and the html element is considered their "parent." The element body is a "sibling" of the element head and vice versa.
Accessing the parent of an element
>>> article.parent
<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
>>> article.parent.tag
'div'
You can chain it as you want, which applies to all similar properties/methods we will review.
>>> article.parent.parent.tag
'body'
Get the children of an element
>>> article.children
[<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
<data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
<data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
<data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>]
Get all elements underneath an element. It acts as a nested version of the children property
>>> article.below_elements
[<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
<data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
<data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
<data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>]
This element returns the same result as the children property because its children don't have children.
Another example of using the element with the product-list class will clear the difference between the children property and the below_elements property
>>> products_list = page.css_first('.product-list')
>>> products_list.children
[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
>>> products_list.below_elements
[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
<data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
<data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
<data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>,
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
...]
Get the siblings of an element
>>> article.siblings
[<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
Get the next element of the current element
>>> article.next # gets the next element, the same logic applies to `quote.previous`
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>
The same logic applies to the previous property
>>> article.previous # It's the first child, so it doesn't have a previous element
>>> second_article = page.css_first('.product[data-id="2"]')
>>> second_article.previous
<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
You can check easily and pretty fast if an element has a specific class name or not
>>> article.has_class('product')
True
If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element, like the example below
for ancestor in article.iterancestors():
# do something with it...
You can search for a specific ancestor of an element that satisfies a function; all you need to do is to pass a function that takes an Adaptor object as an argument and return True if the condition satisfies or False otherwise like below:
>>> article.find_ancestor(lambda ancestor: ancestor.has_class('product-list'))
<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
>>> article.find_ancestor(lambda ancestor: ancestor.css('.product-list')) # Same result, different approach
<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
Adaptors¶
The class Adaptors is the "List" version of the Adaptor class. It inherits from the Python standard List type, so it shares all List properties and methods while adding more methods to make the operations you want to execute on the Adaptor instances within more straightforward.
In the Adaptor class, all methods/properties that should return a group of elements return them as an Adaptors class instance. The only exceptions are when you use the CSS/XPath methods as follows:
If you selected a text node with the selector, then the return type will be TextHandler/TextHandlers.
Examples:
>>> page.css('a::text') # -> TextHandlers
>>> page.xpath('//a/text()') # -> TextHandlers
>>> page.css_first('a::text') # -> TextHandler
>>> page.xpath_first('//a/text()') # -> TextHandler
>>> page.css('a::attr(href)') # -> TextHandlers
>>> page.xpath('//a/@href') # -> TextHandlers
>>> page.css_first('a::attr(href)') # -> TextHandler
>>> page.xpath_first('//a/@href') # -> TextHandler
If you used a combined selector that returns mixed types, the result will be a Python standard List.
Examples:
>>> page.css('.price_color') # -> Adaptors
>>> page.css('.product_pod a::attr(href)') # -> TextHandlers
>>> page.css('.price_color, .product_pod a::attr(href)') # -> List
Let's see what Adaptors class adds to the table with that out of the way.
Properties¶
Apart from the normal operations on Python lists like iteration, slicing, etc...
You can do the following:
Execute CSS and XPath selectors directly on the Adaptor instances it has while the arguments and the return types are the same as Adaptor's css and xpath methods. This, of course, makes chaining methods very straightforward.
>>> page.css('.product_pod a')
[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<div class="image_container"> <a href="c...'>,
<data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<div class="image_container"> <a href="c...'>,
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>,
<data='<a href="catalogue/soumission_998/index....' parent='<div class="image_container"> <a href="c...'>,
<data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
...]
>>> page.css('.product_pod').css('a') # Returns the same result
[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<div class="image_container"> <a href="c...'>,
<data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<div class="image_container"> <a href="c...'>,
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>,
<data='<a href="catalogue/soumission_998/index....' parent='<div class="image_container"> <a href="c...'>,
<data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
...]
Run the re and re_first methods directly. They take the same arguments passed as the Adaptor class. I'm still leaving these methods to be explained in the TextHandler section below.
However, in this class, the re_first behaves differently as it runs re on each Adaptor within and returns the first one with a result. The re method will return a TextHandlers object as normal that has all the results combined in one TextHandlers instance.
>>> page.css('.price_color').re(r'[\d\.]+')
['51.77',
'53.74',
'50.10',
'47.82',
'54.23',
...]
>>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
['a-light-in-the-attic_1000',
'tipping-the-velvet_999',
'soumission_998',
'sharp-objects_997',
...]
With the search method, you can search quickly in the available Adaptor classes. The function you pass must accept an Adaptor instance as the first argument and return True/False. The method will return the first Adaptor instance that satisfies the function; otherwise, it will return None.
# Find all the products with price '53.23'
>>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
>>> page.css('.product_pod').search(search_function)
<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>
You can use the filter method, too, which takes a function like the search method but returns an Adaptors instance of all the Adaptor classes that satisfy the function
# Find all products with prices over $50
>>> filtering_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) > 50
>>> page.css('.product_pod').filter(filtering_function)
[<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
...]
TextHandler¶
This class is mandatory to understand, as all methods/properties that should return a string for you will return TextHandler, and the ones that should return a list of strings will return TextHandlers instead.
TextHandler is a subclass of the standard Python string, so you can do anything with it. So, what is the difference that requires a different naming?
Of course, TextHandler provides extra methods and properties that the standard Python strings can't do. We will review them now, but remember that all methods and properties in all classes that return string(s) are returning TextHandler, which opens the door for creativity and makes the code shorter and cleaner, as you will see. Also, you can import it directly and use it on any string, which we will explain later.
Usage¶
First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like split, replace, strip, etc., all return a TextHandler again, so you can chain them as you want. If you find a method or property that returns a standard string instead of TextHandler, please open an issue, and we will override it as well.
First, we start with the re and re_first methods. These are the same methods that exist in the rest of the classes (Adaptor, Adaptors, and TextHandlers), so they will take the same arguments as well.
The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance.
Also, it takes other helpful arguments, which are:
- **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters.
- **clean_match**: It's disabled by default. This makes the method ignore all whitespaces and consecutive spaces while matching.
- **case_sensitive**: It's enabled by default. As the name implies, disabling it will make the regex ignore letters case while compiling it.
You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method.
```python
>>> page.css('.price_color').re(r'[\d\.]+')
['51.77',
'53.74',
'50.10',
'47.82',
'54.23',
...]
>>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
['a-light-in-the-attic_1000',
'tipping-the-velvet_999',
'soumission_998',
'sharp-objects_997',
...]
```
To explain the other arguments better, we will use a custom string for each example below
```python
>>> from scrapling import TextHandler
>>> test_string = TextHandler('hi there') # Hence the two spaces
>>> test_string.re('hi there')
>>> test_string.re('hi there', clean_match=True) # Using `clean_match` will clean the string before matching the regex
['hi there']
>>> test_string2 = TextHandler('Oh, Hi Mark')
>>> test_string2.re_first('oh, hi Mark')
>>> test_string2.re_first('oh, hi Mark', case_sensitive=False) # Hence disabling `case_sensitive`
'Oh, Hi Mark'
# Mixing arguments
>>> test_string.re('hi there', clean_match=True, case_sensitive=False)
['hi There']
```
Another use of the idea of replacing strings with `TextHandler` everywhere is a property like `html_content` returns `TextHandler` so you can do regex on the HTML content if you want:
```python
>>> page.html_content.re('div class=".*">(.*)</div')
['In stock: 5', 'In stock: 3', 'Out of stock']
```
You also have the .json() method, which tries to convert the content to a json object quickly if possible; otherwise, it throws an error
>>> page.css_first('#page-data::text')
'\n {\n "lastUpdated": "2024-09-22T10:30:00Z",\n "totalProducts": 3\n }\n '
>>> page.css_first('#page-data::text').json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
Hence, if you didn't specify a text node while selecting an element (like the text content or an attribute text content), the text content will be selected automatically like this
>>> page.css_first('#page-data').json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
The Adaptor class adds one thing here, too; let's say this is the page we are working with:
<html>
<body>
<div>
<script id="page-data" type="application/json">
{
"lastUpdated": "2024-09-22T10:30:00Z",
"totalProducts": 3
}
</script>
</div>
</body>
</html>
The Adaptor class has the get_all_text method, which you should be aware of by now. This method returns a TextHandler, of course.
So, as you know here, if you did something like this
>>> page.css_first('div::text').json()
You will get an error because the div tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have text content at all.
In this case, the get_all_text method comes to the rescue, so you can do something like that
>>> page.css_first('div').get_all_text(ignore_tags=[]).json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
I used the ignore_tags argument here because the default value of it is ('script', 'style',), as you are aware.
Another related behavior you should be aware of is the case while using any of the fetchers, which we will explain later. If you have a JSON response like this example:
>>> page = Adaptor("""{"some_key": "some_value"}""")
Because the Adaptor class is optimized to deal with HTML pages, it will deal with it as a broken HTML response and fix it, so if you used the html_content property, you get this
>>> page.html_content
'<html><body><p>{"some_key": "some_value"}</p></body></html>'
Here, you can use json method directly, and it will work
>>> page.json()
{'some_key': 'some_value'}
You might wonder how this happened while the html tag lacks direct text?
Well, for these cases like JSON responses, I made the .json() method inside the Adaptor class to check if the current element doesn't have text content; it will use the get_all_text method directly.
It might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions.
Another handy method is .clean(), this will remove all white spaces and consecutive spaces for you and return a new TextHandler, wonderful
>>> TextHandler('\n wonderful idea, \reh?').clean()
'wonderful idea, eh?'
Another method that might be helpful in some cases is the .sort() method to sort the string for you as you do with lists
>>> TextHandler('acb').sort()
'abc'
Or do it in reverse:
>>> TextHandler('acb').sort(reverse=True)
'cba'
Other methods and properties will be added over time, but remember that this class is returned in place of strings nearly everywhere in the library.
TextHandlers¶
You probably guessed it: This class is similar to Adaptors and Adaptor, but here it inherits the same logic and method as standard lists, with only re and re_first as new methods.
The only difference is that the re_first method logic here does re on each TextHandler within and returns the first result it has or None. Nothing is new to explain here, but new methods will be added here with time.
AttributesHandler¶
This is a read-only version of Python's standard dictionary or dict that's only used to store the attributes of each element or each Adaptor instance, in other words.
>>> print(page.find('script').attrib)
{'id': 'page-data', 'type': 'application/json'}
>>> type(page.find('script').attrib).__name__
'AttributesHandler'
Because it's read-only, it will use fewer resources than the standard dictionary. Still, it has the same dictionary method/properties other than those allowing you to modify/override the data.
It currently adds two extra simple methods:
The search_values method
In standard dictionaries, you can do dict.get("key_name") to check if a key exists. However, if you want to search by values instead of keys, it will take you some code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
A simple example would be
>>> for i in page.find('script').attrib.search_values('page-data'):
print(i)
{'id': 'page-data'}
But this method provides the partial argument as well, which allows you to search by part of the value:
>>> for i in page.find('script').attrib.search_values('page', partial=True):
print(i)
{'id': 'page-data'}
These examples won't happen in the real world; most likely, a more real-world example would be using it with the find_all method to find all elements that have a specific value in their arguments:
>>> page.find_all(lambda element: list(element.attrib.search_values('product')))
[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
All these elements have 'product' as a value for the attribute class.
Hence, I used the list function here because search_values returns a generator, so it would be True for all elements.
The json_string property
This property converts current attributes to JSON string if the attributes are JSON serializable; otherwise, it throws an error
>>> page.find('script').attrib.json_string
b'{"id":"page-data","type":"application/json"}'
Using automatch feature
Introduction¶
Auto-matching is one of Scrapling's most powerful features. It allows your scraper to survive website changes by intelligently tracking and relocating elements.
Let's say you are scraping a page with a structure like this:
<div class="container">
<section class="products">
<article class="product" id="p1">
<h3>Product 1</h3>
<p class="description">Description 1</p>
</article>
<article class="product" id="p2">
<h3>Product 2</h3>
<p class="description">Description 2</p>
</article>
</section>
</div>
And you want to scrape the first product, the one with the p1 ID. You will probably write a selector like this
page.css('#p1')
When website owners implement structural changes like
<div class="new-container">
<div class="product-wrapper">
<section class="products">
<article class="product new-class" data-id="p1">
<div class="product-info">
<h3>Product 1</h3>
<p class="new-description">Description 1</p>
</div>
</article>
<article class="product new-class" data-id="p2">
<div class="product-info">
<h3>Product 2</h3>
<p class="new-description">Description 2</p>
</div>
</article>
</section>
</div>
</div>
The selector will no longer function, and your code needs maintenance. That's where Scrapling's auto-matching feature comes into play.
With Scrapling, you can enable the automatch feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element and without AI :)
from scrapling import Adaptor, Fetcher
# Before the change
page = Adaptor(page_source, auto_match=True, url='example.com')
# or
Fetcher.auto_match = True
page = Fetcher.get('https://example.com')
# then
element = page.css('#p1' auto_save=True)
if not element: # One day website changes?
element = page.css('#p1', auto_match=True) # Scrapling still finds it!
# the rest of your code...
Below, I will show you one usage example for this feature. Then, we will dive deep into how to use it and provide details about this feature.
Real-World Scenario¶
Let's use a real website as an example and use one of the fetchers to fetch its source. To do this, we need to find a website that will soon change its design/structure, take a copy of its source, and then wait for the website to make the change. Of course, that's nearly impossible to know unless I know the website's owner, but that will make it a staged test, haha.
To solve this issue, I will use The Web Archive's Wayback Machine. Here is a copy of StackOverFlow's website in 2010; pretty old, eh?
Let's test if the automatch feature can extract the same button in the old design from 2010 and the current design using the same selector :)
If I want to extract the Questions button from the old design, I can use a selector like this: #hmenus > div:nth-child(1) > ul > li:nth-child(1) > a This selector is too specific because it was generated by Google Chrome.
Now, let's test the same selector in both versions
>> from scrapling import Fetcher
>> selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
>> old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
>> new_url = "https://stackoverflow.com/"
>> Fetcher.configure(auto_match = True, automatch_domain='stackoverflow.com')
>>
>> page = Fetcher.get(old_url, timeout=30)
>> element1 = page.css_first(selector, auto_save=True)
>>
>> # Same selector but used in the updated website
>> page = Fetcher.get(new_url)
>> element2 = page.css_first(selector, auto_match=True)
>>
>> if element1.text == element2.text:
... print('Scrapling found the same element in the old and new designs!')
'Scrapling found the same element in the old and new designs!'
Note that I used a new argument called automatch_domain; this is because, for Scrapling, these are two different domains(archive.org and stackoverflow.com), so scrapling will isolate their auto_match data. To tell Scrapling they are the same website, we need to pass the custom domain we want to use while saving auto-match data for them both so Scrapling doesn't isolate them.
The code will be the same in a real-world scenario, except it will use the same URL for both requests, so you won't need to use the automatch_domain argument. This is the closest example I can give to real-world cases, so I hope it didn't confuse you :)
Hence, in the two examples above, I used both the Adaptor class and the Fetcher class to show you that the logic for automatch is the same.
How the automatch feature works¶
Auto-matching works in two phases:
Save Phase: Store unique properties of elements
Match Phase: Find elements with similar properties later
Let's say you have an element you got through selection or any method and want the library to find it the next time you scrape this website, even if it had structural/design changes.
As little technical details as possible, the general logic goes as the following:
You tell Scrapling to save that element's unique properties in one of the ways we will show below.
Scrapling uses its configured database (SQLite by default) and saves each element's unique properties.
Now, because everything about the element can be changed or removed from the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
The domain of the current website. If you are using the Adaptor class, you should pass it while initializing the class, or if you are using one of the fetchers, the domain will be taken from the URL automatically.
An identifier to query that element's properties from the database. You don't always have to set the identifier yourself, as you will see later when we discuss this.
Together, they will be used to retrieve the element's unique properties from the database later.
Later, when the website structural changes, you tell Scrapling to automatch the element. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated for their similarity to the element we want. In that comparison, everything is taken into consideration, as you will see later
The element(s) with the highest similarity score to the wanted element are returned.
The unique properties¶
You might wonder, if all aspects of an element can be removed or changed, what unique properties we are talking about.
For Scrapling, the unique elements we are relying on are:
Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only).
Element's parent tag name, attributes (names and values), and text.
But you need to understand that the comparison between elements is not exact; it's more about finding how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
How to use automatch feature¶
The automatch feature can be used on any element you have, and it's added as arguments to CSS/XPath Selection methods, as you saw above, but we will get back to that later.
First, you must enable the automatch feature by passing auto_match=True to the Adaptor class when you initialize it or enable it in the fetcher you are using of the available fetchers, as we will show.
Examples:
>>> from scrapling import Adaptor, Fetcher
>>> page = Adaptor(html_doc, auto_match=True)
# OR
>>> Fetcher.auto_match = True
>>> page = Fetcher.fetch('https://example.com')
If you are using the Adaptor class, you need to pass the url of the website you are using with the argument url so Scrapling can separate the properties saved for each element by domain.
If you didn't pass a URL, the word default will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you used the same identifier later for a different website and didn't pass the URL parameter while initializing it. The save process will overwrite the previous data, and auto-matching only uses the latest saved properties.
Besides those arguments, we have storage and storage_args. Both are for the class to be used to connect to the database; by default, it's set to the SQLite class that the library is using. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a separate page in the development section.
Now, after enabling the automatch feature globally, you have two main ways to use it.
The CSS/XPath Selection way¶
As you have seen in the example above, first, you have to use the auto_save argument while selecting an element that exists on the page like below
element = page.css('#p1' auto_save=True)
and when the element doesn't exist, you can use the same selector and the auto_match argument, and the library will find it for you
element = page.css('#p1', auto_match=True)
Pretty simple, eh?
Well, a lot happened under the hood here. Remember the identifier part we mentioned before that you need to set so you can retrieve the element you want? Here, with the css/css_first/xpath/xpath_first methods, the identifier is set automatically as the selector you passed here to make things easier :)
Also, that's why here, for all these methods, you can pass the identifier argument to set it yourself, and there are cases for this, or you can use it to save the properties with the auto_save argument.
The manual way¶
You manually save and retrieve an element, then relocate it, which all happens within the automatch feature, as shown below. This allows you to automatch any element you have by any way or any selection method!
First, let's say you got an element like this by text:
>>> element = page.find_by_text('Tipping the Velvet', first_match=True)
You can save its unique properties with the save method like below, but you must set the identifier yourself. For this example, I chose my_special_element as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
>>> page.save(element, 'my_special_element')
Now, later, when you want to retrieve it and relocate it inside the page with auto-matching, it would be like this
>>> element_dict = page.retrieve('my_special_element')
>>> page.relocate(element_dict, adaptor_type=True)
[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]
>>> page.relocate(element_dict, adaptor_type=True).css('::text')
['Tipping the Velvet']
Hence, the retrieve and relocate` methods are used.
if you want to keep it as lxml.etree object, leave the adaptor_type argument
>>> page.relocate(element_dict)
[<Element a at 0x105a2a7b0>]
Troubleshooting¶
No Matches Found¶
# 1. Check if data was saved
element_data = page.retrieve('identifier')
if not element_data:
print("No data saved for this identifier")
# 2. Try with different identifier
products = page.css('.product', auto_match=True, identifier='old_selector')
# 3. Save again with new identifier
products = page.css('.new-product', auto_save=True, identifier='new_identifier')
Wrong Elements Matched¶
# Use more specific selectors
products = page.css('.product-list .product', auto_save=True)
# Or save with more context
product = page.find_by_text('Product Name').parent
page.save(product, 'specific_product')
Known Issues¶
In the auto-matching save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page in different locations, auto-matching will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone.
Final thoughts¶
Explaining this feature in detail without complications turned out to be challenging, but still, if there's something left unclear, you can head out to the discussions section, and I will reply to you ASAP or reach out to me privately and have a chat :)
Choosing a fetcher
Fetchers are classes that can do requests or fetch pages for you easily in a single-line fashion with many features and then return a Response object.
This feature was introduced because the only option before v0.2 was to fetch the page as you wanted, then pass it manually to the Adaptor class and start playing with it.
Fetchers are not wrappers built on top of other libraries, but they use these libraries as an engine to make requests/fetch pages easily for you while fully utilizing that engine and adding features for you that aren't included in those engines
Fetchers Overview¶
Scrapling provides three different fetcher classes, each designed for specific use cases.
The following table compares them and can be quickly used for guidance.
Feature Fetcher PlayWrightFetcher StealthyFetcher
Relative speed ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐
Stealth ⭐ ⭐⭐ ⭐⭐⭐⭐
Anti-Bot options ⭐ ⭐⭐ ⭐⭐⭐⭐
JavaScript loading ❌ ✅ ✅
Memory Usage ⭐ ⭐⭐⭐ ⭐⭐⭐
Best used for Basic scraping - Dynamically loaded websites
- Small automation
- Slight protections - Dynamically loaded websites
- Small automation
- Complicated protections
Browser(s) ❌ Chromium and Google Chrome Modified Firefox
Browser API used ❌ PlayWright PlayWright
Setup Complexity Simple Simple Simple
In the following pages, we will talk about each one in detail.
Parser configuration in all fetchers¶
All fetchers classes share the same import, as you will see in the upcoming pages
>>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
Then you use it right away without initializing like this, and it will use the default parser settings:
>>> page = StealthyFetcher.fetch('https://example.com')
If you want to configure the parser (Adaptor class) that will be used on the response before returning it for you, then do this first:
>>> from scrapling.fetchers import Fetcher
>>> Fetcher.configure(auto_match=True, encoding="utf8", keep_comments=False, keep_cdata=False) # and the rest
or
>>> from scrapling.fetchers import Fetcher
>>> Fetcher.auto_match=True
>>> Fetcher.encoding="utf8"
>>> Fetcher.keep_comments=False
>>> Fetcher.keep_cdata=False # and the rest
Then, continue your code as usual.
The available configuration arguments are: auto_match, huge_tree, keep_comments, keep_cdata, storage, and storage_args, which are the same ones you give to the Adaptor class. You can display the current configuration anytime by running <fetcher_class>.display_config().
Note: The auto_match argument is disabled by default; you must enable it to use that feature.
Set parser config per request¶
As you probably understood, the logic above for setting the parser config will work globally for all requests/fetches done through that class, and it's intended for simplicity.
If your use case requires a different configuration for each request/fetch, you can pass a dictionary to the request method (fetch/get/post/...) to an argument named custom_config.
Response Object¶
The Response object is the same as the Adaptor class, but it has added details about the response like response headers, status, cookies, etc... as shown below:
>>> from scrapling.fetchers import Fetcher
>>> page = Fetcher.get('https://example.com')
>>> page.status # HTTP status code
>>> page.reason # Status message
>>> page.cookies # Response cookies as a dictionary
>>> page.headers # Response headers
>>> page.request_headers # Request headers
>>> page.history # Response history of redirections, if any
>>> page.body # Raw response body
>>> page.encoding # Response encoding
All fetchers return the Response object.
Static Requests:¶
The Fetcher class provides fast and lightweight HTTP requests with some stealth capabilities. This class uses httpx as an engine for making requests. For advanced usages, you will need some knowledge about httpx, but it becomes simpler and simpler with user feedback and updates.
Basic Usage¶
You have one primary way to import this Fetcher, which is the same for all fetchers.
>>> from scrapling.fetchers import Fetcher
Check out how to configure the parsing options here
Shared arguments¶
All methods for making requests here share some arguments, so let's discuss them first.
url: The URL you want to request, of course :)
proxy: As the name implies, the proxy for this request is used to route all traffic (HTTP and HTTPS). The format accepted here is http://username:password@localhost:8030.
stealthy_headers: Generate and use real browser's headers, then create a referer header as if this request came from a Google search page of this URL's domain. Enabled by default, all headers generated can be overwritten by you through the headers argument.
follow_redirects: As the name implies, tell the fetcher to follow redirections. Enabled by default
timeout: The timeout to wait for each request to be finished in milliseconds. The default is 30000ms (30 seconds).
retries: The number of retries that httpx will do for failed requests. The default number of retries is 3.
Other than this, you can pass any arguments that httpx.<method_name> takes, and that's why I said, in the beginning, you need a bit of knowledge about httpx, but in the following examples, we will try to cover most cases.
HTTP Methods¶
Examples are the best way to explain this
Hence: OPTIONS and HEAD methods are not supported.
GET¶
>>> from scrapling.fetchers import Fetcher
>>> # Basic GET
>>> page = Fetcher.get('https://example.com')
>>> page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
>>> page = Fetcher.get('https://httpbin.org/get', proxy='http://username:password@localhost:8030')
>>> # With parameters
>>> page = Fetcher.get('https://example.com/search', params={'q': 'query'})
>>>
>>> # With headers
>>> page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
>>> # Basic HTTP authentication
>>> page = Fetcher.get("https://example.com", auth=("my_user", "password123"))
And for asynchronous requests, it's a small adjustment
>>> from scrapling.fetchers import AsyncFetcher
>>> # Basic GET
>>> page = await AsyncFetcher.get('https://example.com')
>>> page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
>>> page = await AsyncFetcher.get('https://httpbin.org/get', proxy='http://username:password@localhost:8030')
>>> # With parameters
>>> page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
>>>
>>> # With headers
>>> page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
>>> # Basic HTTP authentication
>>> page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
Needless to say, the page object in all cases is Response object, which is an Adaptor as we said, so you will use it directly
>>> page.css('.something.something')
>>> page = Fetcher.get('https://api.github.com/events')
>>> page.json()
[{'id': '<redacted>',
'type': 'PushEvent',
'actor': {'id': '<redacted>',
'login': '<redacted>',
'display_login': '<redacted>',
'gravatar_id': '',
'url': 'https://api.github.com/users/<redacted>',
'avatar_url': 'https://avatars.githubusercontent.com/u/<redacted>'},
'repo': {'id': '<redacted>',
...
POST¶
>>> from scrapling.fetchers import Fetcher
>>> # Basic POST
>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'})
>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)
>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
>>> # Another example of form-encoded data
>>> page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'})
>>> # JSON data
>>> page = Fetcher.post('https://example.com/api', json={'key': 'value'})
>>> # Uploading file
>>> r = Fetcher.post("https://httpbin.org/post", files={'upload-file': open('something.xlsx', 'rb')})
And for asynchronous requests, it's a small adjustment
>>> from scrapling.fetchers import AsyncFetcher
>>> # Basic POST
>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'})
>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)
>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
>>> # Another example of form-encoded data
>>> page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'})
>>> # JSON data
>>> page = await AsyncFetcher.post('https://example.com/api', json={'key': 'value'})
>>> # Uploading file
>>> r = await AsyncFetcher.post("https://httpbin.org/post", files={'upload-file': open('something.xlsx', 'rb')})
PUT¶
>>> from scrapling.fetchers import Fetcher
>>> # Basic PUT
>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'})
>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True)
>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
>>> # Another example of form-encoded data
>>> page = Fetcher.put("https://httpbin.org/put", data={'key': ['value1', 'value2']})
And for asynchronous requests, it's a small adjustment
>>> from scrapling.fetchers import AsyncFetcher
>>> # Basic PUT
>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})
>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True)
>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
>>> # Another example of form-encoded data
>>> page = await AsyncFetcher.put("https://httpbin.org/put", data={'key': ['value1', 'value2']})
DELETE¶
>>> from scrapling.fetchers import Fetcher
>>> page = Fetcher.delete('https://example.com/resource/123')
>>> page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True)
>>> page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
And for asynchronous requests, it's a small adjustment
>>> from scrapling.fetchers import AsyncFetcher
>>> page = await AsyncFetcher.delete('https://example.com/resource/123')
>>> page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True)
>>> page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
Examples¶
Some well-rounded examples to aid newcomers to Web Scraping
Basic HTTP Request¶
from scrapling.fetchers import Fetcher
# Make a request
page = Fetcher.get('https://example.com')
# Check the status
if page.status == 200:
# Extract title
title = page.css_first('title::text')
print(f"Page title: {title}")
# Extract all links
links = page.css('a::attr(href)')
print(f"Found {len(links)} links")
Product Scraping¶
from scrapling.fetchers import Fetcher
def scrape_products():
page = Fetcher.get('https://example.com/products')
# Find all product elements
products = page.css('.product')
results = []
for product in products:
results.append({
'title': product.css_first('.title::text'),
'price': product.css_first('.price::text').re_first(r'\d+\.\d{2}'),
'description': product.css_first('.description::text'),
'in_stock': product.has_class('in-stock')
})
return results
Pagination Handling¶
from scrapling.fetchers import Fetcher
def scrape_all_pages():
base_url = 'https://example.com/products?page={}'
page_num = 1
all_products = []
while True:
# Get current page
page = Fetcher.get(base_url.format(page_num))
# Find products
products = page.css('.product')
if not products:
break
# Process products
for product in products:
all_products.append({
'name': product.css_first('.name::text'),
'price': product.css_first('.price::text')
})
# Next page
page_num += 1
return all_products
Form Submission¶
from scrapling.fetchers import Fetcher
# Submit login form
response = Fetcher.post(
'https://example.com/login',
data={
'username': 'user@example.com',
'password': 'password123'
}
)
# Check login success
if response.status == 200:
# Extract user info
user_name = response.css_first('.user-name::text')
print(f"Logged in as: {user_name}")
Table Extraction¶
from scrapling.fetchers import Fetcher
def extract_table():
page = Fetcher.get('https://example.com/data')
# Find table
table = page.css_first('table')
# Extract headers
headers = [
th.text for th in table.css('thead th')
]
# Extract rows
rows = []
for row in table.css('tbody tr'):
cells = [td.text for td in row.css('td')]
rows.append(dict(zip(headers, cells)))
return rows
Navigation Menu¶
from scrapling.fetchers import Fetcher
def extract_menu():
page = Fetcher.get('https://example.com')
# Find navigation
nav = page.css_first('nav')
menu = {}
for item in nav.css('li'):
link = item.css_first('a')
if link:
menu[link.text] = {
'url': link.attrib['href'],
'has_submenu': bool(item.css('.submenu'))
}
return menu
When to Use¶
Use Fetcher when:
Need fast HTTP requests
Want minimal overhead
Don't need JavaScript
Want simple configuration
Need basic stealth features
Use other fetchers when:
Need browser automation.
Need advanced anti-bot/stealth.
Need JavaScript support.
Dynamically Loaded Websites ¶:
Here, we will discuss the PlayWrightFetcher class. This class provides flexible browser automation with multiple configuration options and some stealth capabilities. It uses PlayWright as an engine for fetching websites.
As we will explain later, to automate the page, you need some knowledge of PlayWright's Page API.
Basic Usage¶
You have one primary way to import this Fetcher, which is the same for all fetchers.
>>> from scrapling.fetchers import PlayWrightFetcher
Check out how to configure the parsing options here
Now we will go over most of the arguments one by one with examples if you want to jump to a table of all arguments for quick reference click here
Notes:
Every time you fetch a website with this fetcher, it waits by default for all JavaScript to fully load and execute, so you don't have to (waits for the domcontentloaded state).
Of course, the async version of the fetch method is the async_fetch method.
This fetcher currently provides 4 main run options, but they can be mixed as you want.
Which are:
1. Vanilla Playwright¶
PlayWrightFetcher.fetch('https://example.com')
Using it like that will open a Chromium browser and fetch the page. There are no tricks or extra features; it's just a plain PlayWright API.
2. Stealth Mode¶
PlayWrightFetcher.fetch('https://example.com', stealth=True)
It's the same as the vanilla PlayWright option, but it provides a simple stealth mode suitable for websites with a small-to-medium protection layer(s).
Some of the things this fetcher's stealth mode does include:
Patching the CDP runtime fingerprint.
Mimics some of the real browsers' properties by injecting several JS files and using custom options.
Custom flags are used on launch to hide Playwright even more and make it faster.
Generates real browser headers of the same type and user OS, then append them to the request's headers.
3. Real Chrome¶
PlayWrightFetcher.fetch('https://example.com', real_chrome=True)
If you have a Google Chrome browser installed, use this option. It's the same as the first option but will use the Google Chrome browser you installed on your device instead of Chromium.
This will make your requests look more like requests coming from an actual human, so it's less detectable, and you can even use the stealth=True mode with it for better results like below:
PlayWrightFetcher.fetch('https://example.com', real_chrome=True, stealth=True)
If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually:
playwright install chrome
4. CDP Connection¶
PlayWrightFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the Chrome DevTools Protocol.
This fetcher takes it even a step further. You can use NSTBrowser's docker browserless option by passing the CDP URL and enabling nstbrowser_mode option like below
PlayWrightFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222', nstbrowser_mode=True)
There's also a nstbrowser_config argument to send the config you want to send with the requests to the NSTBrowser. If you leave it empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config.
Full list of arguments¶
Scrapling provides many options with this fetcher, which works in all modes except the NSTBrowser mode. To make it as simple as possible, we will list the options here and give examples of using most of them.
Argument Description Optional
url Target url ❌
headless Pass True to run the browser in headless/hidden (default) or False for headful/visible mode. ✔️
disable_resources Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading. ✔️
useragent Pass a useragent string to be used. Otherwise, the fetcher will generate and use a real Useragent of the same browser. ✔️
network_idle Wait for the page until there are no network connections for at least 500 ms. ✔️
timeout The timeout (milliseconds) used in all operations and waits through the page. The default is 30000. ✔️
wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object. ✔️
page_action Added for automation. Pass a function that takes the page object and does the necessary automation, then returns page again. ✔️
wait_selector Wait for a specific css selector to be in a specific state. ✔️
wait_selector_state Scrapling will wait for the given state to be fulfilled for the selector given with wait_selector. Default state is attached. ✔️
google_search Enabled by default, Scrapling will set the referer header as if this request came from a Google search for this website's domain name. ✔️
extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together. ✔️
proxy The proxy to be used with requests. It can be a string or a dictionary with the keys 'server', 'username', and 'password' only. ✔️
hide_canvas Add random noise to canvas operations to prevent fingerprinting. ✔️
disable_webgl Disables WebGL and WebGL 2.0 support entirely. ✔️
stealth Enables stealth mode; you should always check the documentation to see what stealth mode does currently. ✔️
real_chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser and use it. ✔️
locale Set the locale for the browser if wanted. The default value is en-US. ✔️
cdp_url Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP. ✔️
nstbrowser_mode Enables NSTBrowser mode, it have to be used with cdp_url argument or it will get completely ignored. ✔️
nstbrowser_config The config you want to send with requests to the NSTBrowser. Scrapling defaults to an optimized NSTBrowser's docker browserless config if you leave this argument empty. ✔️
Examples¶
It's easier to understand with examples, so let's look at it.
Resource Control¶
# Disable unnecessary resources
page = PlayWrightFetcher.fetch(
'https://example.com',
disable_resources=True # Blocks fonts, images, media, etc...
)
Network Control¶
# Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)
page = PlayWrightFetcher.fetch('https://example.com', network_idle=True)
# Custom timeout (in milliseconds)
page = PlayWrightFetcher.fetch('https://example.com', timeout=30000) # 30 seconds
# Proxy support
page = PlayWrightFetcher.fetch(
'https://example.com',
proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only
)
Browser Automation¶
This is where your knowledge about PlayWright's Page API comes into play. The function you pass here takes the page object from Playwright's API, does what you want, and then returns it again for the current fetcher to continue working on it.
This function is executed right after waiting for network_idle (if enabled) and before waiting for the wait_selector argument, so it can be used for many things, not just automation. You can alter the page as you want.
In the example below, I used page mouse events to move the mouse wheel to scroll the page and then move the mouse.
from playwright.sync_api import Page
def scroll_page(page: Page):
page.mouse.wheel(10, 0)
page.mouse.move(100, 400)
page.mouse.up()
return page
page = PlayWrightFetcher.fetch(
'https://example.com',
page_action=scroll_page
)
Of course, if you use the async fetch version, the function must also be async.
from playwright.async_api import Page
async def scroll_page(page: Page):
await page.mouse.wheel(10, 0)
await page.mouse.move(100, 400)
await page.mouse.up()
return page
page = await PlayWrightFetcher.async_fetch(
'https://example.com',
page_action=scroll_page
)
Wait Conditions¶
# Wait for the selector
page = PlayWrightFetcher.fetch(
'https://example.com',
wait_selector='h1',
wait_selector_state='visible'
)
This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the wait_selector argument, and the fetcher will wait for the state you passed in the wait_selector_state argument to be fulfilled. If you didn't pass a state, the default would be attached, which means it will wait for the element to be present in the DOM.
After that, the fetcher will check again to see if all JS files are loaded and executed (the domcontentloaded state) and wait for them to be. If you have enabled network_idle with this, the fetcher will wait for network_idle to be fulfilled again, as explained above.
The states the fetcher can wait for can be either (source):
attached: Wait for an element to be present in DOM.
detached: Wait for an element to not be present in DOM.
visible: wait for an element to have a non-empty bounding box and no visibility:hidden. Note that an element without any content or with display:none has an empty bounding box and is not considered visible.
hidden: wait for an element to be either detached from DOM, or have an empty bounding box or visibility:hidden. This is opposite to the 'visible' option.
Some Stealth Features¶
# Full stealth mode
page = PlayWrightFetcher.fetch(
'https://example.com',
stealth=True,
hide_canvas=True,
disable_webgl=True,
google_search=True
)
# Custom user agent
page = PlayWrightFetcher.fetch(
'https://example.com',
useragent='Mozilla/5.0...'
)
# Set browser locale
page = PlayWrightFetcher.fetch(
'https://example.com',
locale='en-US'
)
Hence, the hide_canvas argument doesn't disable canvas but hides it by adding random noise to canvas operations to prevent fingerprinting. Also, if you didn't set a useragent (preferred), the fetcher will generate a real Useragent of the same browser and use it.
The google_search argument is enabled by default, making the request look like it came from Google. So, a request for https://example.com will set the referer to https://www.google.com/search?q=example. Also, if used together, it takes priority over the referer set by the extra_headers argument.
General example¶
from scrapling.fetchers import PlayWrightFetcher
def scrape_dynamic_content():
# Use PlayWright for JavaScript content
page = PlayWrightFetcher.fetch(
'https://example.com/dynamic',
network_idle=True,
wait_selector='.content'
)
# Extract dynamic content
content = page.css('.content')
return {
'title': content.css_first('h1::text'),
'items': [
item.text for item in content.css('.item')
]
}
When to Use¶
Use PlayWrightFetcher when:
Need browser automation
Want multiple browser options
Using a real Chrome browser
Need custom browser config
Want flexible stealth options
Full Bypass protections while fetching:
Here, we will discuss the StealthyFetcher class. This class is similar to PlayWrightFetcher in many ways, like browser automation and using PlayWright as an engine for fetching websites. The main difference is that this class provides advanced anti-bot protection bypass capabilities and a modified Firefox browser called Camoufox, from which most stealth comes.
As with PlayWrightFetcher, you will need some knowledge about PlayWright's Page API to automate the page, as we will explain later.
Basic Usage¶
You have one primary way to import this Fetcher, which is the same for all fetchers.
>>> from scrapling.fetchers import StealthyFetcher
Check out how to configure the parsing options here
Notes:
Every time you fetch a website with this fetcher, it waits by default for all JavaScript to fully load and execute, so you don't have to (waits for the domcontentloaded state).
Of course, the async version of the fetch method is the async_fetch method.
Full list of arguments¶
Before jumping to examples, here's the full list of arguments
Argument Description Optional
url Target url ❌
headless Pass True to run the browser in headless/hidden (default), virtual to run it in virtual screen mode, or False for headful/visible mode. The virtual mode requires having xvfb installed. ✔️
block_images Prevent the loading of images through Firefox preferences. This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading. ✔️
disable_resources Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading. ✔️
google_search Enabled by default, Scrapling will set the referer header as if this request came from a Google search for this website's domain name. ✔️
extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together. ✔️
block_webrtc Blocks WebRTC entirely. ✔️
page_action Added for automation. A function that takes the page object and does the automation you need, then returns page again. ✔️
addons List of Firefox addons to use. Must be paths to extracted addons. ✔️
humanize Humanize the cursor movement. The cursor movement takes either True or the MAX duration in seconds. The cursor typically takes up to 1.5 seconds to move across the window. ✔️
allow_webgl Enabled by default. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. ✔️
geoip Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. ✔️
os_randomize If enabled, Scrapling will randomize the OS fingerprints used. The default is matching the fingerprints with the current OS. ✔️
disable_ads Disabled by default; this installs the uBlock Origin addon on the browser if enabled. ✔️
network_idle Wait for the page until there are no network connections for at least 500 ms. ✔️
timeout The timeout used in all operations and waits through the page. It's in milliseconds, and the default is 30000. ✔️
wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object. ✔️
wait_selector Wait for a specific css selector to be in a specific state. ✔️
wait_selector_state Scrapling will wait for the given state to be fulfilled for the selector given with wait_selector. Default state is attached. ✔️
proxy The proxy to be used with requests. It can be a string or a dictionary with the keys 'server', 'username', and 'password' only. ✔️
additional_arguments Arguments passed to Camoufox as additional settings that take higher priority than Scrapling's. ✔️
Examples¶
It's easier to understand with examples, so now we will go over most of the arguments individually with examples.
Browser Modes¶
# Headless/hidden mode (default)
page = StealthyFetcher.fetch('https://example.com', headless=True)
# Virtual display mode (requires having `xvfb` installed)
page = StealthyFetcher.fetch('https://example.com', headless='virtual')
# Visible browser mode
page = StealthyFetcher.fetch('https://example.com', headless=False)
Resource Control¶
# Block images
page = StealthyFetcher.fetch('https://example.com', block_images=True)
# Disable unnecessary resources
page = StealthyFetcher.fetch('https://example.com', disable_resources=True) # Blocks fonts, images, media, etc.
Additional stealth options¶
page = StealthyFetcher.fetch(
'https://example.com',
block_webrtc=True, # Block WebRTC
allow_webgl=False, # Disable WebGL
humanize=True, # Make the mouse move as how a human would move it
geoip=True, # Use IP's longitude, latitude, timezone, country, and locale, then spoof the WebRTC IP address...
os_randomize=True, # Randomize the OS fingerprints used. The default is matching the fingerprints with the current OS.
disable_ads=True, # Block ads with uBlock Origin addon (enabled by default)
google_search=True
)
# Custom user agent
page = StealthyFetcher.fetch(
'https://example.com',
useragent='Mozilla/5.0...'
)
# Custom humanization duration
page = StealthyFetcher.fetch(
'https://example.com',
humanize=1.5 # Max 1.5 seconds for cursor movement
)
The google_search argument is enabled by default. It makes the request as if it came from Google, so for a request for https://example.com, it will set the referer to https://www.google.com/search?q=example. Also, if used together, it takes priority over the referer set by the extra_headers argument.
Network Control¶
# Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)
page = StealthyFetcher.fetch('https://example.com', network_idle=True)
# Custom timeout (in milliseconds)
page = StealthyFetcher.fetch('https://example.com', timeout=30000) # 30 seconds
# Proxy support
page = StealthyFetcher.fetch(
'https://example.com',
proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only
)
Browser Automation¶
This is where your knowledge about PlayWright's Page API comes into play. The function you pass here takes the page object from Playwright's API, does what you want, and then returns it again for the current fetcher to continue working on it.
This function is executed right after waiting for network_idle (if enabled) and before waiting for the wait_selector argument, so it can be used for many things, not just automation. You can alter the page as you want.
In the example below, I used page mouse events to move the mouse wheel to scroll the page and then move the mouse.
from playwright.sync_api import Page
def scroll_page(page: Page):
page.mouse.wheel(10, 0)
page.mouse.move(100, 400)
page.mouse.up()
return page
page = StealthyFetcher.fetch(
'https://example.com',
page_action=scroll_page
)
Of course, if you use the async fetch version, the function must also be async.
from playwright.async_api import Page
async def scroll_page(page: Page):
await page.mouse.wheel(10, 0)
await page.mouse.move(100, 400)
await page.mouse.up()
return page
page = await StealthyFetcher.async_fetch(
'https://example.com',
page_action=scroll_page
)
Wait Conditions¶
# Wait for the selector
page = StealthyFetcher.fetch(
'https://example.com',
wait_selector='h1',
wait_selector_state='visible'
)
This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the wait_selector argument, and the fetcher will wait for the state you passed in the wait_selector_state argument to be fulfilled. If you didn't pass a state, the default would be attached, which means it will wait for the element to be present in the DOM.
After that, the fetcher will check again to see if all JS files are loaded and executed (the domcontentloaded state) and wait for them to be. If you have enabled network_idle with this, the fetcher will wait for network_idle to be fulfilled again, as explained above.
The states the fetcher can wait for can be either (source):
attached: wait for the element to be present in DOM.
detached: wait for the element to not be present in DOM.
visible: wait for the element to have a non-empty bounding box and no visibility:hidden. Note that an element without any content or with display:none has an empty bounding box and is not considered visible.
hidden: Wait for the element to be detached from DOM, have an empty bounding box, or have visibility:hidden. This is opposite to the 'visible' option.
Firefox Addons¶
# Custom Firefox addons
page = StealthyFetcher.fetch(
'https://example.com',
addons=['/path/to/addon1', '/path/to/addon2']
)
The paths here must be paths of extracted addons, which will be installed automatically upon browser launch.
Real-world example (Amazon)¶
This is for educational purposes only; this example was generated by AI, which shows too how easy it is to work with Scrapling through AI
def scrape_amazon_product(url):
# Use StealthyFetcher to bypass protection
page = StealthyFetcher.fetch(url)
# Extract product details
return {
'title': page.css_first('#productTitle::text').clean(),
'price': page.css_first('.a-price .a-offscreen::text'),
'rating': page.css_first('[data-feature-name="averageCustomerReviews"] .a-popover-trigger .a-color-base::text'),
'reviews_count': page.css('#acrCustomerReviewText::text').re_first(r'[\d,]+'),
'features': [
li.clean() for li in page.css('#feature-bullets li span::text')
],
'availability': page.css_first('#availability').get_all_text(strip=True),
'images': [
img.attrib['src'] for img in page.css('#altImages img')
]
}
When to Use¶
Use StealthyFetcher when:
Bypassing anti-bot protection
Need a reliable browser fingerprint
Full JavaScript support needed
Want automatic stealth features
Need browser automation
API Reference:
Adaptor Class¶
The Adaptor class is the core parsing engine in Scrapling that provides HTML parsing and element selection capabilities.
Here's the reference information for the Adaptor class, with all its parameters, attributes, and methods.
You can import the Adaptor class directly from scrapling:
from scrapling.parser import Adaptor
scrapling.parser.Adaptor ¶
Adaptor(
text=None,
url=None,
body=b"",
encoding="utf8",
huge_tree=True,
root=None,
keep_comments=False,
keep_cdata=False,
auto_match=False,
storage=SQLiteStorageSystem,
storage_args=None,
**kwargs
)
Bases: SelectorsGeneration
The main class that works as a wrapper for the HTML input data. Using this class, you can search for elements with expressions in CSS, XPath, or with simply text. Check the docs for more info.
Here we try to extend module lxml.html.HtmlElement while maintaining a simpler interface, We are not inheriting from the lxml.html.HtmlElement because it's not pickleable which makes a lot of reference jobs not possible. You can test it here and see code explodes with AssertionError: invalid Element proxy at.... It's an old issue with lxml, see this entry <https://bugs.launchpad.net/lxml/+bug/736708>
PARAMETER DESCRIPTION
text HTML body passed as text.
TYPE: Optional[str]DEFAULT: None
url allows storing a URL with the html data for retrieving later.
TYPE: Optional[str]DEFAULT: None
body HTML body as bytes object. It can be used instead of the text argument.
TYPE: bytesDEFAULT: b''
encoding The encoding type that will be used in HTML parsing, default is UTF-8
TYPE: strDEFAULT: 'utf8'
huge_tree Enabled by default, should always be enabled when parsing large HTML documents. This controls libxml2 feature that forbids parsing certain large documents to protect from possible memory exhaustion.
TYPE: boolDEFAULT: True
root Used internally to pass etree objects instead of text/body arguments, it takes highest priority. Don't use it unless you know what you are doing!
TYPE: Optional[HtmlElement]DEFAULT: None
keep_comments While parsing the HTML body, drop comments or not. Disabled by default for obvious reasons
TYPE: Optional[bool]DEFAULT: False
keep_cdata While parsing the HTML body, drop cdata or not. Disabled by default for cleaner HTML.
TYPE: Optional[bool]DEFAULT: False
auto_match Globally turn-off the auto-match feature in all functions, this argument takes higher priority over all auto-match related arguments/functions in the class.
TYPE: Optional[bool]DEFAULT: False
storage The storage class to be passed for auto-matching functionalities, see Docs for more info.
TYPE: AnyDEFAULT: SQLiteStorageSystem
storage_args A dictionary of argument->value pairs to be passed for the storage class. If empty, default values will be used.
TYPE: Optional[Dict]DEFAULT: None
Source code in scrapling/parser.py
encoding instance-attribute ¶
encoding = encoding
url instance-attribute ¶
url = url
tag property ¶
tag
Get tag name of the element
text property ¶
text
Get text content of the element
attrib property ¶
attrib
Get attributes of the element
html_content property ¶
html_content
Return the inner html code of the element
body class-attribute instance-attribute ¶
body = html_content
parent property ¶
parent
Return the direct parent of the element or None otherwise
below_elements property ¶
below_elements
Return all elements under the current element in the DOM tree
children property ¶
children
Return the children elements of the current element or empty list otherwise
siblings property ¶
siblings
Return other children of the current element's parent or empty list otherwise
path property ¶
path
Returns list of type :class:Adaptors that contains the path leading to the current element from the root.
next property ¶
next
Returns the next element of the current element in the children of the parent or None otherwise.
previous property ¶
previous
Returns the previous element of the current element in the children of the parent or None otherwise.
extract class-attribute instance-attribute ¶
extract = get_all
extract_first class-attribute instance-attribute ¶
extract_first = get
generate_css_selector property ¶
generate_css_selector
Generate a CSS selector for the current element
RETURNS DESCRIPTION
str A string of the generated selector.
generate_full_css_selector property ¶
generate_full_css_selector
Generate a complete CSS selector for the current element
RETURNS DESCRIPTION
str A string of the generated selector.
generate_xpath_selector property ¶
generate_xpath_selector
Generate a XPath selector for the current element
RETURNS DESCRIPTION
str A string of the generated selector.
generate_full_xpath_selector property ¶
generate_full_xpath_selector
Generate a complete XPath selector for the current element
RETURNS DESCRIPTION
str A string of the generated selector.
get_all_text ¶
get_all_text(
separator="\n",
strip=False,
ignore_tags=("script", "style"),
valid_values=True,
)
Get all child strings of this element, concatenated using the given separator.
PARAMETER DESCRIPTION
separator Strings will be concatenated using this separator.
TYPE: strDEFAULT: '\n'
strip If True, strings will be stripped before being concatenated.
TYPE: boolDEFAULT: False
ignore_tags A tuple of all tag names you want to ignore
TYPE: TupleDEFAULT: ('script', 'style')
valid_values If enabled, elements with text-content that is empty or only whitespaces will be ignored
TYPE: boolDEFAULT: True
RETURNS DESCRIPTION
TextHandler A TextHandler
Source code in scrapling/parser.py
urljoin ¶
urljoin(relative_url)
Join this Adaptor's url with a relative url to form an absolute full URL.
Source code in scrapling/parser.py
prettify ¶
prettify()
Return a prettified version of the element's inner html-code
Source code in scrapling/parser.py
has_class ¶
has_class(class_name)
Check if element has a specific class
PARAMETER DESCRIPTION
class_name The class name to check for
TYPE: str
RETURNS DESCRIPTION
bool True if element has class with that name otherwise False
Source code in scrapling/parser.py
iterancestors ¶
iterancestors()
Return a generator that loops over all ancestors of the element, starting with element's parent.
Source code in scrapling/parser.py
find_ancestor ¶
find_ancestor(func)
Loop over all ancestors of the element till one match the passed function
PARAMETER DESCRIPTION
func A function that takes each ancestor as an argument and returns True/False
TYPE: Callable[[Adaptor], bool]
RETURNS DESCRIPTION
Union[Adaptor, None] The first ancestor that match the function or None otherwise.
Source code in scrapling/parser.py
get ¶
get(default=None)
Source code in scrapling/parser.py
get_all ¶
get_all()
Source code in scrapling/parser.py
relocate ¶
relocate(element, percentage=0, adaptor_type=False)
This function will search again for the element in the page tree, used automatically on page structure change
PARAMETER DESCRIPTION
element The element we want to relocate in the tree
TYPE: Union[Dict, HtmlElement, Adaptor]
percentage The minimum percentage to accept and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing!
TYPE: intDEFAULT: 0
adaptor_type If True, the return result will be converted to Adaptors object
TYPE: boolDEFAULT: False
RETURNS DESCRIPTION
Union[List[Union[HtmlElement, None]], Adaptors] List of pure HTML elements that got the highest matching score or 'Adaptors' object
Source code in scrapling/parser.py
css_first ¶
css_first(
selector,
identifier="",
auto_match=False,
auto_save=False,
percentage=0,
)
Search current tree with CSS3 selectors and return the first result if possible, otherwise return None
Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)
PARAMETER DESCRIPTION
selector The CSS3 selector to be used.
TYPE: str
auto_match Enabled will make function try to relocate the element if it was 'saved' before
TYPE: boolDEFAULT: False
identifier A string that will be used to save/retrieve element's data in auto-matching otherwise the selector will be used.
TYPE: strDEFAULT: ''
auto_save Automatically save new elements for auto_match later
TYPE: boolDEFAULT: False
percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing!
TYPE: intDEFAULT: 0
Source code in scrapling/parser.py
xpath_first ¶
xpath_first(
selector,
identifier="",
auto_match=False,
auto_save=False,
percentage=0,
**kwargs
)
Search current tree with XPath selectors and return the first result if possible, otherwise return None
Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)
Note: Additional keyword arguments will be passed as XPath variables in the XPath expression!
PARAMETER DESCRIPTION
selector The XPath selector to be used.
TYPE: str
auto_match Enabled will make function try to relocate the element if it was 'saved' before
TYPE: boolDEFAULT: False
identifier A string that will be used to save/retrieve element's data in auto-matching otherwise the selector will be used.
TYPE: strDEFAULT: ''
auto_save Automatically save new elements for auto_match later
TYPE: boolDEFAULT: False
percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing!
TYPE: intDEFAULT: 0
Source code in scrapling/parser.py
css ¶
css(
selector,
identifier="",
auto_match=False,
auto_save=False,
percentage=0,
)
Search current tree with CSS3 selectors
Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)
PARAMETER DESCRIPTION
selector The CSS3 selector to be used.
TYPE: str
auto_match Enabled will make function try to relocate the element if it was 'saved' before
TYPE: boolDEFAULT: False
identifier A string that will be used to save/retrieve element's data in auto-matching otherwise the selector will be used.
TYPE: strDEFAULT: ''
auto_save Automatically save new elements for auto_match later
TYPE: boolDEFAULT: False
percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing!
TYPE: intDEFAULT: 0
RETURNS DESCRIPTION
Union[Adaptors[Adaptor], List, TextHandlers[TextHandler]] List as :class:Adaptors
Source code in scrapling/parser.py
xpath ¶
xpath(
selector,
identifier="",
auto_match=False,
auto_save=False,
percentage=0,
**kwargs
)
Search current tree with XPath selectors
Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)
Note: Additional keyword arguments will be passed as XPath variables in the XPath expression!
PARAMETER DESCRIPTION
selector The XPath selector to be used.
TYPE: str
auto_match Enabled will make function try to relocate the element if it was 'saved' before
TYPE: boolDEFAULT: False
identifier A string that will be used to save/retrieve element's data in auto-matching otherwise the selector will be used.
TYPE: strDEFAULT: ''
auto_save Automatically save new elements for auto_match later
TYPE: boolDEFAULT: False
percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing!
TYPE: intDEFAULT: 0
RETURNS DESCRIPTION
Union[Adaptors[Adaptor], List, TextHandlers[TextHandler]] List as :class:Adaptors
Source code in scrapling/parser.py
find_all ¶
find_all(*args, **kwargs)
Find elements by filters of your creations for ease..
PARAMETER DESCRIPTION
args Tag name(s), an iterable of tag names, regex patterns, function, or a dictionary of elements' attributes. Leave empty for selecting all.
TYPE: Union[str, Iterable[str], Pattern, Callable, Dict[str, str]]DEFAULT: ()
kwargs The attributes you want to filter elements based on it.
TYPE: strDEFAULT: {}
RETURNS DESCRIPTION
Adaptors The Adaptors object of the elements or empty list
Source code in scrapling/parser.py
find ¶
find(*args, **kwargs)
Find elements by filters of your creations for ease then return the first result. Otherwise return None.
PARAMETER DESCRIPTION
args Tag name(s), an iterable of tag names, regex patterns, function, or a dictionary of elements' attributes. Leave empty for selecting all.
TYPE: Union[str, Iterable[str], Pattern, Callable, Dict[str, str]]DEFAULT: ()
kwargs The attributes you want to filter elements based on it.
TYPE: strDEFAULT: {}
RETURNS DESCRIPTION
Union[Adaptor, None] The Adaptor object of the element or None if the result didn't match
Source code in scrapling/parser.py
save ¶
save(element, identifier)
Saves the element's unique properties to the storage for retrieval and relocation later
PARAMETER DESCRIPTION
element The element itself that we want to save to storage, it can be a Adaptor or pure HtmlElement
TYPE: Union[Adaptor, HtmlElement]
identifier This is the identifier that will be used to retrieve the element later from the storage. See the docs for more info.
TYPE: str
Source code in scrapling/parser.py
retrieve ¶
retrieve(identifier)
Using the identifier, we search the storage and return the unique properties of the element
PARAMETER DESCRIPTION
identifier This is the identifier that will be used to retrieve the element from the storage. See the docs for more info.
TYPE: str
RETURNS DESCRIPTION
Optional[Dict] A dictionary of the unique properties
Source code in scrapling/parser.py
json ¶
json()
Return json response if the response is jsonable otherwise throws error
Source code in scrapling/parser.py
re ¶
re(
regex,
replace_entities=True,
clean_match=False,
case_sensitive=True,
)
Apply the given regex to the current text and return a list of strings with the matches.
PARAMETER DESCRIPTION
regex Can be either a compiled regular expression or a string.
TYPE: Union[str, Pattern[str]]
replace_entities if enabled character entity references are replaced by their corresponding character
TYPE: boolDEFAULT: True
clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching
TYPE: boolDEFAULT: False
case_sensitive if disabled, function will set the regex to ignore letters case while compiling it
TYPE: boolDEFAULT: True
Source code in scrapling/parser.py
re_first ¶
re_first(
regex,
default=None,
replace_entities=True,
clean_match=False,
case_sensitive=True,
)
Apply the given regex to text and return the first match if found, otherwise return the default value.
PARAMETER DESCRIPTION
regex Can be either a compiled regular expression or a string.
TYPE: Union[str, Pattern[str]]
default The default value to be returned if there is no match
DEFAULT: None
replace_entities if enabled character entity references are replaced by their corresponding character
TYPE: boolDEFAULT: True
clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching
TYPE: boolDEFAULT: False
case_sensitive if disabled, function will set the regex to ignore letters case while compiling it
TYPE: boolDEFAULT: True
Source code in scrapling/parser.py
find_similar ¶
find_similar(
similarity_threshold=0.2,
ignore_attributes=("href", "src"),
match_text=False,
)
Find elements that are in the same tree depth in the page with the same tag name and same parent tag etc... then return the ones that match the current element attributes with percentage higher than the input threshold.
This function is inspired by AutoScraper and made for cases where you, for example, found a product div inside a products-list container and want to find other products using that that element as a starting point EXCEPT this function works in any case without depending on the element type.
PARAMETER DESCRIPTION
similarity_threshold The percentage to use while comparing elements attributes. Note: Elements found before attributes matching/comparison will be sharing the same depth, same tag name, same parent tag name, and same grand parent tag name. So they are 99% likely to be correct unless your are extremely unlucky then attributes matching comes into play so basically don't play with this number unless you are getting the results you don't want. Also, if current element doesn't have attributes and the similar element as well, then it's a 100% match.
TYPE: floatDEFAULT: 0.2
ignore_attributes Attribute names passed will be ignored while matching the attributes in last step. The default value is to ignore href and src as URLs can change a lot between elements so it's unreliable
TYPE: Union[List, Tuple]DEFAULT: ('href', 'src')
match_text If True, elements text content will be taken into calculation while matching. Not recommended to use in normal cases but it depends.
TYPE: boolDEFAULT: False
RETURNS DESCRIPTION
Union[Adaptors[Adaptor], List] A Adaptors container of Adaptor objects or empty list
Source code in scrapling/parser.py
find_by_text ¶
find_by_text(
text,
first_match=True,
partial=False,
case_sensitive=False,
clean_match=True,
)
Find elements that its text content fully/partially matches input.
PARAMETER DESCRIPTION
text Text query to match
TYPE: str
first_match Return first element that matches conditions, enabled by default
TYPE: boolDEFAULT: True
partial If enabled, function return elements that contains the input text
TYPE: boolDEFAULT: False
case_sensitive if enabled, letters case will be taken into consideration
TYPE: boolDEFAULT: False
clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching
TYPE: boolDEFAULT: True
Source code in scrapling/parser.py
find_by_regex ¶
find_by_regex(
query,
first_match=True,
case_sensitive=False,
clean_match=True,
)
Find elements that its text content matches the input regex pattern.
PARAMETER DESCRIPTION
query Regex query/pattern to match
TYPE: Union[str, Pattern[str]]
first_match Return first element that matches conditions, enabled by default
TYPE: boolDEFAULT: True
case_sensitive if enabled, letters case will be taken into consideration in the regex
TYPE: boolDEFAULT: False
clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching
TYPE: boolDEFAULT: True
Source code in scrapling/parser.py
scrapling.parser.Adaptors ¶
Bases: List[Adaptor]
The :class:Adaptors class is a subclass of the builtin List class, which provides a few additional methods.
extract_first class-attribute instance-attribute ¶
extract_first = get
get_all class-attribute instance-attribute ¶
get_all = extract
first property ¶
first
Returns the first item of the current list or None if the list is empty
last property ¶
last
Returns the last item of the current list or None if the list is empty
xpath ¶
xpath(
selector,
identifier="",
auto_save=False,
percentage=0,
**kwargs
)
Call the .xpath() method for each element in this list and return their results as another :class:Adaptors.
Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)
Note: Additional keyword arguments will be passed as XPath variables in the XPath expression!
PARAMETER DESCRIPTION
selector The XPath selector to be used.
TYPE: str
identifier A string that will be used to retrieve element's data in auto-matching otherwise the selector will be used.
TYPE: strDEFAULT: ''
auto_save Automatically save new elements for auto_match later
TYPE: boolDEFAULT: False
percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing!
TYPE: intDEFAULT: 0
RETURNS DESCRIPTION
Adaptors[Adaptor] List as :class:Adaptors
Source code in scrapling/parser.py
css ¶
css(selector, identifier='', auto_save=False, percentage=0)
Call the .css() method for each element in this list and return their results flattened as another :class:Adaptors.
Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)
PARAMETER DESCRIPTION
selector The CSS3 selector to be used.
TYPE: str
identifier A string that will be used to retrieve element's data in auto-matching otherwise the selector will be used.
TYPE: strDEFAULT: ''
auto_save Automatically save new elements for auto_match later
TYPE: boolDEFAULT: False
percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing!
TYPE: intDEFAULT: 0
RETURNS DESCRIPTION
Adaptors[Adaptor] List as :class:Adaptors
Source code in scrapling/parser.py
re ¶
re(
regex,
replace_entities=True,
clean_match=False,
case_sensitive=True,
)
Call the .re() method for each element in this list and return their results flattened as List of TextHandler.
PARAMETER DESCRIPTION
regex Can be either a compiled regular expression or a string.
TYPE: Union[str, Pattern[str]]
replace_entities if enabled character entity references are replaced by their corresponding character
TYPE: boolDEFAULT: True
clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching
TYPE: boolDEFAULT: False
case_sensitive if disabled, function will set the regex to ignore letters case while compiling it
TYPE: boolDEFAULT: True
Source code in scrapling/parser.py
re_first ¶
re_first(
regex,
default=None,
replace_entities=True,
clean_match=False,
case_sensitive=True,
)
Call the .re_first() method for each element in this list and return the first result or the default value otherwise.
PARAMETER DESCRIPTION
regex Can be either a compiled regular expression or a string.
TYPE: Union[str, Pattern[str]]
default The default value to be returned if there is no match
DEFAULT: None
replace_entities if enabled character entity references are replaced by their corresponding character
TYPE: boolDEFAULT: True
clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching
TYPE: boolDEFAULT: False
case_sensitive if disabled, function will set the regex to ignore letters case while compiling it
TYPE: boolDEFAULT: True
Source code in scrapling/parser.py
search ¶
search(func)
Loop over all current elements and return the first element that matches the passed function
PARAMETER DESCRIPTION
func A function that takes each element as an argument and returns True/False
TYPE: Callable[[Adaptor], bool]
RETURNS DESCRIPTION
Union[Adaptor, None] The first element that match the function or None otherwise.
Source code in scrapling/parser.py
filter ¶
filter(func)
Filter current elements based on the passed function
PARAMETER DESCRIPTION
func A function that takes each element as an argument and returns True/False
TYPE: Callable[[Adaptor], bool]
RETURNS DESCRIPTION
Adaptors[Adaptor] The new Adaptors object or empty list otherwise.
Source code in scrapling/parser.py
get ¶
get(default=None)
Returns the first item of the current list
PARAMETER DESCRIPTION
default the default value to return if the current list is empty
DEFAULT: None
Source code in scrapling/parser.py
extract ¶
extract()
Source code in scrapling/parser.py
Fetchers Classes¶
Here's the reference information for all fetcher-type classes' parameters, attributes, and methods.
You can import all of them directly like below:
from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
scrapling.fetchers.Fetcher ¶
Fetcher(*args, **kwargs)
Bases: BaseFetcher
A basic Fetcher class type that can only do basic GET, POST, PUT, and DELETE HTTP requests based on httpx.
Any additional keyword arguments passed to the methods below are passed to the respective httpx's method directly.
Source code in scrapling/engines/toolbelt/custom.py
huge_tree class-attribute instance-attribute ¶
huge_tree = True
auto_match class-attribute instance-attribute ¶
auto_match = False
storage class-attribute instance-attribute ¶
storage = SQLiteStorageSystem
keep_cdata class-attribute instance-attribute ¶
keep_cdata = False
storage_args class-attribute instance-attribute ¶
storage_args = None
keep_comments class-attribute instance-attribute ¶
keep_comments = False
automatch_domain class-attribute instance-attribute ¶
automatch_domain = None
parser_keywords class-attribute instance-attribute ¶
parser_keywords = (
"huge_tree",
"auto_match",
"storage",
"keep_cdata",
"storage_args",
"keep_comments",
"automatch_domain",
)
get classmethod ¶
get(
url,
follow_redirects=True,
timeout=10,
stealthy_headers=True,
proxy=None,
retries=3,
custom_config=None,
**kwargs
)
Make basic HTTP GET request for you but with some added flavors.
PARAMETER DESCRIPTION
url Target url.
TYPE: str
follow_redirects As the name says -- if enabled (default), redirects will be followed.
TYPE: boolDEFAULT: True
timeout The time to wait for the request to finish in seconds. The default is 10 seconds.
TYPE: Optional[Union[int, float]]DEFAULT: 10
stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request had came from Google's search of this URL's domain.
TYPE: boolDEFAULT: True
proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030
TYPE: Optional[str]DEFAULT: None
retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
TYPE: Optional[int]DEFAULT: 3
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
kwargs Any additional keyword arguments are passed directly to httpx.get() function so check httpx documentation for details.
TYPE: DictDEFAULT: {}
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
post classmethod ¶
post(
url,
follow_redirects=True,
timeout=10,
stealthy_headers=True,
proxy=None,
retries=3,
custom_config=None,
**kwargs
)
Make basic HTTP POST request for you but with some added flavors.
PARAMETER DESCRIPTION
url Target url.
TYPE: str
follow_redirects As the name says -- if enabled (default), redirects will be followed.
TYPE: boolDEFAULT: True
timeout The time to wait for the request to finish in seconds. The default is 10 seconds.
TYPE: Optional[Union[int, float]]DEFAULT: 10
stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain.
TYPE: boolDEFAULT: True
proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030
TYPE: Optional[str]DEFAULT: None
retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
TYPE: Optional[int]DEFAULT: 3
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
kwargs Any additional keyword arguments are passed directly to httpx.post() function so check httpx documentation for details.
TYPE: DictDEFAULT: {}
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
put classmethod ¶
put(
url,
follow_redirects=True,
timeout=10,
stealthy_headers=True,
proxy=None,
retries=3,
custom_config=None,
**kwargs
)
Make basic HTTP PUT request for you but with some added flavors.
PARAMETER DESCRIPTION
url Target url
TYPE: str
follow_redirects As the name says -- if enabled (default), redirects will be followed.
TYPE: boolDEFAULT: True
timeout The time to wait for the request to finish in seconds. The default is 10 seconds.
TYPE: Optional[Union[int, float]]DEFAULT: 10
stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain.
TYPE: boolDEFAULT: True
proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030
TYPE: Optional[str]DEFAULT: None
retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
TYPE: Optional[int]DEFAULT: 3
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
kwargs Any additional keyword arguments are passed directly to httpx.put() function so check httpx documentation for details.
TYPE: DictDEFAULT: {}
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
delete classmethod ¶
delete(
url,
follow_redirects=True,
timeout=10,
stealthy_headers=True,
proxy=None,
retries=3,
custom_config=None,
**kwargs
)
Make basic HTTP DELETE request for you but with some added flavors.
PARAMETER DESCRIPTION
url Target url
TYPE: str
follow_redirects As the name says -- if enabled (default), redirects will be followed.
TYPE: boolDEFAULT: True
timeout The time to wait for the request to finish in seconds. The default is 10 seconds.
TYPE: Optional[Union[int, float]]DEFAULT: 10
stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain.
TYPE: boolDEFAULT: True
proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030
TYPE: Optional[str]DEFAULT: None
retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
TYPE: Optional[int]DEFAULT: 3
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
kwargs Any additional keyword arguments are passed directly to httpx.delete() function so check httpx documentation for details.
TYPE: DictDEFAULT: {}
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
display_config classmethod ¶
display_config()
Source code in scrapling/engines/toolbelt/custom.py
configure classmethod ¶
configure(**kwargs)
Set multiple arguments for the parser at once globally
PARAMETER DESCRIPTION
kwargs The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, auto_match, storage, storage_args, automatch_domain
DEFAULT: {}
Source code in scrapling/engines/toolbelt/custom.py
scrapling.fetchers.AsyncFetcher ¶
AsyncFetcher(*args, **kwargs)
Bases: Fetcher
Source code in scrapling/engines/toolbelt/custom.py
huge_tree class-attribute instance-attribute ¶
huge_tree = True
auto_match class-attribute instance-attribute ¶
auto_match = False
storage class-attribute instance-attribute ¶
storage = SQLiteStorageSystem
keep_cdata class-attribute instance-attribute ¶
keep_cdata = False
storage_args class-attribute instance-attribute ¶
storage_args = None
keep_comments class-attribute instance-attribute ¶
keep_comments = False
automatch_domain class-attribute instance-attribute ¶
automatch_domain = None
parser_keywords class-attribute instance-attribute ¶
parser_keywords = (
"huge_tree",
"auto_match",
"storage",
"keep_cdata",
"storage_args",
"keep_comments",
"automatch_domain",
)
get async classmethod ¶
get(
url,
follow_redirects=True,
timeout=10,
stealthy_headers=True,
proxy=None,
retries=3,
custom_config=None,
**kwargs
)
Make basic HTTP GET request for you but with some added flavors.
PARAMETER DESCRIPTION
url Target url.
TYPE: str
follow_redirects As the name says -- if enabled (default), redirects will be followed.
TYPE: boolDEFAULT: True
timeout The time to wait for the request to finish in seconds. The default is 10 seconds.
TYPE: Optional[Union[int, float]]DEFAULT: 10
stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request had came from Google's search of this URL's domain.
TYPE: boolDEFAULT: True
proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030
TYPE: Optional[str]DEFAULT: None
retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
TYPE: Optional[int]DEFAULT: 3
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
kwargs Any additional keyword arguments are passed directly to httpx.get() function so check httpx documentation for details.
TYPE: DictDEFAULT: {}
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
post async classmethod ¶
post(
url,
follow_redirects=True,
timeout=10,
stealthy_headers=True,
proxy=None,
retries=3,
custom_config=None,
**kwargs
)
Make basic HTTP POST request for you but with some added flavors.
PARAMETER DESCRIPTION
url Target url.
TYPE: str
follow_redirects As the name says -- if enabled (default), redirects will be followed.
TYPE: boolDEFAULT: True
timeout The time to wait for the request to finish in seconds. The default is 10 seconds.
TYPE: Optional[Union[int, float]]DEFAULT: 10
stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain.
TYPE: boolDEFAULT: True
proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030
TYPE: Optional[str]DEFAULT: None
retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
TYPE: Optional[int]DEFAULT: 3
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
kwargs Any additional keyword arguments are passed directly to httpx.post() function so check httpx documentation for details.
TYPE: DictDEFAULT: {}
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
put async classmethod ¶
put(
url,
follow_redirects=True,
timeout=10,
stealthy_headers=True,
proxy=None,
retries=3,
custom_config=None,
**kwargs
)
Make basic HTTP PUT request for you but with some added flavors.
PARAMETER DESCRIPTION
url Target url
TYPE: str
follow_redirects As the name says -- if enabled (default), redirects will be followed.
TYPE: boolDEFAULT: True
timeout The time to wait for the request to finish in seconds. The default is 10 seconds.
TYPE: Optional[Union[int, float]]DEFAULT: 10
stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain.
TYPE: boolDEFAULT: True
proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030
TYPE: Optional[str]DEFAULT: None
retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
TYPE: Optional[int]DEFAULT: 3
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
kwargs Any additional keyword arguments are passed directly to httpx.put() function so check httpx documentation for details.
TYPE: DictDEFAULT: {}
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
delete async classmethod ¶
delete(
url,
follow_redirects=True,
timeout=10,
stealthy_headers=True,
proxy=None,
retries=3,
custom_config=None,
**kwargs
)
Make basic HTTP DELETE request for you but with some added flavors.
PARAMETER DESCRIPTION
url Target url
TYPE: str
follow_redirects As the name says -- if enabled (default), redirects will be followed.
TYPE: boolDEFAULT: True
timeout The time to wait for the request to finish in seconds. The default is 10 seconds.
TYPE: Optional[Union[int, float]]DEFAULT: 10
stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain.
TYPE: boolDEFAULT: True
proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030
TYPE: Optional[str]DEFAULT: None
retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
TYPE: Optional[int]DEFAULT: 3
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
kwargs Any additional keyword arguments are passed directly to httpx.delete() function so check httpx documentation for details.
TYPE: DictDEFAULT: {}
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
display_config classmethod ¶
display_config()
Source code in scrapling/engines/toolbelt/custom.py
configure classmethod ¶
configure(**kwargs)
Set multiple arguments for the parser at once globally
PARAMETER DESCRIPTION
kwargs The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, auto_match, storage, storage_args, automatch_domain
DEFAULT: {}
Source code in scrapling/engines/toolbelt/custom.py
scrapling.fetchers.PlayWrightFetcher ¶
PlayWrightFetcher(*args, **kwargs)
Bases: BaseFetcher
A Fetcher class type that provide many options, all of them are based on PlayWright.
Using this Fetcher class, you can do requests with: - Vanilla Playwright without any modifications other than the ones you chose. - Stealthy Playwright with the stealth mode I wrote for it. It's still a work in progress but it bypasses many online tests like bot.sannysoft.com Some of the things stealth mode does include: 1) Patches the CDP runtime fingerprint. 2) Mimics some of the real browsers' properties by injecting several JS files and using custom options. 3) Using custom flags on launch to hide Playwright even more and make it faster. 4) Generates real browser's headers of the same type and same user OS then append it to the request. - Real browsers by passing the real_chrome argument or the CDP URL of your browser to be controlled by the Fetcher and most of the options can be enabled on it. - NSTBrowser's docker browserless option by passing the CDP URL and enabling nstbrowser_mode option.
Note that these are the main options with PlayWright but it can be mixed together.
Source code in scrapling/engines/toolbelt/custom.py
huge_tree class-attribute instance-attribute ¶
huge_tree = True
auto_match class-attribute instance-attribute ¶
auto_match = False
storage class-attribute instance-attribute ¶
storage = SQLiteStorageSystem
keep_cdata class-attribute instance-attribute ¶
keep_cdata = False
storage_args class-attribute instance-attribute ¶
storage_args = None
keep_comments class-attribute instance-attribute ¶
keep_comments = False
automatch_domain class-attribute instance-attribute ¶
automatch_domain = None
parser_keywords class-attribute instance-attribute ¶
parser_keywords = (
"huge_tree",
"auto_match",
"storage",
"keep_cdata",
"storage_args",
"keep_comments",
"automatch_domain",
)
fetch classmethod ¶
fetch(
url,
headless=True,
disable_resources=None,
useragent=None,
network_idle=False,
timeout=30000,
wait=0,
page_action=None,
wait_selector=None,
wait_selector_state="attached",
hide_canvas=False,
disable_webgl=False,
extra_headers=None,
google_search=True,
proxy=None,
locale="en-US",
stealth=False,
real_chrome=False,
cdp_url=None,
nstbrowser_mode=False,
nstbrowser_config=None,
custom_config=None,
)
Opens up a browser and do your request based on your chosen options below.
PARAMETER DESCRIPTION
url Target url.
TYPE: str
headless Run the browser in headless/hidden (default), or headful/visible mode.
TYPE: Union[bool, str]DEFAULT: True
disable_resources Drop requests of unnecessary resources for speed boost. It depends but it made requests ~25% faster in my tests for some websites. Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
TYPE: boolDEFAULT: None
useragent Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
TYPE: Optional[str]DEFAULT: None
network_idle Wait for the page until there are no network connections for at least 500 ms.
TYPE: boolDEFAULT: False
timeout The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000.
TYPE: Optional[float]DEFAULT: 30000
wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning Response object.
TYPE: Optional[int]DEFAULT: 0
locale Set the locale for the browser if wanted. The default value is en-US.
TYPE: Optional[str]DEFAULT: 'en-US'
page_action Added for automation. A function that takes the page object, does the automation you need, then returns page again.
TYPE: Optional[Callable]DEFAULT: None
wait_selector Wait for a specific css selector to be in a specific state.
TYPE: Optional[str]DEFAULT: None
wait_selector_state The state to wait for the selector given with wait_selector. Default state is attached.
TYPE: SelectorWaitStatesDEFAULT: 'attached'
stealth Enables stealth mode, check the documentation to see what stealth mode does currently.
TYPE: boolDEFAULT: False
real_chrome If you have chrome browser installed on your device, enable this and the Fetcher will launch an instance of your browser and use it.
TYPE: boolDEFAULT: False
hide_canvas Add random noise to canvas operations to prevent fingerprinting.
TYPE: boolDEFAULT: False
disable_webgl Disables WebGL and WebGL 2.0 support entirely.
TYPE: boolDEFAULT: False
google_search Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name.
TYPE: boolDEFAULT: True
extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together.
TYPE: Optional[Dict[str, str]]DEFAULT: None
proxy The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
TYPE: Optional[Union[str, Dict[str, str]]]DEFAULT: None
cdp_url Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP.
TYPE: Optional[str]DEFAULT: None
nstbrowser_mode Enables NSTBrowser mode, it have to be used with cdp_url argument or it will get completely ignored.
TYPE: boolDEFAULT: False
nstbrowser_config The config you want to send with requests to the NSTBrowser. If left empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config.
TYPE: Optional[Dict]DEFAULT: None
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
async_fetch async classmethod ¶
async_fetch(
url,
headless=True,
disable_resources=None,
useragent=None,
network_idle=False,
timeout=30000,
wait=0,
page_action=None,
wait_selector=None,
wait_selector_state="attached",
hide_canvas=False,
disable_webgl=False,
extra_headers=None,
google_search=True,
proxy=None,
locale="en-US",
stealth=False,
real_chrome=False,
cdp_url=None,
nstbrowser_mode=False,
nstbrowser_config=None,
custom_config=None,
)
Opens up a browser and do your request based on your chosen options below.
PARAMETER DESCRIPTION
url Target url.
TYPE: str
headless Run the browser in headless/hidden (default), or headful/visible mode.
TYPE: Union[bool, str]DEFAULT: True
disable_resources Drop requests of unnecessary resources for speed boost. It depends but it made requests ~25% faster in my tests for some websites. Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
TYPE: boolDEFAULT: None
useragent Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
TYPE: Optional[str]DEFAULT: None
network_idle Wait for the page until there are no network connections for at least 500 ms.
TYPE: boolDEFAULT: False
timeout The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000.
TYPE: Optional[float]DEFAULT: 30000
wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning Response object.
TYPE: Optional[int]DEFAULT: 0
locale Set the locale for the browser if wanted. The default value is en-US.
TYPE: Optional[str]DEFAULT: 'en-US'
page_action Added for automation. A function that takes the page object, does the automation you need, then returns page again.
TYPE: Optional[Callable]DEFAULT: None
wait_selector Wait for a specific css selector to be in a specific state.
TYPE: Optional[str]DEFAULT: None
wait_selector_state The state to wait for the selector given with wait_selector. Default state is attached.
TYPE: SelectorWaitStatesDEFAULT: 'attached'
stealth Enables stealth mode, check the documentation to see what stealth mode does currently.
TYPE: boolDEFAULT: False
real_chrome If you have chrome browser installed on your device, enable this and the Fetcher will launch an instance of your browser and use it.
TYPE: boolDEFAULT: False
hide_canvas Add random noise to canvas operations to prevent fingerprinting.
TYPE: boolDEFAULT: False
disable_webgl Disables WebGL and WebGL 2.0 support entirely.
TYPE: boolDEFAULT: False
google_search Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name.
TYPE: boolDEFAULT: True
extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together.
TYPE: Optional[Dict[str, str]]DEFAULT: None
proxy The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
TYPE: Optional[Union[str, Dict[str, str]]]DEFAULT: None
cdp_url Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP.
TYPE: Optional[str]DEFAULT: None
nstbrowser_mode Enables NSTBrowser mode, it have to be used with cdp_url argument or it will get completely ignored.
TYPE: boolDEFAULT: False
nstbrowser_config The config you want to send with requests to the NSTBrowser. If left empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config.
TYPE: Optional[Dict]DEFAULT: None
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
display_config classmethod ¶
display_config()
Source code in scrapling/engines/toolbelt/custom.py
configure classmethod ¶
configure(**kwargs)
Set multiple arguments for the parser at once globally
PARAMETER DESCRIPTION
kwargs The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, auto_match, storage, storage_args, automatch_domain
DEFAULT: {}
Source code in scrapling/engines/toolbelt/custom.py
scrapling.fetchers.StealthyFetcher ¶
StealthyFetcher(*args, **kwargs)
Bases: BaseFetcher
A Fetcher class type that is completely stealthy fetcher that uses a modified version of Firefox.
It works as real browsers passing almost all online tests/protections based on Camoufox. Other added flavors include setting the faked OS fingerprints to match the user's OS and the referer of every request is set as if this request came from Google's search of this URL's domain.
Source code in scrapling/engines/toolbelt/custom.py
huge_tree class-attribute instance-attribute ¶
huge_tree = True
auto_match class-attribute instance-attribute ¶
auto_match = False
storage class-attribute instance-attribute ¶
storage = SQLiteStorageSystem
keep_cdata class-attribute instance-attribute ¶
keep_cdata = False
storage_args class-attribute instance-attribute ¶
storage_args = None
keep_comments class-attribute instance-attribute ¶
keep_comments = False
automatch_domain class-attribute instance-attribute ¶
automatch_domain = None
parser_keywords class-attribute instance-attribute ¶
parser_keywords = (
"huge_tree",
"auto_match",
"storage",
"keep_cdata",
"storage_args",
"keep_comments",
"automatch_domain",
)
fetch classmethod ¶
fetch(
url,
headless=True,
block_images=False,
disable_resources=False,
block_webrtc=False,
allow_webgl=True,
network_idle=False,
addons=None,
wait=0,
timeout=30000,
page_action=None,
wait_selector=None,
humanize=True,
wait_selector_state="attached",
google_search=True,
extra_headers=None,
proxy=None,
os_randomize=False,
disable_ads=False,
geoip=False,
custom_config=None,
additional_arguments=None,
)
Opens up a browser and do your request based on your chosen options below.
PARAMETER DESCRIPTION
url Target url.
TYPE: str
headless Run the browser in headless/hidden (default), 'virtual' screen mode, or headful/visible mode.
TYPE: Union[bool, Literal[virtual]]DEFAULT: True
block_images Prevent the loading of images through Firefox preferences. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
TYPE: boolDEFAULT: False
disable_resources Drop requests of unnecessary resources for a speed boost. It depends but it made requests ~25% faster in my tests for some websites. Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
TYPE: boolDEFAULT: False
block_webrtc Blocks WebRTC entirely.
TYPE: boolDEFAULT: False
addons List of Firefox addons to use. Must be paths to extracted addons.
TYPE: Optional[List[str]]DEFAULT: None
disable_ads Disabled by default, this installs uBlock Origin addon on the browser if enabled.
TYPE: boolDEFAULT: False
humanize Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
TYPE: Optional[Union[bool, float]]DEFAULT: True
allow_webgl Enabled by default. Disabling it WebGL not recommended as many WAFs now checks if WebGL is enabled.
TYPE: boolDEFAULT: True
geoip Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
TYPE: boolDEFAULT: False
network_idle Wait for the page until there are no network connections for at least 500 ms.
TYPE: boolDEFAULT: False
os_randomize If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
TYPE: boolDEFAULT: False
timeout The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000.
TYPE: Optional[float]DEFAULT: 30000
wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning Response object.
TYPE: Optional[int]DEFAULT: 0
page_action Added for automation. A function that takes the page object, does the automation you need, then returns page again.
TYPE: CallableDEFAULT: None
wait_selector Wait for a specific css selector to be in a specific state.
TYPE: Optional[str]DEFAULT: None
wait_selector_state The state to wait for the selector given with wait_selector. Default state is attached.
TYPE: SelectorWaitStatesDEFAULT: 'attached'
google_search Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name.
TYPE: boolDEFAULT: True
extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together.
TYPE: Optional[Dict[str, str]]DEFAULT: None
proxy The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
TYPE: Optional[Union[str, Dict[str, str]]]DEFAULT: None
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
additional_arguments Additional arguments to be passed to Camoufox as additional settings and it takes higher priority than Scrapling's settings.
TYPE: DictDEFAULT: None
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
async_fetch async classmethod ¶
async_fetch(
url,
headless=True,
block_images=False,
disable_resources=False,
block_webrtc=False,
allow_webgl=True,
network_idle=False,
addons=None,
wait=0,
timeout=30000,
page_action=None,
wait_selector=None,
humanize=True,
wait_selector_state="attached",
google_search=True,
extra_headers=None,
proxy=None,
os_randomize=False,
disable_ads=False,
geoip=False,
custom_config=None,
additional_arguments=None,
)
Opens up a browser and do your request based on your chosen options below.
PARAMETER DESCRIPTION
url Target url.
TYPE: str
headless Run the browser in headless/hidden (default), 'virtual' screen mode, or headful/visible mode.
TYPE: Union[bool, Literal[virtual]]DEFAULT: True
block_images Prevent the loading of images through Firefox preferences. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
TYPE: boolDEFAULT: False
disable_resources Drop requests of unnecessary resources for a speed boost. It depends but it made requests ~25% faster in my tests for some websites. Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
TYPE: boolDEFAULT: False
block_webrtc Blocks WebRTC entirely.
TYPE: boolDEFAULT: False
addons List of Firefox addons to use. Must be paths to extracted addons.
TYPE: Optional[List[str]]DEFAULT: None
disable_ads Disabled by default, this installs uBlock Origin addon on the browser if enabled.
TYPE: boolDEFAULT: False
humanize Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
TYPE: Optional[Union[bool, float]]DEFAULT: True
allow_webgl Enabled by default. Disabling it WebGL not recommended as many WAFs now checks if WebGL is enabled.
TYPE: boolDEFAULT: True
geoip Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
TYPE: boolDEFAULT: False
network_idle Wait for the page until there are no network connections for at least 500 ms.
TYPE: boolDEFAULT: False
os_randomize If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
TYPE: boolDEFAULT: False
timeout The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000
TYPE: Optional[float]DEFAULT: 30000
wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning Response object.
TYPE: Optional[int]DEFAULT: 0
page_action Added for automation. A function that takes the page object, does the automation you need, then returns page again.
TYPE: CallableDEFAULT: None
wait_selector Wait for a specific css selector to be in a specific state.
TYPE: Optional[str]DEFAULT: None
wait_selector_state The state to wait for the selector given with wait_selector. Default state is attached.
TYPE: SelectorWaitStatesDEFAULT: 'attached'
google_search Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name.
TYPE: boolDEFAULT: True
extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together.
TYPE: Optional[Dict[str, str]]DEFAULT: None
proxy The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
TYPE: Optional[Union[str, Dict[str, str]]]DEFAULT: None
custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
TYPE: DictDEFAULT: None
additional_arguments Additional arguments to be passed to Camoufox as additional settings and it takes higher priority than Scrapling's settings.
TYPE: DictDEFAULT: None
RETURNS DESCRIPTION
Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers
Source code in scrapling/fetchers.py
display_config classmethod ¶
display_config()
Source code in scrapling/engines/toolbelt/custom.py
configure classmethod ¶
configure(**kwargs)
Set multiple arguments for the parser at once globally
PARAMETER DESCRIPTION
kwargs The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, auto_match, storage, storage_args, automatch_domain
DEFAULT: {}
Source code in scrapling/engines/toolbelt/custom.py
Custom Types API Reference¶
Here's the reference information for all custom types of classes Scrapling implemented, with all their parameters, attributes, and methods.
You can import all of them directly like below:
from scrapling.core.custom_types import TextHandler, TextHandlers, AttributesHandler
scrapling.core.custom_types.TextHandler ¶
Bases: str
Extends standard Python string by adding more functionality
extract class-attribute instance-attribute ¶
extract = get_all
extract_first class-attribute instance-attribute ¶
extract_first = get
split ¶
split(sep=None, maxsplit=-1)
Source code in scrapling/core/custom_types.py
strip ¶
strip(chars=None)
Source code in scrapling/core/custom_types.py
lstrip ¶
lstrip(chars=None)
Source code in scrapling/core/custom_types.py
rstrip ¶
rstrip(chars=None)
Source code in scrapling/core/custom_types.py
capitalize ¶
capitalize()
Source code in scrapling/core/custom_types.py
casefold ¶
casefold()
Source code in scrapling/core/custom_types.py
center ¶
center(width, fillchar=' ')
Source code in scrapling/core/custom_types.py
expandtabs ¶
expandtabs(tabsize=8)
Source code in scrapling/core/custom_types.py
format ¶
format(*args, **kwargs)
Source code in scrapling/core/custom_types.py
format_map ¶
format_map(mapping)
Source code in scrapling/core/custom_types.py
join ¶
join(iterable)
Source code in scrapling/core/custom_types.py
ljust ¶
ljust(width, fillchar=' ')
Source code in scrapling/core/custom_types.py
rjust ¶
rjust(width, fillchar=' ')
Source code in scrapling/core/custom_types.py
swapcase ¶
swapcase()
Source code in scrapling/core/custom_types.py
title ¶
title()
Source code in scrapling/core/custom_types.py
translate ¶
translate(table)
Source code in scrapling/core/custom_types.py
zfill ¶
zfill(width)
Source code in scrapling/core/custom_types.py
replace ¶
replace(old, new, count=-1)
Source code in scrapling/core/custom_types.py
upper ¶
upper()
Source code in scrapling/core/custom_types.py
lower ¶
lower()
Source code in scrapling/core/custom_types.py
sort ¶
sort(reverse=False)
Return a sorted version of the string
Source code in scrapling/core/custom_types.py
clean ¶
clean()
Return a new version of the string after removing all white spaces and consecutive spaces
Source code in scrapling/core/custom_types.py
get ¶
get(default=None)
Source code in scrapling/core/custom_types.py
get_all ¶
get_all()
Source code in scrapling/core/custom_types.py
json ¶
json()
Return json response if the response is jsonable otherwise throw error
Source code in scrapling/core/custom_types.py
re ¶
re(
regex: Union[str, Pattern[str]],
check_match: Literal[True],
replace_entities: bool = True,
clean_match: bool = False,
case_sensitive: bool = True,
) -> bool
re(
regex: Union[str, Pattern[str]],
replace_entities: bool = True,
clean_match: bool = False,
case_sensitive: bool = True,
check_match: Literal[False] = False,
) -> TextHandlers[TextHandler]
re(
regex,
replace_entities=True,
clean_match=False,
case_sensitive=True,
check_match=False,
)
Apply the given regex to the current text and return a list of strings with the matches.
PARAMETER DESCRIPTION
regex Can be either a compiled regular expression or a string.
TYPE: Union[str, Pattern[str]]
replace_entities if enabled character entity references are replaced by their corresponding character
TYPE: boolDEFAULT: True
clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching
TYPE: boolDEFAULT: False
case_sensitive if disabled, function will set the regex to ignore letters case while compiling it
TYPE: boolDEFAULT: True
check_match used to quickly check if this regex matches or not without any operations on the results
TYPE: boolDEFAULT: False
Source code in scrapling/core/custom_types.py
re_first ¶
re_first(
regex,
default=None,
replace_entities=True,
clean_match=False,
case_sensitive=True,
)
Apply the given regex to text and return the first match if found, otherwise return the default value.
PARAMETER DESCRIPTION
regex Can be either a compiled regular expression or a string.
TYPE: Union[str, Pattern[str]]
default The default value to be returned if there is no match
DEFAULT: None
replace_entities if enabled character entity references are replaced by their corresponding character
TYPE: boolDEFAULT: True
clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching
TYPE: boolDEFAULT: False
case_sensitive if disabled, function will set the regex to ignore letters case while compiling it
TYPE: boolDEFAULT: True
Source code in scrapling/core/custom_types.py
scrapling.core.custom_types.TextHandlers ¶
Bases: List[TextHandler]
The :class:TextHandlers class is a subclass of the builtin List class, which provides a few additional methods.
extract_first class-attribute instance-attribute ¶
extract_first = get
get_all class-attribute instance-attribute ¶
get_all = extract
re ¶
re(
regex,
replace_entities=True,
clean_match=False,
case_sensitive=True,
)
Call the .re() method for each element in this list and return their results flattened as TextHandlers.
PARAMETER DESCRIPTION
regex Can be either a compiled regular expression or a string.
TYPE: Union[str, Pattern[str]]
replace_entities if enabled character entity references are replaced by their corresponding character
TYPE: boolDEFAULT: True
clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching
TYPE: boolDEFAULT: False
case_sensitive if disabled, function will set the regex to ignore letters case while compiling it
TYPE: boolDEFAULT: True
Source code in scrapling/core/custom_types.py
re_first ¶
re_first(
regex,
default=None,
replace_entities=True,
clean_match=False,
case_sensitive=True,
)
Call the .re_first() method for each element in this list and return the first result or the default value otherwise.
PARAMETER DESCRIPTION
regex Can be either a compiled regular expression or a string.
TYPE: Union[str, Pattern[str]]
default The default value to be returned if there is no match
DEFAULT: None
replace_entities if enabled character entity references are replaced by their corresponding character
TYPE: boolDEFAULT: True
clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching
TYPE: boolDEFAULT: False
case_sensitive if disabled, function will set the regex to ignore letters case while compiling it
TYPE: boolDEFAULT: True
Source code in scrapling/core/custom_types.py
get ¶
get(default=None)
Returns the first item of the current list
PARAMETER DESCRIPTION
default the default value to return if the current list is empty
DEFAULT: None
Source code in scrapling/core/custom_types.py
extract ¶
extract()
Source code in scrapling/core/custom_types.py
scrapling.core.custom_types.AttributesHandler ¶
AttributesHandler(mapping=None, **kwargs)
Bases: Mapping[str, _TextHandlerType]
A read-only mapping to use instead of the standard dictionary for the speed boost but at the same time I use it to add more functionalities. If standard dictionary is needed, just convert this class to dictionary with dict function
Source code in scrapling/core/custom_types.py
json_string property ¶
json_string
Convert current attributes to JSON string if the attributes are JSON serializable otherwise throws error
get ¶
get(key, default=None)
Acts like standard dictionary .get() method
Source code in scrapling/core/custom_types.py
search_values ¶
search_values(keyword, partial=False)
Search current attributes by values and return dictionary of each matching item
PARAMETER DESCRIPTION
keyword The keyword to search for in the attributes values
partial If True, the function will search if keyword in each value instead of perfect match
DEFAULT: False
Source code in scrapling/core/custom_types.py
Writing your retrieval system:
Writing your retrieval system
Scrapling uses SQLite by default, but this tutorial covers writing your storage system to store element properties there for auto-matching.
You might want to use FireBase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because the spiders will share with each other.
So first, to make your storage class work, it must do the big 3:
Inherit from the abstract class scrapling.core.storage_adaptors.StorageSystemMixin and accept a string argument, which will be the url argument to maintain the library logic.
Use the decorator functools.lru_cache on top of the class to follow the Singleton design pattern as other classes.
Implement methods save and retrieve, as you see from the type hints:
The method save returns nothing and will get two arguments from the library
The first one is of type lxml.html.HtmlElement, which is the element itself. It must be converted to a dictionary using the function element_to_dict in submodule scrapling.core.utils._StorageTools to keep the same format and save it to your database as you wish.
The second one is a string, the identifier used for retrieval. The combination result of this identifier and the url argument from initialization must be unique for each row, or the auto-match will be messed up.
The method retrieve takes a string, which is the identifier; using it with the url passed on initialization, the element's dictionary is retrieved from the database and returned if it exists; otherwise, it returns None.
If the instructions weren't clear enough for you, you can check my implementation using SQLite3 in storage_adaptors file
If your class meets these criteria, the rest is easy. If you plan to use the library in a threaded application, ensure your class supports it. The default used class is thread-safe.
Some helper functions are added to the abstract class if you want to use them. It's easier to see it for yourself in the code; it's heavily commented :)
Real-World Example: Redis Storage¶
Here's a more practical example generated by AI using Redis:
import redis
import orjson
from functools import lru_cache
from scrapling.core.storage_adaptors import StorageSystemMixin
from scrapling.core.utils import _StorageTools
@lru_cache(None)
class RedisStorage(StorageSystemMixin):
def __init__(self, host='localhost', port=6379, db=0, url=None):
super().__init__(url)
self.redis = redis.Redis(
host=host,
port=port,
db=db,
decode_responses=False
)
def save(self, element, identifier: str) -> None:
# Convert element to dictionary
element_dict = _StorageTools.element_to_dict(element)
# Create key
key = f"scrapling:{self._get_base_url()}:{identifier}"
# Store as JSON
self.redis.set(
key,
orjson.dumps(element_dict)
)
def retrieve(self, identifier: str) -> dict:
# Get data
key = f"scrapling:{self._get_base_url()}:{identifier}"
data = self.redis.get(key)
# Parse JSON if exists
if data:
return orjson.loads(data)
return None
Using Scrapling's custom types:
You can take advantage of the custom-made types for Scrapling and use them outside the library if you want. It's better than copying their code, after all :)
All current types can be imported alone like below¶
>>> from scrapling.core.custom_types import TextHandler, AttributesHandler
>>> somestring = TextHandler('{}')
>>> somestring.json()
'{}'
>>> somedict_1 = AttributesHandler({'a': 1})
>>> somedict_2 = AttributesHandler(a=1)
Note that TextHandler is a subclass of Python's str, so all normal operations/methods that work with Python strings will work. If you want to check for the type in your code, it's better to depend on Python's built-in function issubclass.
The class AttributesHandler is a subclass of collections.abc.Mapping, so it's immutable (read-only), and all operations are inherited from it. The data passed can be accessed later through the _data property, but be careful; it's of type types.MappingProxyType, so it's immutable (read-only) as well (faster than collections.abc.Mapping by fractions of seconds).
So, to make it simple for you if you are new to Python, the same operations and methods from the Python standard dict type will all work with class AttributesHandler except the ones that try to modify the actual data.
If you want to modify the data inside AttributesHandler, you have to convert it to a dictionary first, like using the dict function, and then modify it outside.
Reference in New Issue View Git Blame Copy Permalink