EricRollei-Download_Tools/Docs/scrapling_docs.md

Key Features¶ Fetch websites as you prefer with async support¶ HTTP Requests: Fast and stealthy HTTP requests with the Fetcher class. Dynamic Loading & Automation: Fetch dynamic websites with the PlayWrightFetcher class through your real browser, Scrapling's stealth mode, Playwright's Chromium browser, or NSTbrowser's browserless! Anti-bot Protections Bypass: Easily bypass protections with the StealthyFetcher and PlayWrightFetcher classes. Easy Scraping¶ Smart Element Tracking: Relocate elements after website changes using an intelligent similarity system and integrated storage. Flexible Selection: CSS selectors, XPath selectors, filters-based search, text search, regex search, and more. Find Similar Elements: Automatically locate elements similar to the element you found! Smart Content Scraping: Extract data from multiple websites without specific selectors using Scrapling powerful features. High Performance¶ Lightning Fast: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries. Memory Efficient: Optimized data structures for minimal memory footprint. Fast JSON serialization: 10x faster than standard library. Developer Friendly¶ Powerful Navigation API: Easy DOM traversal in all directions. Rich Text Processing: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries that use less memory than standard dictionaries with added methods. Auto Selectors Generation: Generate robust short and full CSS/XPath selectors for any element. Familiar API: Similar to Scrapy/BeautifulSoup and the same CSS pseudo-elements used in Scrapy. Type hints: Complete type/doc-strings coverage for future-proofing and best autocompletion support.

Overview We will start by quickly reviewing the parsing capabilities. Then, we will fetch websites with custom browsers, make requests, and parse the response.

Here's an HTML document generated by ChatGPT we will be using as an example throughout this page:

<html> <head> <style> .hidden { display: none; } </style> </head>

• Home
• About
• Contact

Products

Product 1

This is product 1

$10.99

In stock: 5

      <article class="product" data-id="2">
        <h3>Product 2</h3>
        <p class="description">This is product 2</p>
        <span class="price">$20.99</span>
        <div class="hidden stock">In stock: 3</div>
      </article>

      <article class="product" data-id="3">
        <h3>Product 3</h3>
        <p class="description">This is product 3</p>
        <span class="price">$15.99</span>
        <div class="hidden stock">Out of stock</div>
      </article>
    </div>
  </section>

  <section id="reviews">
    <h2>Customer Reviews</h2>
    <div class="review-list">
      <div class="review" data-rating="5">
        <p class="review-text">Great product!</p>
        <span class="reviewer">John Doe</span>
      </div>
      <div class="review" data-rating="4">
        <p class="review-text">Good value for money.</p>
        <span class="reviewer">Jane Smith</span>
      </div>
    </div>
  </section>
</main>
<script id="page-data" type="application/json">
  {
    "lastUpdated": "2024-09-22T10:30:00Z",
    "totalProducts": 3
  }
</script>

</html> Starting with loading raw HTML above like this

from scrapling.parser import Adaptor page = Adaptor(html_doc) page # <data='<html><head> </head>

Product 1

This is product 1

$10.99

In stock: 5

  <article class="product" data-id="2">
    <h3>Product 2</h3>
    <p class="description">This is product 2</p>
    <span class="price">$20.99</span>
    <div class="hidden stock">In stock: 3</div>
  </article>

  <article class="product" data-id="3">
    <h3>Product 3</h3>
    <p class="description">This is product 3</p>
    <span class="price">$15.99</span>
    <div class="hidden stock">Out of stock</div>
  </article>
</div>

<script id="page-data" type="application/json">
  {
    "lastUpdated": "2024-09-22T10:30:00Z",
    "totalProducts": 3
  }
</script>

</html> Load the page directly as shown before:

from scrapling import Adaptor page = Adaptor(html_doc) Get all text content on the page recursively

page.get_all_text() 'Some page\n\n \n\n \nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock' Get the first article as explained before; we will use it as an example

article = page.find('article') With the same logic, get all text content on the element recursively

article.get_all_text() 'Product 1\nThis is product 1\n$10.99\nIn stock: 5' But if you try to get the direct text content, it will be empty; notice the logic difference

article.text '' The get_all_text method has the following optional arguments: separator: All strings collected will be concatenated using this separator. The default is '\n' strip: If enabled, strings will be stripped before concatenation. Disabled by default. ignore_tags: A tuple of all tag names you want to ignore in the final results. The default is ('script', 'style',). valid_values: If enabled, the method will only collect elements with real values, so all elements with empty text content or only whitespaces will be ignored. It's enabled by default By the way, the text returned here is not a standard string but a TextHandler; we will get to this in detail later, so if the text content can be serialized to JSON, then use .json() on it

script = page.find('script') script.json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} Let's continue to get the element tag

article.tag 'article' If you used it on the page directly, you will find you are operating on the root html element

page.tag 'html' Now, I think I hammered the (page/element) idea, so I won't return to it again. Getting the attributes of the element

print(article.attrib) {'class': 'product', 'data-id': '1'} Get the HTML content of the element

article.html_content '

Product 1

\n

This is product 1

\n $10.99\n

In stock: 5

\n

' It's the same if you used the .body property

article.body '

Product 1

\n

This is product 1

\n $10.99\n

In stock: 5

\n

' Get the prettified version of the HTML content of the element

print(article.prettify())

Product 1

This is product 1

$10.99

In stock: 5

To get all the ancestors in the DOM tree of this element

article.path [<data='

<article clas...' parent='

<artic...'>, <data='

<artic...' parent='<html><head></he...'>, <data='<html><head></he...'>] Generate a CSS shortened selector if possible, or generate the full selector

article.generate_css_selector 'body > div > article' article.generate_full_css_selector 'body > div > article' Same case with XPath

article.generate_xpath_selector "//body/div/article" article.generate_full_xpath_selector "//body/div/article" Traversal¶ Using the elements we found above, we will go over the properties/methods for moving in the page in detail.

If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online for a better understanding.

If you are too lazy to search about it, here's a quick explanation to give you a good idea. Simply put, the html element is the root of the website's tree, as every page starts with an html element. This element will be directly above elements like head and body. These are considered "children" of the html element, and the html element is considered their "parent." The element body is a "sibling" of the element head and vice versa.

Accessing the parent of an element

article.parent <data='

<article clas...' parent='

<artic...'> article.parent.tag 'div' You can chain it as you want, which applies to all similar properties/methods we will review.

article.parent.parent.tag 'body' Get the children of an element

article.children [<data='

Product 1

' parent='

<h3...'>, <data='

This is product 1...' parent='

<h3...'>, <data='$10.99' parent='

<h3...'>, <data='

In stock: 5</d...' parent='

<h3...'>] Get all elements underneath an element. It acts as a nested version of the children property

article.below_elements [<data='

Product 1

' parent='

<h3...'>, <data='

This is product 1...' parent='

<h3...'>, <data='$10.99' parent='

<h3...'>, <data='

In stock: 5</d...' parent='

<h3...'>] This element returns the same result as the children property because its children don't have children. Another example of using the element with the product-list class will clear the difference between the children property and the below_elements property

products_list = page.css_first('.product-list') products_list.children [<data='

<h3...' parent='

<article clas...'>, <data='

<h3...' parent='

<article clas...'>, <data='

<h3...' parent='

<article clas...'>]

products_list.below_elements [<data='

<h3...' parent='

<article clas...'>, <data='

Product 1

' parent='

<h3...'>, <data='

This is product 1...' parent='

<h3...'>, <data='$10.99' parent='

<h3...'>, <data='

In stock: 5</d...' parent='

<h3...'>, <data='

<h3...' parent='

<article clas...'>, ...] Get the siblings of an element

article.siblings [<data='

<h3...' parent='

<article clas...'>, <data='

<h3...' parent='

<article clas...'>] Get the next element of the current element

article.next # gets the next element, the same logic applies to quote.previous <data='

<h3...' parent='

<article clas...'> The same logic applies to the previous property

article.previous # It's the first child, so it doesn't have a previous element second_article = page.css_first('.product[data-id="2"]') second_article.previous <data='

<h3...' parent='

<article clas...'> You can check easily and pretty fast if an element has a specific class name or not

article.has_class('product') True If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element, like the example below

for ancestor in article.iterancestors(): # do something with it... You can search for a specific ancestor of an element that satisfies a function; all you need to do is to pass a function that takes an Adaptor object as an argument and return True if the condition satisfies or False otherwise like below:

article.find_ancestor(lambda ancestor: ancestor.has_class('product-list')) <data='

<article clas...' parent='

<artic...'>

article.find_ancestor(lambda ancestor: ancestor.css('.product-list')) # Same result, different approach <data='

<article clas...' parent='

<artic...'> Adaptors¶ The class Adaptors is the "List" version of the Adaptor class. It inherits from the Python standard List type, so it shares all List properties and methods while adding more methods to make the operations you want to execute on the Adaptor instances within more straightforward.

In the Adaptor class, all methods/properties that should return a group of elements return them as an Adaptors class instance. The only exceptions are when you use the CSS/XPath methods as follows:

If you selected a text node with the selector, then the return type will be TextHandler/TextHandlers. Examples:

page.css('a::text') # -> TextHandlers page.xpath('//a/text()') # -> TextHandlers page.css_first('a::text') # -> TextHandler page.xpath_first('//a/text()') # -> TextHandler page.css('a::attr(href)') # -> TextHandlers page.xpath('//a/@href') # -> TextHandlers page.css_first('a::attr(href)') # -> TextHandler page.xpath_first('//a/@href') # -> TextHandler If you used a combined selector that returns mixed types, the result will be a Python standard List. Examples:

page.css('.price_color') # -> Adaptors page.css('.product_pod a::attr(href)') # -> TextHandlers page.css('.price_color, .product_pod a::attr(href)') # -> List Let's see what Adaptors class adds to the table with that out of the way.

Properties¶ Apart from the normal operations on Python lists like iteration, slicing, etc...

You can do the following:

Execute CSS and XPath selectors directly on the Adaptor instances it has while the arguments and the return types are the same as Adaptor's css and xpath methods. This, of course, makes chaining methods very straightforward.

page.css('.product_pod a') [<data='<a href="catalogue/a-light-in-the-attic_...' parent='

<a href="c...'>, <data='<a href="catalogue/a-light-in-the-attic_...' parent='

<a href="catalogue/a-light-in-the-at...'>, <data='<a href="catalogue/tipping-the-velvet_99...' parent='

<a href="c...'>, <data='<a href="catalogue/tipping-the-velvet_99...' parent='

<a href="catalogue/tipping-the-velve...'>, <data='<a href="catalogue/soumission_998/index....' parent='

<a href="c...'>, <data='<a href="catalogue/soumission_998/index....' parent='

<a href="catalogue/soumission_998/in...'>, ...]

page.css('.product_pod').css('a') # Returns the same result [<data='<a href="catalogue/a-light-in-the-attic_...' parent='

<a href="c...'>, <data='<a href="catalogue/a-light-in-the-attic_...' parent='

<a href="catalogue/a-light-in-the-at...'>, <data='<a href="catalogue/tipping-the-velvet_99...' parent='

<a href="c...'>, <data='<a href="catalogue/tipping-the-velvet_99...' parent='

<a href="catalogue/tipping-the-velve...'>, <data='<a href="catalogue/soumission_998/index....' parent='

<a href="c...'>, <data='<a href="catalogue/soumission_998/index....' parent='

<a href="catalogue/soumission_998/in...'>, ...] Run the re and re_first methods directly. They take the same arguments passed as the Adaptor class. I'm still leaving these methods to be explained in the TextHandler section below. However, in this class, the re_first behaves differently as it runs re on each Adaptor within and returns the first one with a result. The re method will return a TextHandlers object as normal that has all the results combined in one TextHandlers instance.

page.css('.price_color').re(r'[\d.]+') ['51.77', '53.74', '50.10', '47.82', '54.23', ...]

page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html') ['a-light-in-the-attic_1000', 'tipping-the-velvet_999', 'soumission_998', 'sharp-objects_997', ...] With the search method, you can search quickly in the available Adaptor classes. The function you pass must accept an Adaptor instance as the first argument and return True/False. The method will return the first Adaptor instance that satisfies the function; otherwise, it will return None.

Find all the products with price '53.23'

search_function = lambda p: float(p.css('.price_color').re_first(r'[\d.]+')) == 54.23 page.css('.product_pod').search(search_function) <data='

<div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'> You can use the filter method, too, which takes a function like the search method but returns an Adaptors instance of all the Adaptor classes that satisfy the function

Find all products with prices over $50

filtering_function = lambda p: float(p.css('.price_color').re_first(r'[\d.]+')) > 50 page.css('.product_pod').filter(filtering_function) [<data='

<div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>, <data='

<div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>, <data='

<div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>, ...] TextHandler¶ This class is mandatory to understand, as all methods/properties that should return a string for you will return TextHandler, and the ones that should return a list of strings will return TextHandlers instead.

TextHandler is a subclass of the standard Python string, so you can do anything with it. So, what is the difference that requires a different naming?

Of course, TextHandler provides extra methods and properties that the standard Python strings can't do. We will review them now, but remember that all methods and properties in all classes that return string(s) are returning TextHandler, which opens the door for creativity and makes the code shorter and cleaner, as you will see. Also, you can import it directly and use it on any string, which we will explain later.

Usage¶ First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like split, replace, strip, etc., all return a TextHandler again, so you can chain them as you want. If you find a method or property that returns a standard string instead of TextHandler, please open an issue, and we will override it as well.

First, we start with the re and re_first methods. These are the same methods that exist in the rest of the classes (Adaptor, Adaptors, and TextHandlers), so they will take the same arguments as well.

The re method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a TextHandlers instance. The re_first method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a TextHandler instance.

Also, it takes other helpful arguments, which are:

• replace_entities: This is enabled by default. It replaces character entity references with their corresponding characters.
  • clean_match: It's disabled by default. This makes the method ignore all whitespaces and consecutive spaces while matching.
  • case_sensitive: It's enabled by default. As the name implies, disabling it will make the regex ignore letters case while compiling it.

You have seen these examples before; the return result is TextHandlers because we used the re method.

>>> page.css('.price_color').re(r'[\d\.]+')
['51.77',
 '53.74',
 '50.10',
 '47.82',
 '54.23',
...]

>>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
['a-light-in-the-attic_1000',
 'tipping-the-velvet_999',
 'soumission_998',
 'sharp-objects_997',
...]

To explain the other arguments better, we will use a custom string for each example below

>>> from scrapling import TextHandler
>>> test_string = TextHandler('hi  there')  # Hence the two spaces
>>> test_string.re('hi there')
>>> test_string.re('hi there', clean_match=True)  # Using `clean_match` will clean the string before matching the regex
['hi there']

>>> test_string2 = TextHandler('Oh, Hi Mark')
>>> test_string2.re_first('oh, hi Mark')
>>> test_string2.re_first('oh, hi Mark', case_sensitive=False)  # Hence disabling `case_sensitive`
'Oh, Hi Mark'

# Mixing arguments
>>> test_string.re('hi there', clean_match=True, case_sensitive=False)
['hi There']

Another use of the idea of replacing strings with TextHandler everywhere is a property like html_content returns TextHandler so you can do regex on the HTML content if you want:

>>> page.html_content.re('div class=".*">(.*)</div')
['In stock: 5', 'In stock: 3', 'Out of stock']

You also have the .json() method, which tries to convert the content to a json object quickly if possible; otherwise, it throws an error

page.css_first('#page-data::text') '\n {\n "lastUpdated": "2024-09-22T10:30:00Z",\n "totalProducts": 3\n }\n ' page.css_first('#page-data::text').json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} Hence, if you didn't specify a text node while selecting an element (like the text content or an attribute text content), the text content will be selected automatically like this

page.css_first('#page-data').json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} The Adaptor class adds one thing here, too; let's say this is the page we are working with:

<html>

<script id="page-data" type="application/json"> { "lastUpdated": "2024-09-22T10:30:00Z", "totalProducts": 3 } </script>

</html> The Adaptor class has the get_all_text method, which you should be aware of by now. This method returns a TextHandler, of course.

So, as you know here, if you did something like this

page.css_first('div::text').json() You will get an error because the div tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have text content at all.

In this case, the get_all_text method comes to the rescue, so you can do something like that

page.css_first('div').get_all_text(ignore_tags=[]).json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} I used the ignore_tags argument here because the default value of it is ('script', 'style',), as you are aware.

Another related behavior you should be aware of is the case while using any of the fetchers, which we will explain later. If you have a JSON response like this example:

page = Adaptor("""{"some_key": "some_value"}""") Because the Adaptor class is optimized to deal with HTML pages, it will deal with it as a broken HTML response and fix it, so if you used the html_content property, you get this

page.html_content '<html>

{"some_key": "some_value"}

</html>' Here, you can use json method directly, and it will work

page.json() {'some_key': 'some_value'} You might wonder how this happened while the html tag lacks direct text? Well, for these cases like JSON responses, I made the .json() method inside the Adaptor class to check if the current element doesn't have text content; it will use the get_all_text method directly.

It might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions. Another handy method is .clean(), this will remove all white spaces and consecutive spaces for you and return a new TextHandler, wonderful

TextHandler('\n wonderful idea, \reh?').clean() 'wonderful idea, eh?' Another method that might be helpful in some cases is the .sort() method to sort the string for you as you do with lists

TextHandler('acb').sort() 'abc' Or do it in reverse:

TextHandler('acb').sort(reverse=True) 'cba' Other methods and properties will be added over time, but remember that this class is returned in place of strings nearly everywhere in the library.

TextHandlers¶ You probably guessed it: This class is similar to Adaptors and Adaptor, but here it inherits the same logic and method as standard lists, with only re and re_first as new methods.

The only difference is that the re_first method logic here does re on each TextHandler within and returns the first result it has or None. Nothing is new to explain here, but new methods will be added here with time.

AttributesHandler¶ This is a read-only version of Python's standard dictionary or dict that's only used to store the attributes of each element or each Adaptor instance, in other words.

print(page.find('script').attrib) {'id': 'page-data', 'type': 'application/json'} type(page.find('script').attrib).name 'AttributesHandler' Because it's read-only, it will use fewer resources than the standard dictionary. Still, it has the same dictionary method/properties other than those allowing you to modify/override the data. It currently adds two extra simple methods:

The search_values method

In standard dictionaries, you can do dict.get("key_name") to check if a key exists. However, if you want to search by values instead of keys, it will take you some code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.

A simple example would be

for i in page.find('script').attrib.search_values('page-data'): print(i) {'id': 'page-data'} But this method provides the partial argument as well, which allows you to search by part of the value:

for i in page.find('script').attrib.search_values('page', partial=True): print(i) {'id': 'page-data'} These examples won't happen in the real world; most likely, a more real-world example would be using it with the find_all method to find all elements that have a specific value in their arguments:

page.find_all(lambda element: list(element.attrib.search_values('product'))) [<data='

<h3...' parent='

<article clas...'>, <data='

<h3...' parent='

<article clas...'>, <data='

<h3...' parent='

<article clas...'>] All these elements have 'product' as a value for the attribute class. Hence, I used the list function here because search_values returns a generator, so it would be True for all elements.

The json_string property

This property converts current attributes to JSON string if the attributes are JSON serializable; otherwise, it throws an error

page.find('script').attrib.json_string b'{"id":"page-data","type":"application/json"}'

Using automatch feature Introduction¶ Auto-matching is one of Scrapling's most powerful features. It allows your scraper to survive website changes by intelligently tracking and relocating elements.

Let's say you are scraping a page with a structure like this:

Product 1

Description 1

Product 2

Description 2

And you want to scrape the first product, the one with the p1 ID. You will probably write a selector like this

page.css('#p1') When website owners implement structural changes like

Product 1

Description 1

Product 2

Description 2

The selector will no longer function, and your code needs maintenance. That's where Scrapling's auto-matching feature comes into play. With Scrapling, you can enable the automatch feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element and without AI :)

from scrapling import Adaptor, Fetcher

Before the change

page = Adaptor(page_source, auto_match=True, url='example.com')

or

Fetcher.auto_match = True page = Fetcher.get('https://example.com')

then

element = page.css('#p1' auto_save=True) if not element: # One day website changes? element = page.css('#p1', auto_match=True) # Scrapling still finds it!

the rest of your code...

Below, I will show you one usage example for this feature. Then, we will dive deep into how to use it and provide details about this feature. Real-World Scenario¶ Let's use a real website as an example and use one of the fetchers to fetch its source. To do this, we need to find a website that will soon change its design/structure, take a copy of its source, and then wait for the website to make the change. Of course, that's nearly impossible to know unless I know the website's owner, but that will make it a staged test, haha.

To solve this issue, I will use The Web Archive's Wayback Machine. Here is a copy of StackOverFlow's website in 2010; pretty old, eh? Let's test if the automatch feature can extract the same button in the old design from 2010 and the current design using the same selector :)

If I want to extract the Questions button from the old design, I can use a selector like this: #hmenus > div:nth-child(1) > ul > li:nth-child(1) > a This selector is too specific because it was generated by Google Chrome.

Now, let's test the same selector in both versions

from scrapling import Fetcher selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a' old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/" new_url = "https://stackoverflow.com/" Fetcher.configure(auto_match = True, automatch_domain='stackoverflow.com')

page = Fetcher.get(old_url, timeout=30) element1 = page.css_first(selector, auto_save=True)

Same selector but used in the updated website

page = Fetcher.get(new_url) element2 = page.css_first(selector, auto_match=True)

if element1.text == element2.text: ... print('Scrapling found the same element in the old and new designs!') 'Scrapling found the same element in the old and new designs!' Note that I used a new argument called automatch_domain; this is because, for Scrapling, these are two different domains(archive.org and stackoverflow.com), so scrapling will isolate their auto_match data. To tell Scrapling they are the same website, we need to pass the custom domain we want to use while saving auto-match data for them both so Scrapling doesn't isolate them. The code will be the same in a real-world scenario, except it will use the same URL for both requests, so you won't need to use the automatch_domain argument. This is the closest example I can give to real-world cases, so I hope it didn't confuse you :)

Hence, in the two examples above, I used both the Adaptor class and the Fetcher class to show you that the logic for automatch is the same.

How the automatch feature works¶ Auto-matching works in two phases:

Save Phase: Store unique properties of elements Match Phase: Find elements with similar properties later Let's say you have an element you got through selection or any method and want the library to find it the next time you scrape this website, even if it had structural/design changes.

As little technical details as possible, the general logic goes as the following:

You tell Scrapling to save that element's unique properties in one of the ways we will show below. Scrapling uses its configured database (SQLite by default) and saves each element's unique properties. Now, because everything about the element can be changed or removed from the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:

The domain of the current website. If you are using the Adaptor class, you should pass it while initializing the class, or if you are using one of the fetchers, the domain will be taken from the URL automatically. An identifier to query that element's properties from the database. You don't always have to set the identifier yourself, as you will see later when we discuss this. Together, they will be used to retrieve the element's unique properties from the database later.

Later, when the website structural changes, you tell Scrapling to automatch the element. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated for their similarity to the element we want. In that comparison, everything is taken into consideration, as you will see later

The element(s) with the highest similarity score to the wanted element are returned. The unique properties¶ You might wonder, if all aspects of an element can be removed or changed, what unique properties we are talking about.

For Scrapling, the unique elements we are relying on are:

Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only). Element's parent tag name, attributes (names and values), and text. But you need to understand that the comparison between elements is not exact; it's more about finding how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.

How to use automatch feature¶ The automatch feature can be used on any element you have, and it's added as arguments to CSS/XPath Selection methods, as you saw above, but we will get back to that later.

First, you must enable the automatch feature by passing auto_match=True to the Adaptor class when you initialize it or enable it in the fetcher you are using of the available fetchers, as we will show.

Examples:

from scrapling import Adaptor, Fetcher page = Adaptor(html_doc, auto_match=True)

OR

Fetcher.auto_match = True page = Fetcher.fetch('https://example.com') If you are using the Adaptor class, you need to pass the url of the website you are using with the argument url so Scrapling can separate the properties saved for each element by domain. If you didn't pass a URL, the word default will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you used the same identifier later for a different website and didn't pass the URL parameter while initializing it. The save process will overwrite the previous data, and auto-matching only uses the latest saved properties.

Besides those arguments, we have storage and storage_args. Both are for the class to be used to connect to the database; by default, it's set to the SQLite class that the library is using. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a separate page in the development section.

Now, after enabling the automatch feature globally, you have two main ways to use it.

The CSS/XPath Selection way¶ As you have seen in the example above, first, you have to use the auto_save argument while selecting an element that exists on the page like below

element = page.css('#p1' auto_save=True) and when the element doesn't exist, you can use the same selector and the auto_match argument, and the library will find it for you

element = page.css('#p1', auto_match=True) Pretty simple, eh? Well, a lot happened under the hood here. Remember the identifier part we mentioned before that you need to set so you can retrieve the element you want? Here, with the css/css_first/xpath/xpath_first methods, the identifier is set automatically as the selector you passed here to make things easier :)

Also, that's why here, for all these methods, you can pass the identifier argument to set it yourself, and there are cases for this, or you can use it to save the properties with the auto_save argument.

The manual way¶ You manually save and retrieve an element, then relocate it, which all happens within the automatch feature, as shown below. This allows you to automatch any element you have by any way or any selection method!

First, let's say you got an element like this by text:

element = page.find_by_text('Tipping the Velvet', first_match=True) You can save its unique properties with the save method like below, but you must set the identifier yourself. For this example, I chose my_special_element as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)

page.save(element, 'my_special_element') Now, later, when you want to retrieve it and relocate it inside the page with auto-matching, it would be like this

element_dict = page.retrieve('my_special_element') page.relocate(element_dict, adaptor_type=True) [<data='<a href="catalogue/tipping-the-velvet_99...' parent='

<a href="catalogue/tipping-the-velve...'>] page.relocate(element_dict, adaptor_type=True).css('::text') ['Tipping the Velvet'] Hence, the retrieve and relocate` methods are used. if you want to keep it as lxml.etree object, leave the adaptor_type argument

page.relocate(element_dict) [<Element a at 0x105a2a7b0>] Troubleshooting¶ No Matches Found¶

1. Check if data was saved

element_data = page.retrieve('identifier') if not element_data: print("No data saved for this identifier")

2. Try with different identifier

products = page.css('.product', auto_match=True, identifier='old_selector')

3. Save again with new identifier

products = page.css('.new-product', auto_save=True, identifier='new_identifier') Wrong Elements Matched¶

Use more specific selectors

products = page.css('.product-list .product', auto_save=True)

Or save with more context

product = page.find_by_text('Product Name').parent page.save(product, 'specific_product') Known Issues¶ In the auto-matching save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page in different locations, auto-matching will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone.

Final thoughts¶ Explaining this feature in detail without complications turned out to be challenging, but still, if there's something left unclear, you can head out to the discussions section, and I will reply to you ASAP or reach out to me privately and have a chat :)

Choosing a fetcher

Fetchers are classes that can do requests or fetch pages for you easily in a single-line fashion with many features and then return a Response object.

This feature was introduced because the only option before v0.2 was to fetch the page as you wanted, then pass it manually to the Adaptor class and start playing with it.

Fetchers are not wrappers built on top of other libraries, but they use these libraries as an engine to make requests/fetch pages easily for you while fully utilizing that engine and adding features for you that aren't included in those engines

Fetchers Overview¶ Scrapling provides three different fetcher classes, each designed for specific use cases.

The following table compares them and can be quickly used for guidance.

Feature Fetcher PlayWrightFetcher StealthyFetcher Relative speed ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐ Stealth ⭐ ⭐⭐ ⭐⭐⭐⭐ Anti-Bot options ⭐ ⭐⭐ ⭐⭐⭐⭐ JavaScript loading ❌ ✅ ✅ Memory Usage ⭐ ⭐⭐⭐ ⭐⭐⭐ Best used for Basic scraping - Dynamically loaded websites

• Small automation
• Slight protections - Dynamically loaded websites
• Small automation
• Complicated protections Browser(s) ❌ Chromium and Google Chrome Modified Firefox Browser API used ❌ PlayWright PlayWright Setup Complexity Simple Simple Simple In the following pages, we will talk about each one in detail.

Parser configuration in all fetchers¶ All fetchers classes share the same import, as you will see in the upcoming pages

from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher Then you use it right away without initializing like this, and it will use the default parser settings:

page = StealthyFetcher.fetch('https://example.com') If you want to configure the parser (Adaptor class) that will be used on the response before returning it for you, then do this first:

from scrapling.fetchers import Fetcher Fetcher.configure(auto_match=True, encoding="utf8", keep_comments=False, keep_cdata=False) # and the rest or

from scrapling.fetchers import Fetcher Fetcher.auto_match=True Fetcher.encoding="utf8" Fetcher.keep_comments=False Fetcher.keep_cdata=False # and the rest Then, continue your code as usual. The available configuration arguments are: auto_match, huge_tree, keep_comments, keep_cdata, storage, and storage_args, which are the same ones you give to the Adaptor class. You can display the current configuration anytime by running <fetcher_class>.display_config().

Note: The auto_match argument is disabled by default; you must enable it to use that feature.

Set parser config per request¶ As you probably understood, the logic above for setting the parser config will work globally for all requests/fetches done through that class, and it's intended for simplicity.

If your use case requires a different configuration for each request/fetch, you can pass a dictionary to the request method (fetch/get/post/...) to an argument named custom_config.

Response Object¶ The Response object is the same as the Adaptor class, but it has added details about the response like response headers, status, cookies, etc... as shown below:

from scrapling.fetchers import Fetcher page = Fetcher.get('https://example.com')

page.status # HTTP status code page.reason # Status message page.cookies # Response cookies as a dictionary page.headers # Response headers page.request_headers # Request headers page.history # Response history of redirections, if any page.body # Raw response body page.encoding # Response encoding All fetchers return the Response object.

Static Requests:¶

The Fetcher class provides fast and lightweight HTTP requests with some stealth capabilities. This class uses httpx as an engine for making requests. For advanced usages, you will need some knowledge about httpx, but it becomes simpler and simpler with user feedback and updates.

Basic Usage¶ You have one primary way to import this Fetcher, which is the same for all fetchers.

from scrapling.fetchers import Fetcher Check out how to configure the parsing options here Shared arguments¶ All methods for making requests here share some arguments, so let's discuss them first.

url: The URL you want to request, of course :) proxy: As the name implies, the proxy for this request is used to route all traffic (HTTP and HTTPS). The format accepted here is http://username:password@localhost:8030. stealthy_headers: Generate and use real browser's headers, then create a referer header as if this request came from a Google search page of this URL's domain. Enabled by default, all headers generated can be overwritten by you through the headers argument. follow_redirects: As the name implies, tell the fetcher to follow redirections. Enabled by default timeout: The timeout to wait for each request to be finished in milliseconds. The default is 30000ms (30 seconds). retries: The number of retries that httpx will do for failed requests. The default number of retries is 3. Other than this, you can pass any arguments that httpx.<method_name> takes, and that's why I said, in the beginning, you need a bit of knowledge about httpx, but in the following examples, we will try to cover most cases.

HTTP Methods¶ Examples are the best way to explain this

Hence: OPTIONS and HEAD methods are not supported.

GET¶

from scrapling.fetchers import Fetcher

Basic GET

page = Fetcher.get('https://example.com') page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True) page = Fetcher.get('https://httpbin.org/get', proxy='http://username:password@localhost:8030')

With parameters

page = Fetcher.get('https://example.com/search', params={'q': 'query'})

With headers

page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})

Basic HTTP authentication

page = Fetcher.get("https://example.com", auth=("my_user", "password123")) And for asynchronous requests, it's a small adjustment

from scrapling.fetchers import AsyncFetcher

Basic GET

page = await AsyncFetcher.get('https://example.com') page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True) page = await AsyncFetcher.get('https://httpbin.org/get', proxy='http://username:password@localhost:8030')

With parameters

page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})

With headers

page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})

Basic HTTP authentication

page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123")) Needless to say, the page object in all cases is Response object, which is an Adaptor as we said, so you will use it directly

page.css('.something.something')

page = Fetcher.get('https://api.github.com/events') page.json() [{'id': '', 'type': 'PushEvent', 'actor': {'id': '', 'login': '', 'display_login': '', 'gravatar_id': '', 'url': 'https://api.github.com/users/', 'avatar_url': 'https://avatars.githubusercontent.com/u/'}, 'repo': {'id': '', ... POST¶

from scrapling.fetchers import Fetcher

Basic POST

page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}) page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True) page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')

Another example of form-encoded data

page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'})

JSON data

page = Fetcher.post('https://example.com/api', json={'key': 'value'})

Uploading file

r = Fetcher.post("https://httpbin.org/post", files={'upload-file': open('something.xlsx', 'rb')}) And for asynchronous requests, it's a small adjustment

from scrapling.fetchers import AsyncFetcher

Basic POST

page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}) page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True) page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')

Another example of form-encoded data

page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'})

JSON data

page = await AsyncFetcher.post('https://example.com/api', json={'key': 'value'})

Uploading file

r = await AsyncFetcher.post("https://httpbin.org/post", files={'upload-file': open('something.xlsx', 'rb')}) PUT¶

from scrapling.fetchers import Fetcher

Basic PUT

page = Fetcher.put('https://example.com/update', data={'status': 'updated'}) page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True) page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')

Another example of form-encoded data

page = Fetcher.put("https://httpbin.org/put", data={'key': ['value1', 'value2']}) And for asynchronous requests, it's a small adjustment

from scrapling.fetchers import AsyncFetcher

Basic PUT

page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}) page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True) page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')

Another example of form-encoded data

page = await AsyncFetcher.put("https://httpbin.org/put", data={'key': ['value1', 'value2']}) DELETE¶

from scrapling.fetchers import Fetcher page = Fetcher.delete('https://example.com/resource/123') page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True) page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030') And for asynchronous requests, it's a small adjustment

from scrapling.fetchers import AsyncFetcher page = await AsyncFetcher.delete('https://example.com/resource/123') page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True) page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030') Examples¶ Some well-rounded examples to aid newcomers to Web Scraping

Basic HTTP Request¶

from scrapling.fetchers import Fetcher

Make a request

page = Fetcher.get('https://example.com')

Check the status

if page.status == 200: # Extract title title = page.css_first('title::text') print(f"Page title: {title}")

# Extract all links
links = page.css('a::attr(href)')
print(f"Found {len(links)} links")

Product Scraping¶

from scrapling.fetchers import Fetcher

def scrape_products(): page = Fetcher.get('https://example.com/products')

# Find all product elements
products = page.css('.product')

results = []
for product in products:
    results.append({
        'title': product.css_first('.title::text'),
        'price': product.css_first('.price::text').re_first(r'\d+\.\d{2}'),
        'description': product.css_first('.description::text'),
        'in_stock': product.has_class('in-stock')
    })

return results

Pagination Handling¶

from scrapling.fetchers import Fetcher

def scrape_all_pages(): base_url = 'https://example.com/products?page={}' page_num = 1 all_products = []

while True:
    # Get current page
    page = Fetcher.get(base_url.format(page_num))

    # Find products
    products = page.css('.product')
    if not products:
        break

    # Process products
    for product in products:
        all_products.append({
            'name': product.css_first('.name::text'),
            'price': product.css_first('.price::text')
        })

    # Next page
    page_num += 1

return all_products

Form Submission¶

from scrapling.fetchers import Fetcher

Submit login form

response = Fetcher.post( 'https://example.com/login', data={ 'username': 'user@example.com', 'password': 'password123' } )

Check login success

if response.status == 200: # Extract user info user_name = response.css_first('.user-name::text') print(f"Logged in as: {user_name}") Table Extraction¶

from scrapling.fetchers import Fetcher

def extract_table(): page = Fetcher.get('https://example.com/data')

# Find table
table = page.css_first('table')

# Extract headers
headers = [
    th.text for th in table.css('thead th')
]

# Extract rows
rows = []
for row in table.css('tbody tr'):
    cells = [td.text for td in row.css('td')]
    rows.append(dict(zip(headers, cells)))

return rows

Navigation Menu¶

from scrapling.fetchers import Fetcher

def extract_menu(): page = Fetcher.get('https://example.com')

# Find navigation
nav = page.css_first('nav')

menu = {}
for item in nav.css('li'):
    link = item.css_first('a')
    if link:
        menu[link.text] = {
            'url': link.attrib['href'],
            'has_submenu': bool(item.css('.submenu'))
        }

return menu

When to Use¶ Use Fetcher when:

Need fast HTTP requests Want minimal overhead Don't need JavaScript Want simple configuration Need basic stealth features Use other fetchers when:

Need browser automation. Need advanced anti-bot/stealth. Need JavaScript support.

Dynamically Loaded Websites ¶:

Here, we will discuss the PlayWrightFetcher class. This class provides flexible browser automation with multiple configuration options and some stealth capabilities. It uses PlayWright as an engine for fetching websites.

As we will explain later, to automate the page, you need some knowledge of PlayWright's Page API.

Basic Usage¶ You have one primary way to import this Fetcher, which is the same for all fetchers.

from scrapling.fetchers import PlayWrightFetcher Check out how to configure the parsing options here Now we will go over most of the arguments one by one with examples if you want to jump to a table of all arguments for quick reference click here

Notes:

Every time you fetch a website with this fetcher, it waits by default for all JavaScript to fully load and execute, so you don't have to (waits for the domcontentloaded state). Of course, the async version of the fetch method is the async_fetch method. This fetcher currently provides 4 main run options, but they can be mixed as you want.

Which are:

1. Vanilla Playwright¶

PlayWrightFetcher.fetch('https://example.com') Using it like that will open a Chromium browser and fetch the page. There are no tricks or extra features; it's just a plain PlayWright API. 2. Stealth Mode¶

PlayWrightFetcher.fetch('https://example.com', stealth=True) It's the same as the vanilla PlayWright option, but it provides a simple stealth mode suitable for websites with a small-to-medium protection layer(s). Some of the things this fetcher's stealth mode does include:

Patching the CDP runtime fingerprint. Mimics some of the real browsers' properties by injecting several JS files and using custom options. Custom flags are used on launch to hide Playwright even more and make it faster. Generates real browser headers of the same type and user OS, then append them to the request's headers. 3. Real Chrome¶

PlayWrightFetcher.fetch('https://example.com', real_chrome=True) If you have a Google Chrome browser installed, use this option. It's the same as the first option but will use the Google Chrome browser you installed on your device instead of Chromium. This will make your requests look more like requests coming from an actual human, so it's less detectable, and you can even use the stealth=True mode with it for better results like below:

PlayWrightFetcher.fetch('https://example.com', real_chrome=True, stealth=True) If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually:

playwright install chrome 4. CDP Connection¶

PlayWrightFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222') Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the Chrome DevTools Protocol. This fetcher takes it even a step further. You can use NSTBrowser's docker browserless option by passing the CDP URL and enabling nstbrowser_mode option like below

PlayWrightFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222', nstbrowser_mode=True) There's also a nstbrowser_config argument to send the config you want to send with the requests to the NSTBrowser. If you leave it empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config. Full list of arguments¶ Scrapling provides many options with this fetcher, which works in all modes except the NSTBrowser mode. To make it as simple as possible, we will list the options here and give examples of using most of them.

Argument Description Optional url Target url ❌ headless Pass True to run the browser in headless/hidden (default) or False for headful/visible mode. ✔️ disable_resources Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites. Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading. ✔️ useragent Pass a useragent string to be used. Otherwise, the fetcher will generate and use a real Useragent of the same browser. ✔️ network_idle Wait for the page until there are no network connections for at least 500 ms. ✔️ timeout The timeout (milliseconds) used in all operations and waits through the page. The default is 30000. ✔️ wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object. ✔️ page_action Added for automation. Pass a function that takes the page object and does the necessary automation, then returns page again. ✔️ wait_selector Wait for a specific css selector to be in a specific state. ✔️ wait_selector_state Scrapling will wait for the given state to be fulfilled for the selector given with wait_selector. Default state is attached. ✔️ google_search Enabled by default, Scrapling will set the referer header as if this request came from a Google search for this website's domain name. ✔️ extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together. ✔️ proxy The proxy to be used with requests. It can be a string or a dictionary with the keys 'server', 'username', and 'password' only. ✔️ hide_canvas Add random noise to canvas operations to prevent fingerprinting. ✔️ disable_webgl Disables WebGL and WebGL 2.0 support entirely. ✔️ stealth Enables stealth mode; you should always check the documentation to see what stealth mode does currently. ✔️ real_chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser and use it. ✔️ locale Set the locale for the browser if wanted. The default value is en-US. ✔️ cdp_url Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP. ✔️ nstbrowser_mode Enables NSTBrowser mode, it have to be used with cdp_url argument or it will get completely ignored. ✔️ nstbrowser_config The config you want to send with requests to the NSTBrowser. Scrapling defaults to an optimized NSTBrowser's docker browserless config if you leave this argument empty. ✔️ Examples¶ It's easier to understand with examples, so let's look at it.

Resource Control¶

Disable unnecessary resources

page = PlayWrightFetcher.fetch( 'https://example.com', disable_resources=True # Blocks fonts, images, media, etc... ) Network Control¶

Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)

page = PlayWrightFetcher.fetch('https://example.com', network_idle=True)

Custom timeout (in milliseconds)

page = PlayWrightFetcher.fetch('https://example.com', timeout=30000) # 30 seconds

Proxy support

page = PlayWrightFetcher.fetch( 'https://example.com', proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only ) Browser Automation¶ This is where your knowledge about PlayWright's Page API comes into play. The function you pass here takes the page object from Playwright's API, does what you want, and then returns it again for the current fetcher to continue working on it.

This function is executed right after waiting for network_idle (if enabled) and before waiting for the wait_selector argument, so it can be used for many things, not just automation. You can alter the page as you want.

In the example below, I used page mouse events to move the mouse wheel to scroll the page and then move the mouse.

from playwright.sync_api import Page

def scroll_page(page: Page): page.mouse.wheel(10, 0) page.mouse.move(100, 400) page.mouse.up() return page

page = PlayWrightFetcher.fetch( 'https://example.com', page_action=scroll_page ) Of course, if you use the async fetch version, the function must also be async.

from playwright.async_api import Page

async def scroll_page(page: Page): await page.mouse.wheel(10, 0) await page.mouse.move(100, 400) await page.mouse.up() return page

page = await PlayWrightFetcher.async_fetch( 'https://example.com', page_action=scroll_page ) Wait Conditions¶

Wait for the selector

page = PlayWrightFetcher.fetch( 'https://example.com', wait_selector='h1', wait_selector_state='visible' ) This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the wait_selector argument, and the fetcher will wait for the state you passed in the wait_selector_state argument to be fulfilled. If you didn't pass a state, the default would be attached, which means it will wait for the element to be present in the DOM. After that, the fetcher will check again to see if all JS files are loaded and executed (the domcontentloaded state) and wait for them to be. If you have enabled network_idle with this, the fetcher will wait for network_idle to be fulfilled again, as explained above.

The states the fetcher can wait for can be either (source):

attached: Wait for an element to be present in DOM. detached: Wait for an element to not be present in DOM. visible: wait for an element to have a non-empty bounding box and no visibility:hidden. Note that an element without any content or with display:none has an empty bounding box and is not considered visible. hidden: wait for an element to be either detached from DOM, or have an empty bounding box or visibility:hidden. This is opposite to the 'visible' option. Some Stealth Features¶

Full stealth mode

page = PlayWrightFetcher.fetch( 'https://example.com', stealth=True, hide_canvas=True, disable_webgl=True, google_search=True )

Custom user agent

page = PlayWrightFetcher.fetch( 'https://example.com', useragent='Mozilla/5.0...' )

Set browser locale

page = PlayWrightFetcher.fetch( 'https://example.com', locale='en-US' ) Hence, the hide_canvas argument doesn't disable canvas but hides it by adding random noise to canvas operations to prevent fingerprinting. Also, if you didn't set a useragent (preferred), the fetcher will generate a real Useragent of the same browser and use it. The google_search argument is enabled by default, making the request look like it came from Google. So, a request for https://example.com will set the referer to https://www.google.com/search?q=example. Also, if used together, it takes priority over the referer set by the extra_headers argument.

General example¶

from scrapling.fetchers import PlayWrightFetcher

def scrape_dynamic_content(): # Use PlayWright for JavaScript content page = PlayWrightFetcher.fetch( 'https://example.com/dynamic', network_idle=True, wait_selector='.content' )

# Extract dynamic content
content = page.css('.content')

return {
    'title': content.css_first('h1::text'),
    'items': [
        item.text for item in content.css('.item')
    ]
}

When to Use¶ Use PlayWrightFetcher when:

Need browser automation Want multiple browser options Using a real Chrome browser Need custom browser config Want flexible stealth options

Full Bypass protections while fetching:

Here, we will discuss the StealthyFetcher class. This class is similar to PlayWrightFetcher in many ways, like browser automation and using PlayWright as an engine for fetching websites. The main difference is that this class provides advanced anti-bot protection bypass capabilities and a modified Firefox browser called Camoufox, from which most stealth comes.

As with PlayWrightFetcher, you will need some knowledge about PlayWright's Page API to automate the page, as we will explain later.

Basic Usage¶ You have one primary way to import this Fetcher, which is the same for all fetchers.

from scrapling.fetchers import StealthyFetcher Check out how to configure the parsing options here Notes:

Every time you fetch a website with this fetcher, it waits by default for all JavaScript to fully load and execute, so you don't have to (waits for the domcontentloaded state). Of course, the async version of the fetch method is the async_fetch method. Full list of arguments¶ Before jumping to examples, here's the full list of arguments

Argument Description Optional url Target url ❌ headless Pass True to run the browser in headless/hidden (default), virtual to run it in virtual screen mode, or False for headful/visible mode. The virtual mode requires having xvfb installed. ✔️ block_images Prevent the loading of images through Firefox preferences. This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading. ✔️ disable_resources Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites. Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading. ✔️ google_search Enabled by default, Scrapling will set the referer header as if this request came from a Google search for this website's domain name. ✔️ extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together. ✔️ block_webrtc Blocks WebRTC entirely. ✔️ page_action Added for automation. A function that takes the page object and does the automation you need, then returns page again. ✔️ addons List of Firefox addons to use. Must be paths to extracted addons. ✔️ humanize Humanize the cursor movement. The cursor movement takes either True or the MAX duration in seconds. The cursor typically takes up to 1.5 seconds to move across the window. ✔️ allow_webgl Enabled by default. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. ✔️ geoip Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. ✔️ os_randomize If enabled, Scrapling will randomize the OS fingerprints used. The default is matching the fingerprints with the current OS. ✔️ disable_ads Disabled by default; this installs the uBlock Origin addon on the browser if enabled. ✔️ network_idle Wait for the page until there are no network connections for at least 500 ms. ✔️ timeout The timeout used in all operations and waits through the page. It's in milliseconds, and the default is 30000. ✔️ wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object. ✔️ wait_selector Wait for a specific css selector to be in a specific state. ✔️ wait_selector_state Scrapling will wait for the given state to be fulfilled for the selector given with wait_selector. Default state is attached. ✔️ proxy The proxy to be used with requests. It can be a string or a dictionary with the keys 'server', 'username', and 'password' only. ✔️ additional_arguments Arguments passed to Camoufox as additional settings that take higher priority than Scrapling's. ✔️ Examples¶ It's easier to understand with examples, so now we will go over most of the arguments individually with examples.

Browser Modes¶

Headless/hidden mode (default)

page = StealthyFetcher.fetch('https://example.com', headless=True)

Virtual display mode (requires having xvfb installed)

page = StealthyFetcher.fetch('https://example.com', headless='virtual')

Visible browser mode

page = StealthyFetcher.fetch('https://example.com', headless=False) Resource Control¶

Block images

page = StealthyFetcher.fetch('https://example.com', block_images=True)

Disable unnecessary resources

page = StealthyFetcher.fetch('https://example.com', disable_resources=True) # Blocks fonts, images, media, etc. Additional stealth options¶

page = StealthyFetcher.fetch( 'https://example.com', block_webrtc=True, # Block WebRTC allow_webgl=False, # Disable WebGL humanize=True, # Make the mouse move as how a human would move it geoip=True, # Use IP's longitude, latitude, timezone, country, and locale, then spoof the WebRTC IP address... os_randomize=True, # Randomize the OS fingerprints used. The default is matching the fingerprints with the current OS. disable_ads=True, # Block ads with uBlock Origin addon (enabled by default) google_search=True )

Custom user agent

page = StealthyFetcher.fetch( 'https://example.com', useragent='Mozilla/5.0...' )

Custom humanization duration

page = StealthyFetcher.fetch( 'https://example.com', humanize=1.5 # Max 1.5 seconds for cursor movement ) The google_search argument is enabled by default. It makes the request as if it came from Google, so for a request for https://example.com, it will set the referer to https://www.google.com/search?q=example. Also, if used together, it takes priority over the referer set by the extra_headers argument.

Network Control¶

Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)

page = StealthyFetcher.fetch('https://example.com', network_idle=True)

Custom timeout (in milliseconds)

page = StealthyFetcher.fetch('https://example.com', timeout=30000) # 30 seconds

Proxy support

page = StealthyFetcher.fetch( 'https://example.com', proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only ) Browser Automation¶ This is where your knowledge about PlayWright's Page API comes into play. The function you pass here takes the page object from Playwright's API, does what you want, and then returns it again for the current fetcher to continue working on it.

This function is executed right after waiting for network_idle (if enabled) and before waiting for the wait_selector argument, so it can be used for many things, not just automation. You can alter the page as you want.

In the example below, I used page mouse events to move the mouse wheel to scroll the page and then move the mouse.

from playwright.sync_api import Page

def scroll_page(page: Page): page.mouse.wheel(10, 0) page.mouse.move(100, 400) page.mouse.up() return page

page = StealthyFetcher.fetch( 'https://example.com', page_action=scroll_page ) Of course, if you use the async fetch version, the function must also be async.

from playwright.async_api import Page

async def scroll_page(page: Page): await page.mouse.wheel(10, 0) await page.mouse.move(100, 400) await page.mouse.up() return page

page = await StealthyFetcher.async_fetch( 'https://example.com', page_action=scroll_page ) Wait Conditions¶

Wait for the selector

page = StealthyFetcher.fetch( 'https://example.com', wait_selector='h1', wait_selector_state='visible' ) This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the wait_selector argument, and the fetcher will wait for the state you passed in the wait_selector_state argument to be fulfilled. If you didn't pass a state, the default would be attached, which means it will wait for the element to be present in the DOM. After that, the fetcher will check again to see if all JS files are loaded and executed (the domcontentloaded state) and wait for them to be. If you have enabled network_idle with this, the fetcher will wait for network_idle to be fulfilled again, as explained above.

The states the fetcher can wait for can be either (source):

attached: wait for the element to be present in DOM. detached: wait for the element to not be present in DOM. visible: wait for the element to have a non-empty bounding box and no visibility:hidden. Note that an element without any content or with display:none has an empty bounding box and is not considered visible. hidden: Wait for the element to be detached from DOM, have an empty bounding box, or have visibility:hidden. This is opposite to the 'visible' option. Firefox Addons¶

Custom Firefox addons

page = StealthyFetcher.fetch( 'https://example.com', addons=['/path/to/addon1', '/path/to/addon2'] ) The paths here must be paths of extracted addons, which will be installed automatically upon browser launch. Real-world example (Amazon)¶ This is for educational purposes only; this example was generated by AI, which shows too how easy it is to work with Scrapling through AI

def scrape_amazon_product(url): # Use StealthyFetcher to bypass protection page = StealthyFetcher.fetch(url)

# Extract product details
return {
    'title': page.css_first('#productTitle::text').clean(),
    'price': page.css_first('.a-price .a-offscreen::text'),
    'rating': page.css_first('[data-feature-name="averageCustomerReviews"] .a-popover-trigger .a-color-base::text'),
    'reviews_count': page.css('#acrCustomerReviewText::text').re_first(r'[\d,]+'),
    'features': [
        li.clean() for li in page.css('#feature-bullets li span::text')
    ],
    'availability': page.css_first('#availability').get_all_text(strip=True),
    'images': [
        img.attrib['src'] for img in page.css('#altImages img')
    ]
}

When to Use¶ Use StealthyFetcher when:

Bypassing anti-bot protection Need a reliable browser fingerprint Full JavaScript support needed Want automatic stealth features Need browser automation

API Reference:

Adaptor Class¶ The Adaptor class is the core parsing engine in Scrapling that provides HTML parsing and element selection capabilities.

Here's the reference information for the Adaptor class, with all its parameters, attributes, and methods.

You can import the Adaptor class directly from scrapling:

from scrapling.parser import Adaptor scrapling.parser.Adaptor ¶

Adaptor( text=None, url=None, body=b"", encoding="utf8", huge_tree=True, root=None, keep_comments=False, keep_cdata=False, auto_match=False, storage=SQLiteStorageSystem, storage_args=None, **kwargs ) Bases: SelectorsGeneration

The main class that works as a wrapper for the HTML input data. Using this class, you can search for elements with expressions in CSS, XPath, or with simply text. Check the docs for more info.

Here we try to extend module lxml.html.HtmlElement while maintaining a simpler interface, We are not inheriting from the lxml.html.HtmlElement because it's not pickleable which makes a lot of reference jobs not possible. You can test it here and see code explodes with AssertionError: invalid Element proxy at.... It's an old issue with lxml, see this entry https://bugs.launchpad.net/lxml/+bug/736708

PARAMETER DESCRIPTION text HTML body passed as text. TYPE: Optional[str]DEFAULT: None

url allows storing a URL with the html data for retrieving later. TYPE: Optional[str]DEFAULT: None

body HTML body as bytes object. It can be used instead of the text argument. TYPE: bytesDEFAULT: b''

encoding The encoding type that will be used in HTML parsing, default is UTF-8 TYPE: strDEFAULT: 'utf8'

huge_tree Enabled by default, should always be enabled when parsing large HTML documents. This controls libxml2 feature that forbids parsing certain large documents to protect from possible memory exhaustion. TYPE: boolDEFAULT: True

root Used internally to pass etree objects instead of text/body arguments, it takes highest priority. Don't use it unless you know what you are doing! TYPE: Optional[HtmlElement]DEFAULT: None

keep_comments While parsing the HTML body, drop comments or not. Disabled by default for obvious reasons TYPE: Optional[bool]DEFAULT: False

keep_cdata While parsing the HTML body, drop cdata or not. Disabled by default for cleaner HTML. TYPE: Optional[bool]DEFAULT: False

auto_match Globally turn-off the auto-match feature in all functions, this argument takes higher priority over all auto-match related arguments/functions in the class. TYPE: Optional[bool]DEFAULT: False

storage The storage class to be passed for auto-matching functionalities, see Docs for more info. TYPE: AnyDEFAULT: SQLiteStorageSystem

storage_args A dictionary of argument->value pairs to be passed for the storage class. If empty, default values will be used. TYPE: Optional[Dict]DEFAULT: None

Source code in scrapling/parser.py encoding instance-attribute ¶

encoding = encoding url instance-attribute ¶

url = url tag property ¶

tag Get tag name of the element

text property ¶

text Get text content of the element

attrib property ¶

attrib Get attributes of the element

html_content property ¶

html_content Return the inner html code of the element

body class-attribute instance-attribute ¶

body = html_content parent property ¶

parent Return the direct parent of the element or None otherwise

below_elements property ¶

below_elements Return all elements under the current element in the DOM tree

children property ¶

children Return the children elements of the current element or empty list otherwise

siblings property ¶

siblings Return other children of the current element's parent or empty list otherwise

path property ¶

path Returns list of type :class:Adaptors that contains the path leading to the current element from the root.

next property ¶

next Returns the next element of the current element in the children of the parent or None otherwise.

previous property ¶

previous Returns the previous element of the current element in the children of the parent or None otherwise.

extract class-attribute instance-attribute ¶

extract = get_all extract_first class-attribute instance-attribute ¶

extract_first = get generate_css_selector property ¶

generate_css_selector Generate a CSS selector for the current element

RETURNS DESCRIPTION str A string of the generated selector. generate_full_css_selector property ¶

generate_full_css_selector Generate a complete CSS selector for the current element

RETURNS DESCRIPTION str A string of the generated selector. generate_xpath_selector property ¶

generate_xpath_selector Generate a XPath selector for the current element

RETURNS DESCRIPTION str A string of the generated selector. generate_full_xpath_selector property ¶

generate_full_xpath_selector Generate a complete XPath selector for the current element

RETURNS DESCRIPTION str A string of the generated selector. get_all_text ¶

get_all_text( separator="\n", strip=False, ignore_tags=("script", "style"), valid_values=True, ) Get all child strings of this element, concatenated using the given separator.

PARAMETER DESCRIPTION separator Strings will be concatenated using this separator. TYPE: strDEFAULT: '\n'

strip If True, strings will be stripped before being concatenated. TYPE: boolDEFAULT: False

ignore_tags A tuple of all tag names you want to ignore TYPE: TupleDEFAULT: ('script', 'style')

valid_values If enabled, elements with text-content that is empty or only whitespaces will be ignored TYPE: boolDEFAULT: True

RETURNS DESCRIPTION TextHandler A TextHandler Source code in scrapling/parser.py urljoin ¶

urljoin(relative_url) Join this Adaptor's url with a relative url to form an absolute full URL.

Source code in scrapling/parser.py prettify ¶

prettify() Return a prettified version of the element's inner html-code

Source code in scrapling/parser.py has_class ¶

has_class(class_name) Check if element has a specific class

PARAMETER DESCRIPTION class_name The class name to check for TYPE: str

RETURNS DESCRIPTION bool True if element has class with that name otherwise False Source code in scrapling/parser.py iterancestors ¶

iterancestors() Return a generator that loops over all ancestors of the element, starting with element's parent.

Source code in scrapling/parser.py find_ancestor ¶

find_ancestor(func) Loop over all ancestors of the element till one match the passed function

PARAMETER DESCRIPTION func A function that takes each ancestor as an argument and returns True/False TYPE: Callable[[Adaptor], bool]

RETURNS DESCRIPTION Union[Adaptor, None] The first ancestor that match the function or None otherwise. Source code in scrapling/parser.py get ¶

get(default=None) Source code in scrapling/parser.py get_all ¶

get_all() Source code in scrapling/parser.py relocate ¶

relocate(element, percentage=0, adaptor_type=False) This function will search again for the element in the page tree, used automatically on page structure change

PARAMETER DESCRIPTION element The element we want to relocate in the tree TYPE: Union[Dict, HtmlElement, Adaptor]

percentage The minimum percentage to accept and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing! TYPE: intDEFAULT: 0

adaptor_type If True, the return result will be converted to Adaptors object TYPE: boolDEFAULT: False

RETURNS DESCRIPTION Union[List[Union[HtmlElement, None]], Adaptors] List of pure HTML elements that got the highest matching score or 'Adaptors' object Source code in scrapling/parser.py css_first ¶

css_first( selector, identifier="", auto_match=False, auto_save=False, percentage=0, ) Search current tree with CSS3 selectors and return the first result if possible, otherwise return None

Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)

PARAMETER DESCRIPTION selector The CSS3 selector to be used. TYPE: str

auto_match Enabled will make function try to relocate the element if it was 'saved' before TYPE: boolDEFAULT: False

identifier A string that will be used to save/retrieve element's data in auto-matching otherwise the selector will be used. TYPE: strDEFAULT: ''

auto_save Automatically save new elements for auto_match later TYPE: boolDEFAULT: False

percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing! TYPE: intDEFAULT: 0

Source code in scrapling/parser.py xpath_first ¶

xpath_first( selector, identifier="", auto_match=False, auto_save=False, percentage=0, **kwargs ) Search current tree with XPath selectors and return the first result if possible, otherwise return None

Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)

Note: Additional keyword arguments will be passed as XPath variables in the XPath expression!

PARAMETER DESCRIPTION selector The XPath selector to be used. TYPE: str

auto_match Enabled will make function try to relocate the element if it was 'saved' before TYPE: boolDEFAULT: False

identifier A string that will be used to save/retrieve element's data in auto-matching otherwise the selector will be used. TYPE: strDEFAULT: ''

auto_save Automatically save new elements for auto_match later TYPE: boolDEFAULT: False

percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing! TYPE: intDEFAULT: 0

Source code in scrapling/parser.py css ¶

css( selector, identifier="", auto_match=False, auto_save=False, percentage=0, ) Search current tree with CSS3 selectors

Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)

PARAMETER DESCRIPTION selector The CSS3 selector to be used. TYPE: str

auto_match Enabled will make function try to relocate the element if it was 'saved' before TYPE: boolDEFAULT: False

identifier A string that will be used to save/retrieve element's data in auto-matching otherwise the selector will be used. TYPE: strDEFAULT: ''

auto_save Automatically save new elements for auto_match later TYPE: boolDEFAULT: False

percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing! TYPE: intDEFAULT: 0

RETURNS DESCRIPTION Union[Adaptors[Adaptor], List, TextHandlers[TextHandler]] List as :class:Adaptors Source code in scrapling/parser.py xpath ¶

xpath( selector, identifier="", auto_match=False, auto_save=False, percentage=0, **kwargs ) Search current tree with XPath selectors

Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)

Note: Additional keyword arguments will be passed as XPath variables in the XPath expression!

PARAMETER DESCRIPTION selector The XPath selector to be used. TYPE: str

auto_match Enabled will make function try to relocate the element if it was 'saved' before TYPE: boolDEFAULT: False

identifier A string that will be used to save/retrieve element's data in auto-matching otherwise the selector will be used. TYPE: strDEFAULT: ''

auto_save Automatically save new elements for auto_match later TYPE: boolDEFAULT: False

percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing! TYPE: intDEFAULT: 0

RETURNS DESCRIPTION Union[Adaptors[Adaptor], List, TextHandlers[TextHandler]] List as :class:Adaptors Source code in scrapling/parser.py find_all ¶

find_all(*args, **kwargs) Find elements by filters of your creations for ease..

PARAMETER DESCRIPTION args Tag name(s), an iterable of tag names, regex patterns, function, or a dictionary of elements' attributes. Leave empty for selecting all. TYPE: Union[str, Iterable[str], Pattern, Callable, Dict[str, str]]DEFAULT: ()

kwargs The attributes you want to filter elements based on it. TYPE: strDEFAULT: {}

RETURNS DESCRIPTION Adaptors The Adaptors object of the elements or empty list Source code in scrapling/parser.py find ¶

find(*args, **kwargs) Find elements by filters of your creations for ease then return the first result. Otherwise return None.

PARAMETER DESCRIPTION args Tag name(s), an iterable of tag names, regex patterns, function, or a dictionary of elements' attributes. Leave empty for selecting all. TYPE: Union[str, Iterable[str], Pattern, Callable, Dict[str, str]]DEFAULT: ()

kwargs The attributes you want to filter elements based on it. TYPE: strDEFAULT: {}

RETURNS DESCRIPTION Union[Adaptor, None] The Adaptor object of the element or None if the result didn't match Source code in scrapling/parser.py save ¶

save(element, identifier) Saves the element's unique properties to the storage for retrieval and relocation later

PARAMETER DESCRIPTION element The element itself that we want to save to storage, it can be a Adaptor or pure HtmlElement TYPE: Union[Adaptor, HtmlElement]

identifier This is the identifier that will be used to retrieve the element later from the storage. See the docs for more info. TYPE: str

Source code in scrapling/parser.py retrieve ¶

retrieve(identifier) Using the identifier, we search the storage and return the unique properties of the element

PARAMETER DESCRIPTION identifier This is the identifier that will be used to retrieve the element from the storage. See the docs for more info. TYPE: str

RETURNS DESCRIPTION Optional[Dict] A dictionary of the unique properties Source code in scrapling/parser.py json ¶

json() Return json response if the response is jsonable otherwise throws error

Source code in scrapling/parser.py re ¶

re( regex, replace_entities=True, clean_match=False, case_sensitive=True, ) Apply the given regex to the current text and return a list of strings with the matches.

PARAMETER DESCRIPTION regex Can be either a compiled regular expression or a string. TYPE: Union[str, Pattern[str]]

replace_entities if enabled character entity references are replaced by their corresponding character TYPE: boolDEFAULT: True

clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching TYPE: boolDEFAULT: False

case_sensitive if disabled, function will set the regex to ignore letters case while compiling it TYPE: boolDEFAULT: True

Source code in scrapling/parser.py re_first ¶

re_first( regex, default=None, replace_entities=True, clean_match=False, case_sensitive=True, ) Apply the given regex to text and return the first match if found, otherwise return the default value.

PARAMETER DESCRIPTION regex Can be either a compiled regular expression or a string. TYPE: Union[str, Pattern[str]]

default The default value to be returned if there is no match DEFAULT: None

replace_entities if enabled character entity references are replaced by their corresponding character TYPE: boolDEFAULT: True

clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching TYPE: boolDEFAULT: False

case_sensitive if disabled, function will set the regex to ignore letters case while compiling it TYPE: boolDEFAULT: True

Source code in scrapling/parser.py find_similar ¶

find_similar( similarity_threshold=0.2, ignore_attributes=("href", "src"), match_text=False, ) Find elements that are in the same tree depth in the page with the same tag name and same parent tag etc... then return the ones that match the current element attributes with percentage higher than the input threshold.

This function is inspired by AutoScraper and made for cases where you, for example, found a product div inside a products-list container and want to find other products using that that element as a starting point EXCEPT this function works in any case without depending on the element type.

PARAMETER DESCRIPTION similarity_threshold The percentage to use while comparing elements attributes. Note: Elements found before attributes matching/comparison will be sharing the same depth, same tag name, same parent tag name, and same grand parent tag name. So they are 99% likely to be correct unless your are extremely unlucky then attributes matching comes into play so basically don't play with this number unless you are getting the results you don't want. Also, if current element doesn't have attributes and the similar element as well, then it's a 100% match. TYPE: floatDEFAULT: 0.2

ignore_attributes Attribute names passed will be ignored while matching the attributes in last step. The default value is to ignore href and src as URLs can change a lot between elements so it's unreliable TYPE: Union[List, Tuple]DEFAULT: ('href', 'src')

match_text If True, elements text content will be taken into calculation while matching. Not recommended to use in normal cases but it depends. TYPE: boolDEFAULT: False

RETURNS DESCRIPTION Union[Adaptors[Adaptor], List] A Adaptors container of Adaptor objects or empty list Source code in scrapling/parser.py find_by_text ¶

find_by_text( text, first_match=True, partial=False, case_sensitive=False, clean_match=True, ) Find elements that its text content fully/partially matches input.

PARAMETER DESCRIPTION text Text query to match TYPE: str

first_match Return first element that matches conditions, enabled by default TYPE: boolDEFAULT: True

partial If enabled, function return elements that contains the input text TYPE: boolDEFAULT: False

case_sensitive if enabled, letters case will be taken into consideration TYPE: boolDEFAULT: False

clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching TYPE: boolDEFAULT: True

Source code in scrapling/parser.py find_by_regex ¶

find_by_regex( query, first_match=True, case_sensitive=False, clean_match=True, ) Find elements that its text content matches the input regex pattern.

PARAMETER DESCRIPTION query Regex query/pattern to match TYPE: Union[str, Pattern[str]]

first_match Return first element that matches conditions, enabled by default TYPE: boolDEFAULT: True

case_sensitive if enabled, letters case will be taken into consideration in the regex TYPE: boolDEFAULT: False

clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching TYPE: boolDEFAULT: True

Source code in scrapling/parser.py scrapling.parser.Adaptors ¶ Bases: List[Adaptor]

The :class:Adaptors class is a subclass of the builtin List class, which provides a few additional methods.

extract_first class-attribute instance-attribute ¶

extract_first = get get_all class-attribute instance-attribute ¶

get_all = extract first property ¶

first Returns the first item of the current list or None if the list is empty

last property ¶

last Returns the last item of the current list or None if the list is empty

xpath ¶

xpath( selector, identifier="", auto_save=False, percentage=0, **kwargs ) Call the .xpath() method for each element in this list and return their results as another :class:Adaptors.

Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)

Note: Additional keyword arguments will be passed as XPath variables in the XPath expression!

PARAMETER DESCRIPTION selector The XPath selector to be used. TYPE: str

identifier A string that will be used to retrieve element's data in auto-matching otherwise the selector will be used. TYPE: strDEFAULT: ''

auto_save Automatically save new elements for auto_match later TYPE: boolDEFAULT: False

percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing! TYPE: intDEFAULT: 0

RETURNS DESCRIPTION Adaptors[Adaptor] List as :class:Adaptors Source code in scrapling/parser.py css ¶

css(selector, identifier='', auto_save=False, percentage=0) Call the .css() method for each element in this list and return their results flattened as another :class:Adaptors.

Important: It's recommended to use the identifier argument if you plan to use different selector later and want to relocate the same element(s)

PARAMETER DESCRIPTION selector The CSS3 selector to be used. TYPE: str

identifier A string that will be used to retrieve element's data in auto-matching otherwise the selector will be used. TYPE: strDEFAULT: ''

auto_save Automatically save new elements for auto_match later TYPE: boolDEFAULT: False

percentage The minimum percentage to accept while auto-matching and not going lower than that. Be aware that the percentage calculation depends solely on the page structure so don't play with this number unless you must know what you are doing! TYPE: intDEFAULT: 0

RETURNS DESCRIPTION Adaptors[Adaptor] List as :class:Adaptors Source code in scrapling/parser.py re ¶

re( regex, replace_entities=True, clean_match=False, case_sensitive=True, ) Call the .re() method for each element in this list and return their results flattened as List of TextHandler.

PARAMETER DESCRIPTION regex Can be either a compiled regular expression or a string. TYPE: Union[str, Pattern[str]]

replace_entities if enabled character entity references are replaced by their corresponding character TYPE: boolDEFAULT: True

clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching TYPE: boolDEFAULT: False

case_sensitive if disabled, function will set the regex to ignore letters case while compiling it TYPE: boolDEFAULT: True

Source code in scrapling/parser.py re_first ¶

re_first( regex, default=None, replace_entities=True, clean_match=False, case_sensitive=True, ) Call the .re_first() method for each element in this list and return the first result or the default value otherwise.

PARAMETER DESCRIPTION regex Can be either a compiled regular expression or a string. TYPE: Union[str, Pattern[str]]

default The default value to be returned if there is no match DEFAULT: None

replace_entities if enabled character entity references are replaced by their corresponding character TYPE: boolDEFAULT: True

clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching TYPE: boolDEFAULT: False

case_sensitive if disabled, function will set the regex to ignore letters case while compiling it TYPE: boolDEFAULT: True

Source code in scrapling/parser.py search ¶

search(func) Loop over all current elements and return the first element that matches the passed function

PARAMETER DESCRIPTION func A function that takes each element as an argument and returns True/False TYPE: Callable[[Adaptor], bool]

RETURNS DESCRIPTION Union[Adaptor, None] The first element that match the function or None otherwise. Source code in scrapling/parser.py filter ¶

filter(func) Filter current elements based on the passed function

PARAMETER DESCRIPTION func A function that takes each element as an argument and returns True/False TYPE: Callable[[Adaptor], bool]

RETURNS DESCRIPTION Adaptors[Adaptor] The new Adaptors object or empty list otherwise. Source code in scrapling/parser.py get ¶

get(default=None) Returns the first item of the current list

PARAMETER DESCRIPTION default the default value to return if the current list is empty DEFAULT: None

Source code in scrapling/parser.py extract ¶

extract() Source code in scrapling/parser.py

Fetchers Classes¶ Here's the reference information for all fetcher-type classes' parameters, attributes, and methods.

You can import all of them directly like below:

from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher scrapling.fetchers.Fetcher ¶

Fetcher(*args, **kwargs) Bases: BaseFetcher

A basic Fetcher class type that can only do basic GET, POST, PUT, and DELETE HTTP requests based on httpx.

Any additional keyword arguments passed to the methods below are passed to the respective httpx's method directly.

Source code in scrapling/engines/toolbelt/custom.py huge_tree class-attribute instance-attribute ¶

huge_tree = True auto_match class-attribute instance-attribute ¶

auto_match = False storage class-attribute instance-attribute ¶

storage = SQLiteStorageSystem keep_cdata class-attribute instance-attribute ¶

keep_cdata = False storage_args class-attribute instance-attribute ¶

storage_args = None keep_comments class-attribute instance-attribute ¶

keep_comments = False automatch_domain class-attribute instance-attribute ¶

automatch_domain = None parser_keywords class-attribute instance-attribute ¶

parser_keywords = ( "huge_tree", "auto_match", "storage", "keep_cdata", "storage_args", "keep_comments", "automatch_domain", ) get classmethod ¶

get( url, follow_redirects=True, timeout=10, stealthy_headers=True, proxy=None, retries=3, custom_config=None, **kwargs ) Make basic HTTP GET request for you but with some added flavors.

PARAMETER DESCRIPTION url Target url. TYPE: str

follow_redirects As the name says -- if enabled (default), redirects will be followed. TYPE: boolDEFAULT: True

timeout The time to wait for the request to finish in seconds. The default is 10 seconds. TYPE: Optional[Union[int, float]]DEFAULT: 10

stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request had came from Google's search of this URL's domain. TYPE: boolDEFAULT: True

proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030 TYPE: Optional[str]DEFAULT: None

retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries. TYPE: Optional[int]DEFAULT: 3

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

kwargs Any additional keyword arguments are passed directly to httpx.get() function so check httpx documentation for details. TYPE: DictDEFAULT: {}

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py post classmethod ¶

post( url, follow_redirects=True, timeout=10, stealthy_headers=True, proxy=None, retries=3, custom_config=None, **kwargs ) Make basic HTTP POST request for you but with some added flavors.

PARAMETER DESCRIPTION url Target url. TYPE: str

follow_redirects As the name says -- if enabled (default), redirects will be followed. TYPE: boolDEFAULT: True

timeout The time to wait for the request to finish in seconds. The default is 10 seconds. TYPE: Optional[Union[int, float]]DEFAULT: 10

stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain. TYPE: boolDEFAULT: True

proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030 TYPE: Optional[str]DEFAULT: None

retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries. TYPE: Optional[int]DEFAULT: 3

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

kwargs Any additional keyword arguments are passed directly to httpx.post() function so check httpx documentation for details. TYPE: DictDEFAULT: {}

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py put classmethod ¶

put( url, follow_redirects=True, timeout=10, stealthy_headers=True, proxy=None, retries=3, custom_config=None, **kwargs ) Make basic HTTP PUT request for you but with some added flavors.

PARAMETER DESCRIPTION url Target url TYPE: str

follow_redirects As the name says -- if enabled (default), redirects will be followed. TYPE: boolDEFAULT: True

timeout The time to wait for the request to finish in seconds. The default is 10 seconds. TYPE: Optional[Union[int, float]]DEFAULT: 10

stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain. TYPE: boolDEFAULT: True

proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030 TYPE: Optional[str]DEFAULT: None

retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries. TYPE: Optional[int]DEFAULT: 3

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

kwargs Any additional keyword arguments are passed directly to httpx.put() function so check httpx documentation for details. TYPE: DictDEFAULT: {}

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py delete classmethod ¶

delete( url, follow_redirects=True, timeout=10, stealthy_headers=True, proxy=None, retries=3, custom_config=None, **kwargs ) Make basic HTTP DELETE request for you but with some added flavors.

PARAMETER DESCRIPTION url Target url TYPE: str

follow_redirects As the name says -- if enabled (default), redirects will be followed. TYPE: boolDEFAULT: True

timeout The time to wait for the request to finish in seconds. The default is 10 seconds. TYPE: Optional[Union[int, float]]DEFAULT: 10

stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain. TYPE: boolDEFAULT: True

proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030 TYPE: Optional[str]DEFAULT: None

retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries. TYPE: Optional[int]DEFAULT: 3

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

kwargs Any additional keyword arguments are passed directly to httpx.delete() function so check httpx documentation for details. TYPE: DictDEFAULT: {}

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py display_config classmethod ¶

display_config() Source code in scrapling/engines/toolbelt/custom.py configure classmethod ¶

configure(**kwargs) Set multiple arguments for the parser at once globally

PARAMETER DESCRIPTION kwargs The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, auto_match, storage, storage_args, automatch_domain DEFAULT: {}

Source code in scrapling/engines/toolbelt/custom.py scrapling.fetchers.AsyncFetcher ¶

AsyncFetcher(*args, **kwargs) Bases: Fetcher

Source code in scrapling/engines/toolbelt/custom.py huge_tree class-attribute instance-attribute ¶

huge_tree = True auto_match class-attribute instance-attribute ¶

auto_match = False storage class-attribute instance-attribute ¶

storage = SQLiteStorageSystem keep_cdata class-attribute instance-attribute ¶

keep_cdata = False storage_args class-attribute instance-attribute ¶

storage_args = None keep_comments class-attribute instance-attribute ¶

keep_comments = False automatch_domain class-attribute instance-attribute ¶

automatch_domain = None parser_keywords class-attribute instance-attribute ¶

parser_keywords = ( "huge_tree", "auto_match", "storage", "keep_cdata", "storage_args", "keep_comments", "automatch_domain", ) get async classmethod ¶

get( url, follow_redirects=True, timeout=10, stealthy_headers=True, proxy=None, retries=3, custom_config=None, **kwargs ) Make basic HTTP GET request for you but with some added flavors.

PARAMETER DESCRIPTION url Target url. TYPE: str

follow_redirects As the name says -- if enabled (default), redirects will be followed. TYPE: boolDEFAULT: True

timeout The time to wait for the request to finish in seconds. The default is 10 seconds. TYPE: Optional[Union[int, float]]DEFAULT: 10

stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request had came from Google's search of this URL's domain. TYPE: boolDEFAULT: True

proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030 TYPE: Optional[str]DEFAULT: None

retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries. TYPE: Optional[int]DEFAULT: 3

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

kwargs Any additional keyword arguments are passed directly to httpx.get() function so check httpx documentation for details. TYPE: DictDEFAULT: {}

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py post async classmethod ¶

post( url, follow_redirects=True, timeout=10, stealthy_headers=True, proxy=None, retries=3, custom_config=None, **kwargs ) Make basic HTTP POST request for you but with some added flavors.

PARAMETER DESCRIPTION url Target url. TYPE: str

follow_redirects As the name says -- if enabled (default), redirects will be followed. TYPE: boolDEFAULT: True

timeout The time to wait for the request to finish in seconds. The default is 10 seconds. TYPE: Optional[Union[int, float]]DEFAULT: 10

stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain. TYPE: boolDEFAULT: True

proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030 TYPE: Optional[str]DEFAULT: None

retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries. TYPE: Optional[int]DEFAULT: 3

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

kwargs Any additional keyword arguments are passed directly to httpx.post() function so check httpx documentation for details. TYPE: DictDEFAULT: {}

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py put async classmethod ¶

put( url, follow_redirects=True, timeout=10, stealthy_headers=True, proxy=None, retries=3, custom_config=None, **kwargs ) Make basic HTTP PUT request for you but with some added flavors.

PARAMETER DESCRIPTION url Target url TYPE: str

follow_redirects As the name says -- if enabled (default), redirects will be followed. TYPE: boolDEFAULT: True

timeout The time to wait for the request to finish in seconds. The default is 10 seconds. TYPE: Optional[Union[int, float]]DEFAULT: 10

stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain. TYPE: boolDEFAULT: True

proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030 TYPE: Optional[str]DEFAULT: None

retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries. TYPE: Optional[int]DEFAULT: 3

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

kwargs Any additional keyword arguments are passed directly to httpx.put() function so check httpx documentation for details. TYPE: DictDEFAULT: {}

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py delete async classmethod ¶

delete( url, follow_redirects=True, timeout=10, stealthy_headers=True, proxy=None, retries=3, custom_config=None, **kwargs ) Make basic HTTP DELETE request for you but with some added flavors.

PARAMETER DESCRIPTION url Target url TYPE: str

follow_redirects As the name says -- if enabled (default), redirects will be followed. TYPE: boolDEFAULT: True

timeout The time to wait for the request to finish in seconds. The default is 10 seconds. TYPE: Optional[Union[int, float]]DEFAULT: 10

stealthy_headers If enabled (default), Fetcher will create and add real browser's headers and create a referer header as if this request came from Google's search of this URL's domain. TYPE: boolDEFAULT: True

proxy A string of a proxy to use for http and https requests, the format accepted is http://username:password@localhost:8030 TYPE: Optional[str]DEFAULT: None

retries The number of retries to do through httpx if the request failed for any reason. The default is 3 retries. TYPE: Optional[int]DEFAULT: 3

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

kwargs Any additional keyword arguments are passed directly to httpx.delete() function so check httpx documentation for details. TYPE: DictDEFAULT: {}

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py display_config classmethod ¶

display_config() Source code in scrapling/engines/toolbelt/custom.py configure classmethod ¶

configure(**kwargs) Set multiple arguments for the parser at once globally

PARAMETER DESCRIPTION kwargs The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, auto_match, storage, storage_args, automatch_domain DEFAULT: {}

Source code in scrapling/engines/toolbelt/custom.py scrapling.fetchers.PlayWrightFetcher ¶

PlayWrightFetcher(*args, **kwargs) Bases: BaseFetcher

A Fetcher class type that provide many options, all of them are based on PlayWright.

Using this Fetcher class, you can do requests with: - Vanilla Playwright without any modifications other than the ones you chose. - Stealthy Playwright with the stealth mode I wrote for it. It's still a work in progress but it bypasses many online tests like bot.sannysoft.com Some of the things stealth mode does include: 1) Patches the CDP runtime fingerprint. 2) Mimics some of the real browsers' properties by injecting several JS files and using custom options. 3) Using custom flags on launch to hide Playwright even more and make it faster. 4) Generates real browser's headers of the same type and same user OS then append it to the request. - Real browsers by passing the real_chrome argument or the CDP URL of your browser to be controlled by the Fetcher and most of the options can be enabled on it. - NSTBrowser's docker browserless option by passing the CDP URL and enabling nstbrowser_mode option.

Note that these are the main options with PlayWright but it can be mixed together.

Source code in scrapling/engines/toolbelt/custom.py huge_tree class-attribute instance-attribute ¶

huge_tree = True auto_match class-attribute instance-attribute ¶

auto_match = False storage class-attribute instance-attribute ¶

storage = SQLiteStorageSystem keep_cdata class-attribute instance-attribute ¶

keep_cdata = False storage_args class-attribute instance-attribute ¶

storage_args = None keep_comments class-attribute instance-attribute ¶

keep_comments = False automatch_domain class-attribute instance-attribute ¶

automatch_domain = None parser_keywords class-attribute instance-attribute ¶

parser_keywords = ( "huge_tree", "auto_match", "storage", "keep_cdata", "storage_args", "keep_comments", "automatch_domain", ) fetch classmethod ¶

fetch( url, headless=True, disable_resources=None, useragent=None, network_idle=False, timeout=30000, wait=0, page_action=None, wait_selector=None, wait_selector_state="attached", hide_canvas=False, disable_webgl=False, extra_headers=None, google_search=True, proxy=None, locale="en-US", stealth=False, real_chrome=False, cdp_url=None, nstbrowser_mode=False, nstbrowser_config=None, custom_config=None, ) Opens up a browser and do your request based on your chosen options below.

PARAMETER DESCRIPTION url Target url. TYPE: str

headless Run the browser in headless/hidden (default), or headful/visible mode. TYPE: Union[bool, str]DEFAULT: True

disable_resources Drop requests of unnecessary resources for speed boost. It depends but it made requests ~25% faster in my tests for some websites. Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. TYPE: boolDEFAULT: None

useragent Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it. TYPE: Optional[str]DEFAULT: None

network_idle Wait for the page until there are no network connections for at least 500 ms. TYPE: boolDEFAULT: False

timeout The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000. TYPE: Optional[float]DEFAULT: 30000

wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning Response object. TYPE: Optional[int]DEFAULT: 0

locale Set the locale for the browser if wanted. The default value is en-US. TYPE: Optional[str]DEFAULT: 'en-US'

page_action Added for automation. A function that takes the page object, does the automation you need, then returns page again. TYPE: Optional[Callable]DEFAULT: None

wait_selector Wait for a specific css selector to be in a specific state. TYPE: Optional[str]DEFAULT: None

wait_selector_state The state to wait for the selector given with wait_selector. Default state is attached. TYPE: SelectorWaitStatesDEFAULT: 'attached'

stealth Enables stealth mode, check the documentation to see what stealth mode does currently. TYPE: boolDEFAULT: False

real_chrome If you have chrome browser installed on your device, enable this and the Fetcher will launch an instance of your browser and use it. TYPE: boolDEFAULT: False

hide_canvas Add random noise to canvas operations to prevent fingerprinting. TYPE: boolDEFAULT: False

disable_webgl Disables WebGL and WebGL 2.0 support entirely. TYPE: boolDEFAULT: False

google_search Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name. TYPE: boolDEFAULT: True

extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together. TYPE: Optional[Dict[str, str]]DEFAULT: None

proxy The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. TYPE: Optional[Union[str, Dict[str, str]]]DEFAULT: None

cdp_url Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP. TYPE: Optional[str]DEFAULT: None

nstbrowser_mode Enables NSTBrowser mode, it have to be used with cdp_url argument or it will get completely ignored. TYPE: boolDEFAULT: False

nstbrowser_config The config you want to send with requests to the NSTBrowser. If left empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config. TYPE: Optional[Dict]DEFAULT: None

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py async_fetch async classmethod ¶

async_fetch( url, headless=True, disable_resources=None, useragent=None, network_idle=False, timeout=30000, wait=0, page_action=None, wait_selector=None, wait_selector_state="attached", hide_canvas=False, disable_webgl=False, extra_headers=None, google_search=True, proxy=None, locale="en-US", stealth=False, real_chrome=False, cdp_url=None, nstbrowser_mode=False, nstbrowser_config=None, custom_config=None, ) Opens up a browser and do your request based on your chosen options below.

PARAMETER DESCRIPTION url Target url. TYPE: str

headless Run the browser in headless/hidden (default), or headful/visible mode. TYPE: Union[bool, str]DEFAULT: True

disable_resources Drop requests of unnecessary resources for speed boost. It depends but it made requests ~25% faster in my tests for some websites. Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. TYPE: boolDEFAULT: None

useragent Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it. TYPE: Optional[str]DEFAULT: None

network_idle Wait for the page until there are no network connections for at least 500 ms. TYPE: boolDEFAULT: False

timeout The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000. TYPE: Optional[float]DEFAULT: 30000

wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning Response object. TYPE: Optional[int]DEFAULT: 0

locale Set the locale for the browser if wanted. The default value is en-US. TYPE: Optional[str]DEFAULT: 'en-US'

page_action Added for automation. A function that takes the page object, does the automation you need, then returns page again. TYPE: Optional[Callable]DEFAULT: None

wait_selector Wait for a specific css selector to be in a specific state. TYPE: Optional[str]DEFAULT: None

wait_selector_state The state to wait for the selector given with wait_selector. Default state is attached. TYPE: SelectorWaitStatesDEFAULT: 'attached'

stealth Enables stealth mode, check the documentation to see what stealth mode does currently. TYPE: boolDEFAULT: False

real_chrome If you have chrome browser installed on your device, enable this and the Fetcher will launch an instance of your browser and use it. TYPE: boolDEFAULT: False

hide_canvas Add random noise to canvas operations to prevent fingerprinting. TYPE: boolDEFAULT: False

disable_webgl Disables WebGL and WebGL 2.0 support entirely. TYPE: boolDEFAULT: False

google_search Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name. TYPE: boolDEFAULT: True

extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together. TYPE: Optional[Dict[str, str]]DEFAULT: None

proxy The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. TYPE: Optional[Union[str, Dict[str, str]]]DEFAULT: None

cdp_url Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP. TYPE: Optional[str]DEFAULT: None

nstbrowser_mode Enables NSTBrowser mode, it have to be used with cdp_url argument or it will get completely ignored. TYPE: boolDEFAULT: False

nstbrowser_config The config you want to send with requests to the NSTBrowser. If left empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config. TYPE: Optional[Dict]DEFAULT: None

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py display_config classmethod ¶

display_config() Source code in scrapling/engines/toolbelt/custom.py configure classmethod ¶

configure(**kwargs) Set multiple arguments for the parser at once globally

PARAMETER DESCRIPTION kwargs The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, auto_match, storage, storage_args, automatch_domain DEFAULT: {}

Source code in scrapling/engines/toolbelt/custom.py scrapling.fetchers.StealthyFetcher ¶

StealthyFetcher(*args, **kwargs) Bases: BaseFetcher

A Fetcher class type that is completely stealthy fetcher that uses a modified version of Firefox.

It works as real browsers passing almost all online tests/protections based on Camoufox. Other added flavors include setting the faked OS fingerprints to match the user's OS and the referer of every request is set as if this request came from Google's search of this URL's domain.

Source code in scrapling/engines/toolbelt/custom.py huge_tree class-attribute instance-attribute ¶

huge_tree = True auto_match class-attribute instance-attribute ¶

auto_match = False storage class-attribute instance-attribute ¶

storage = SQLiteStorageSystem keep_cdata class-attribute instance-attribute ¶

keep_cdata = False storage_args class-attribute instance-attribute ¶

storage_args = None keep_comments class-attribute instance-attribute ¶

keep_comments = False automatch_domain class-attribute instance-attribute ¶

automatch_domain = None parser_keywords class-attribute instance-attribute ¶

parser_keywords = ( "huge_tree", "auto_match", "storage", "keep_cdata", "storage_args", "keep_comments", "automatch_domain", ) fetch classmethod ¶

fetch( url, headless=True, block_images=False, disable_resources=False, block_webrtc=False, allow_webgl=True, network_idle=False, addons=None, wait=0, timeout=30000, page_action=None, wait_selector=None, humanize=True, wait_selector_state="attached", google_search=True, extra_headers=None, proxy=None, os_randomize=False, disable_ads=False, geoip=False, custom_config=None, additional_arguments=None, ) Opens up a browser and do your request based on your chosen options below.

PARAMETER DESCRIPTION url Target url. TYPE: str

headless Run the browser in headless/hidden (default), 'virtual' screen mode, or headful/visible mode. TYPE: Union[bool, Literal[virtual]]DEFAULT: True

block_images Prevent the loading of images through Firefox preferences. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. TYPE: boolDEFAULT: False

disable_resources Drop requests of unnecessary resources for a speed boost. It depends but it made requests ~25% faster in my tests for some websites. Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. TYPE: boolDEFAULT: False

block_webrtc Blocks WebRTC entirely. TYPE: boolDEFAULT: False

addons List of Firefox addons to use. Must be paths to extracted addons. TYPE: Optional[List[str]]DEFAULT: None

disable_ads Disabled by default, this installs uBlock Origin addon on the browser if enabled. TYPE: boolDEFAULT: False

humanize Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window. TYPE: Optional[Union[bool, float]]DEFAULT: True

allow_webgl Enabled by default. Disabling it WebGL not recommended as many WAFs now checks if WebGL is enabled. TYPE: boolDEFAULT: True

geoip Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. TYPE: boolDEFAULT: False

network_idle Wait for the page until there are no network connections for at least 500 ms. TYPE: boolDEFAULT: False

os_randomize If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS. TYPE: boolDEFAULT: False

timeout The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000. TYPE: Optional[float]DEFAULT: 30000

wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning Response object. TYPE: Optional[int]DEFAULT: 0

page_action Added for automation. A function that takes the page object, does the automation you need, then returns page again. TYPE: CallableDEFAULT: None

wait_selector Wait for a specific css selector to be in a specific state. TYPE: Optional[str]DEFAULT: None

wait_selector_state The state to wait for the selector given with wait_selector. Default state is attached. TYPE: SelectorWaitStatesDEFAULT: 'attached'

google_search Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name. TYPE: boolDEFAULT: True

extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together. TYPE: Optional[Dict[str, str]]DEFAULT: None

proxy The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. TYPE: Optional[Union[str, Dict[str, str]]]DEFAULT: None

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

additional_arguments Additional arguments to be passed to Camoufox as additional settings and it takes higher priority than Scrapling's settings. TYPE: DictDEFAULT: None

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py async_fetch async classmethod ¶

async_fetch( url, headless=True, block_images=False, disable_resources=False, block_webrtc=False, allow_webgl=True, network_idle=False, addons=None, wait=0, timeout=30000, page_action=None, wait_selector=None, humanize=True, wait_selector_state="attached", google_search=True, extra_headers=None, proxy=None, os_randomize=False, disable_ads=False, geoip=False, custom_config=None, additional_arguments=None, ) Opens up a browser and do your request based on your chosen options below.

PARAMETER DESCRIPTION url Target url. TYPE: str

headless Run the browser in headless/hidden (default), 'virtual' screen mode, or headful/visible mode. TYPE: Union[bool, Literal[virtual]]DEFAULT: True

block_images Prevent the loading of images through Firefox preferences. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. TYPE: boolDEFAULT: False

disable_resources Drop requests of unnecessary resources for a speed boost. It depends but it made requests ~25% faster in my tests for some websites. Requests dropped are of type font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, and stylesheet. This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. TYPE: boolDEFAULT: False

block_webrtc Blocks WebRTC entirely. TYPE: boolDEFAULT: False

addons List of Firefox addons to use. Must be paths to extracted addons. TYPE: Optional[List[str]]DEFAULT: None

disable_ads Disabled by default, this installs uBlock Origin addon on the browser if enabled. TYPE: boolDEFAULT: False

humanize Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window. TYPE: Optional[Union[bool, float]]DEFAULT: True

allow_webgl Enabled by default. Disabling it WebGL not recommended as many WAFs now checks if WebGL is enabled. TYPE: boolDEFAULT: True

geoip Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. TYPE: boolDEFAULT: False

network_idle Wait for the page until there are no network connections for at least 500 ms. TYPE: boolDEFAULT: False

os_randomize If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS. TYPE: boolDEFAULT: False

timeout The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000 TYPE: Optional[float]DEFAULT: 30000

wait The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning Response object. TYPE: Optional[int]DEFAULT: 0

page_action Added for automation. A function that takes the page object, does the automation you need, then returns page again. TYPE: CallableDEFAULT: None

wait_selector Wait for a specific css selector to be in a specific state. TYPE: Optional[str]DEFAULT: None

wait_selector_state The state to wait for the selector given with wait_selector. Default state is attached. TYPE: SelectorWaitStatesDEFAULT: 'attached'

google_search Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name. TYPE: boolDEFAULT: True

extra_headers A dictionary of extra headers to add to the request. The referer set by the google_search argument takes priority over the referer set here if used together. TYPE: Optional[Dict[str, str]]DEFAULT: None

proxy The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. TYPE: Optional[Union[str, Dict[str, str]]]DEFAULT: None

custom_config A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values. TYPE: DictDEFAULT: None

additional_arguments Additional arguments to be passed to Camoufox as additional settings and it takes higher priority than Scrapling's settings. TYPE: DictDEFAULT: None

RETURNS DESCRIPTION Response A Response object that is the same as Adaptor object except it has these added attributes: status, reason, cookies, headers, and request_headers Source code in scrapling/fetchers.py display_config classmethod ¶

display_config() Source code in scrapling/engines/toolbelt/custom.py configure classmethod ¶

configure(**kwargs) Set multiple arguments for the parser at once globally

PARAMETER DESCRIPTION kwargs The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, auto_match, storage, storage_args, automatch_domain DEFAULT: {}

Source code in scrapling/engines/toolbelt/custom.py

Custom Types API Reference¶ Here's the reference information for all custom types of classes Scrapling implemented, with all their parameters, attributes, and methods.

You can import all of them directly like below:

from scrapling.core.custom_types import TextHandler, TextHandlers, AttributesHandler scrapling.core.custom_types.TextHandler ¶ Bases: str

Extends standard Python string by adding more functionality

extract class-attribute instance-attribute ¶

extract = get_all extract_first class-attribute instance-attribute ¶

extract_first = get split ¶

split(sep=None, maxsplit=-1) Source code in scrapling/core/custom_types.py strip ¶

strip(chars=None) Source code in scrapling/core/custom_types.py lstrip ¶

lstrip(chars=None) Source code in scrapling/core/custom_types.py rstrip ¶

rstrip(chars=None) Source code in scrapling/core/custom_types.py capitalize ¶

capitalize() Source code in scrapling/core/custom_types.py casefold ¶

casefold() Source code in scrapling/core/custom_types.py center ¶

center(width, fillchar=' ') Source code in scrapling/core/custom_types.py expandtabs ¶

expandtabs(tabsize=8) Source code in scrapling/core/custom_types.py format ¶

format(*args, **kwargs) Source code in scrapling/core/custom_types.py format_map ¶

format_map(mapping) Source code in scrapling/core/custom_types.py join ¶

join(iterable) Source code in scrapling/core/custom_types.py ljust ¶

ljust(width, fillchar=' ') Source code in scrapling/core/custom_types.py rjust ¶

rjust(width, fillchar=' ') Source code in scrapling/core/custom_types.py swapcase ¶

swapcase() Source code in scrapling/core/custom_types.py title ¶

title() Source code in scrapling/core/custom_types.py translate ¶

translate(table) Source code in scrapling/core/custom_types.py zfill ¶

zfill(width) Source code in scrapling/core/custom_types.py replace ¶

replace(old, new, count=-1) Source code in scrapling/core/custom_types.py upper ¶

upper() Source code in scrapling/core/custom_types.py lower ¶

lower() Source code in scrapling/core/custom_types.py sort ¶

sort(reverse=False) Return a sorted version of the string

Source code in scrapling/core/custom_types.py clean ¶

clean() Return a new version of the string after removing all white spaces and consecutive spaces

Source code in scrapling/core/custom_types.py get ¶

get(default=None) Source code in scrapling/core/custom_types.py get_all ¶

get_all() Source code in scrapling/core/custom_types.py json ¶

json() Return json response if the response is jsonable otherwise throw error

Source code in scrapling/core/custom_types.py re ¶

re( regex: Union[str, Pattern[str]], check_match: Literal[True], replace_entities: bool = True, clean_match: bool = False, case_sensitive: bool = True, ) -> bool

re( regex: Union[str, Pattern[str]], replace_entities: bool = True, clean_match: bool = False, case_sensitive: bool = True, check_match: Literal[False] = False, ) -> TextHandlers[TextHandler]

re( regex, replace_entities=True, clean_match=False, case_sensitive=True, check_match=False, ) Apply the given regex to the current text and return a list of strings with the matches.

PARAMETER DESCRIPTION regex Can be either a compiled regular expression or a string. TYPE: Union[str, Pattern[str]]

replace_entities if enabled character entity references are replaced by their corresponding character TYPE: boolDEFAULT: True

clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching TYPE: boolDEFAULT: False

case_sensitive if disabled, function will set the regex to ignore letters case while compiling it TYPE: boolDEFAULT: True

check_match used to quickly check if this regex matches or not without any operations on the results TYPE: boolDEFAULT: False

Source code in scrapling/core/custom_types.py re_first ¶

re_first( regex, default=None, replace_entities=True, clean_match=False, case_sensitive=True, ) Apply the given regex to text and return the first match if found, otherwise return the default value.

PARAMETER DESCRIPTION regex Can be either a compiled regular expression or a string. TYPE: Union[str, Pattern[str]]

default The default value to be returned if there is no match DEFAULT: None

replace_entities if enabled character entity references are replaced by their corresponding character TYPE: boolDEFAULT: True

clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching TYPE: boolDEFAULT: False

case_sensitive if disabled, function will set the regex to ignore letters case while compiling it TYPE: boolDEFAULT: True

Source code in scrapling/core/custom_types.py scrapling.core.custom_types.TextHandlers ¶ Bases: List[TextHandler]

The :class:TextHandlers class is a subclass of the builtin List class, which provides a few additional methods.

extract_first class-attribute instance-attribute ¶

extract_first = get get_all class-attribute instance-attribute ¶

get_all = extract re ¶

re( regex, replace_entities=True, clean_match=False, case_sensitive=True, ) Call the .re() method for each element in this list and return their results flattened as TextHandlers.

PARAMETER DESCRIPTION regex Can be either a compiled regular expression or a string. TYPE: Union[str, Pattern[str]]

replace_entities if enabled character entity references are replaced by their corresponding character TYPE: boolDEFAULT: True

clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching TYPE: boolDEFAULT: False

case_sensitive if disabled, function will set the regex to ignore letters case while compiling it TYPE: boolDEFAULT: True

Source code in scrapling/core/custom_types.py re_first ¶

re_first( regex, default=None, replace_entities=True, clean_match=False, case_sensitive=True, ) Call the .re_first() method for each element in this list and return the first result or the default value otherwise.

PARAMETER DESCRIPTION regex Can be either a compiled regular expression or a string. TYPE: Union[str, Pattern[str]]

default The default value to be returned if there is no match DEFAULT: None

replace_entities if enabled character entity references are replaced by their corresponding character TYPE: boolDEFAULT: True

clean_match if enabled, this will ignore all whitespaces and consecutive spaces while matching TYPE: boolDEFAULT: False

case_sensitive if disabled, function will set the regex to ignore letters case while compiling it TYPE: boolDEFAULT: True

Source code in scrapling/core/custom_types.py get ¶

get(default=None) Returns the first item of the current list

PARAMETER DESCRIPTION default the default value to return if the current list is empty DEFAULT: None

Source code in scrapling/core/custom_types.py extract ¶

extract() Source code in scrapling/core/custom_types.py scrapling.core.custom_types.AttributesHandler ¶

AttributesHandler(mapping=None, **kwargs) Bases: Mapping[str, _TextHandlerType]

A read-only mapping to use instead of the standard dictionary for the speed boost but at the same time I use it to add more functionalities. If standard dictionary is needed, just convert this class to dictionary with dict function

Source code in scrapling/core/custom_types.py json_string property ¶

json_string Convert current attributes to JSON string if the attributes are JSON serializable otherwise throws error

get ¶

get(key, default=None) Acts like standard dictionary .get() method

Source code in scrapling/core/custom_types.py search_values ¶

search_values(keyword, partial=False) Search current attributes by values and return dictionary of each matching item

PARAMETER DESCRIPTION keyword The keyword to search for in the attributes values

partial If True, the function will search if keyword in each value instead of perfect match DEFAULT: False

Source code in scrapling/core/custom_types.py

Writing your retrieval system: Writing your retrieval system Scrapling uses SQLite by default, but this tutorial covers writing your storage system to store element properties there for auto-matching.

You might want to use FireBase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because the spiders will share with each other.

So first, to make your storage class work, it must do the big 3:

Inherit from the abstract class scrapling.core.storage_adaptors.StorageSystemMixin and accept a string argument, which will be the url argument to maintain the library logic. Use the decorator functools.lru_cache on top of the class to follow the Singleton design pattern as other classes. Implement methods save and retrieve, as you see from the type hints: The method save returns nothing and will get two arguments from the library The first one is of type lxml.html.HtmlElement, which is the element itself. It must be converted to a dictionary using the function element_to_dict in submodule scrapling.core.utils._StorageTools to keep the same format and save it to your database as you wish. The second one is a string, the identifier used for retrieval. The combination result of this identifier and the url argument from initialization must be unique for each row, or the auto-match will be messed up. The method retrieve takes a string, which is the identifier; using it with the url passed on initialization, the element's dictionary is retrieved from the database and returned if it exists; otherwise, it returns None. If the instructions weren't clear enough for you, you can check my implementation using SQLite3 in storage_adaptors file

If your class meets these criteria, the rest is easy. If you plan to use the library in a threaded application, ensure your class supports it. The default used class is thread-safe.

Some helper functions are added to the abstract class if you want to use them. It's easier to see it for yourself in the code; it's heavily commented :)

Real-World Example: Redis Storage¶ Here's a more practical example generated by AI using Redis:

import redis import orjson from functools import lru_cache from scrapling.core.storage_adaptors import StorageSystemMixin from scrapling.core.utils import _StorageTools

@lru_cache(None) class RedisStorage(StorageSystemMixin): def init(self, host='localhost', port=6379, db=0, url=None): super().init(url) self.redis = redis.Redis( host=host, port=port, db=db, decode_responses=False )

def save(self, element, identifier: str) -> None:
    # Convert element to dictionary
    element_dict = _StorageTools.element_to_dict(element)

    # Create key
    key = f"scrapling:{self._get_base_url()}:{identifier}"

    # Store as JSON
    self.redis.set(
        key,
        orjson.dumps(element_dict)
    )

def retrieve(self, identifier: str) -> dict:
    # Get data
    key = f"scrapling:{self._get_base_url()}:{identifier}"
    data = self.redis.get(key)

    # Parse JSON if exists
    if data:
        return orjson.loads(data)
    return None

Using Scrapling's custom types:

You can take advantage of the custom-made types for Scrapling and use them outside the library if you want. It's better than copying their code, after all :)

All current types can be imported alone like below¶

from scrapling.core.custom_types import TextHandler, AttributesHandler

somestring = TextHandler('{}') somestring.json() '{}' somedict_1 = AttributesHandler({'a': 1}) somedict_2 = AttributesHandler(a=1) Note that TextHandler is a subclass of Python's str, so all normal operations/methods that work with Python strings will work. If you want to check for the type in your code, it's better to depend on Python's built-in function issubclass.

The class AttributesHandler is a subclass of collections.abc.Mapping, so it's immutable (read-only), and all operations are inherited from it. The data passed can be accessed later through the _data property, but be careful; it's of type types.MappingProxyType, so it's immutable (read-only) as well (faster than collections.abc.Mapping by fractions of seconds).

So, to make it simple for you if you are new to Python, the same operations and methods from the Python standard dict type will all work with class AttributesHandler except the ones that try to modify the actual data.

If you want to modify the data inside AttributesHandler, you have to convert it to a dictionary first, like using the dict function, and then modify it outside.