Complex Web Page

, Some page] Generate a CSS shortened selector if possible, or generate the full selector >>> article.generate_css_selector 'body > div > article' >>> article.generate_full_css_selector 'body > div > article' Same case with XPath >>> article.generate_xpath_selector "//body/div/article" >>> article.generate_full_xpath_selector "//body/div/article" Traversal¶ Using the elements we found above, we will go over the properties/methods for moving in the page in detail. If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online for a better understanding. If you are too lazy to search about it, here's a quick explanation to give you a good idea. Simply put, the html element is the root of the website's tree, as every page starts with an html element. This element will be directly above elements like head and body. These are considered "children" of the html element, and the html element is considered their "parent." The element body is a "sibling" of the element head and vice versa. Accessing the parent of an element >>> article.parent

>>> article.parent.tag 'div' You can chain it as you want, which applies to all similar properties/methods we will review. >>> article.parent.parent.tag 'body' Get the children of an element >>> article.children [Product 1' parent='

, This is product 1...' parent='

, $10.99' parent='

, In stock: 5] Get all elements underneath an element. It acts as a nested version of the children property >>> article.below_elements [Product 1' parent='

, This is product 1...' parent='

, $10.99' parent='

, In stock: 5] This element returns the same result as the children property because its children don't have children. Another example of using the element with the product-list class will clear the difference between the children property and the below_elements property >>> products_list = page.css_first('.product-list') >>> products_list.children [, , ] >>> products_list.below_elements [, Product 1' parent='

, This is product 1...' parent='

, $10.99' parent='

, In stock: 5, , ...] Get the siblings of an element >>> article.siblings [, ] Get the next element of the current element >>> article.next # gets the next element, the same logic applies to `quote.previous` The same logic applies to the previous property >>> article.previous # It's the first child, so it doesn't have a previous element >>> second_article = page.css_first('.product[data-id="2"]') >>> second_article.previous You can check easily and pretty fast if an element has a specific class name or not >>> article.has_class('product') True If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element, like the example below for ancestor in article.iterancestors(): # do something with it... You can search for a specific ancestor of an element that satisfies a function; all you need to do is to pass a function that takes an Adaptor object as an argument and return True if the condition satisfies or False otherwise like below: >>> article.find_ancestor(lambda ancestor: ancestor.has_class('product-list'))

>>> article.find_ancestor(lambda ancestor: ancestor.css('.product-list')) # Same result, different approach

Adaptors¶ The class Adaptors is the "List" version of the Adaptor class. It inherits from the Python standard List type, so it shares all List properties and methods while adding more methods to make the operations you want to execute on the Adaptor instances within more straightforward. In the Adaptor class, all methods/properties that should return a group of elements return them as an Adaptors class instance. The only exceptions are when you use the CSS/XPath methods as follows: If you selected a text node with the selector, then the return type will be TextHandler/TextHandlers. Examples: >>> page.css('a::text') # -> TextHandlers >>> page.xpath('//a/text()') # -> TextHandlers >>> page.css_first('a::text') # -> TextHandler >>> page.xpath_first('//a/text()') # -> TextHandler >>> page.css('a::attr(href)') # -> TextHandlers >>> page.xpath('//a/@href') # -> TextHandlers >>> page.css_first('a::attr(href)') # -> TextHandler >>> page.xpath_first('//a/@href') # -> TextHandler If you used a combined selector that returns mixed types, the result will be a Python standard List. Examples: >>> page.css('.price_color') # -> Adaptors >>> page.css('.product_pod a::attr(href)') # -> TextHandlers >>> page.css('.price_color, .product_pod a::attr(href)') # -> List Let's see what Adaptors class adds to the table with that out of the way. Properties¶ Apart from the normal operations on Python lists like iteration, slicing, etc... You can do the following: Execute CSS and XPath selectors directly on the Adaptor instances it has while the arguments and the return types are the same as Adaptor's css and xpath methods. This, of course, makes chaining methods very straightforward. >>> page.css('.product_pod a') [ , , , ...] >>> page.css('.product_pod').css('a') # Returns the same result [ , , , ...] Run the re and re_first methods directly. They take the same arguments passed as the Adaptor class. I'm still leaving these methods to be explained in the TextHandler section below. However, in this class, the re_first behaves differently as it runs re on each Adaptor within and returns the first one with a result. The re method will return a TextHandlers object as normal that has all the results combined in one TextHandlers instance. >>> page.css('.price_color').re(r'[\d\.]+') ['51.77', '53.74', '50.10', '47.82', '54.23', ...] >>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html') ['a-light-in-the-attic_1000', 'tipping-the-velvet_999', 'soumission_998', 'sharp-objects_997', ...] With the search method, you can search quickly in the available Adaptor classes. The function you pass must accept an Adaptor instance as the first argument and return True/False. The method will return the first Adaptor instance that satisfies the function; otherwise, it will return None. # Find all the products with price '53.23' >>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23 >>> page.css('.product_pod').search(search_function)

You can use the filter method, too, which takes a function like the search method but returns an Adaptors instance of all the Adaptor classes that satisfy the function # Find all products with prices over $50 >>> filtering_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) > 50 >>> page.css('.product_pod').filter(filtering_function) [

, ...] TextHandler¶ This class is mandatory to understand, as all methods/properties that should return a string for you will return TextHandler, and the ones that should return a list of strings will return TextHandlers instead. TextHandler is a subclass of the standard Python string, so you can do anything with it. So, what is the difference that requires a different naming? Of course, TextHandler provides extra methods and properties that the standard Python strings can't do. We will review them now, but remember that all methods and properties in all classes that return string(s) are returning TextHandler, which opens the door for creativity and makes the code shorter and cleaner, as you will see. Also, you can import it directly and use it on any string, which we will explain later. Usage¶ First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like split, replace, strip, etc., all return a TextHandler again, so you can chain them as you want. If you find a method or property that returns a standard string instead of TextHandler, please open an issue, and we will override it as well. First, we start with the re and re_first methods. These are the same methods that exist in the rest of the classes (Adaptor, Adaptors, and TextHandlers), so they will take the same arguments as well. The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance. Also, it takes other helpful arguments, which are: - **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters. - **clean_match**: It's disabled by default. This makes the method ignore all whitespaces and consecutive spaces while matching. - **case_sensitive**: It's enabled by default. As the name implies, disabling it will make the regex ignore letters case while compiling it. You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method. ```python >>> page.css('.price_color').re(r'[\d\.]+') ['51.77', '53.74', '50.10', '47.82', '54.23', ...] >>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html') ['a-light-in-the-attic_1000', 'tipping-the-velvet_999', 'soumission_998', 'sharp-objects_997', ...] ``` To explain the other arguments better, we will use a custom string for each example below ```python >>> from scrapling import TextHandler >>> test_string = TextHandler('hi there') # Hence the two spaces >>> test_string.re('hi there') >>> test_string.re('hi there', clean_match=True) # Using `clean_match` will clean the string before matching the regex ['hi there'] >>> test_string2 = TextHandler('Oh, Hi Mark') >>> test_string2.re_first('oh, hi Mark') >>> test_string2.re_first('oh, hi Mark', case_sensitive=False) # Hence disabling `case_sensitive` 'Oh, Hi Mark' # Mixing arguments >>> test_string.re('hi there', clean_match=True, case_sensitive=False) ['hi There'] ``` Another use of the idea of replacing strings with `TextHandler` everywhere is a property like `html_content` returns `TextHandler` so you can do regex on the HTML content if you want: ```python >>> page.html_content.re('div class=".*">(.*)>> page.css_first('#page-data::text') '\n {\n "lastUpdated": "2024-09-22T10:30:00Z",\n "totalProducts": 3\n }\n ' >>> page.css_first('#page-data::text').json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} Hence, if you didn't specify a text node while selecting an element (like the text content or an attribute text content), the text content will be selected automatically like this >>> page.css_first('#page-data').json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} The Adaptor class adds one thing here, too; let's say this is the page we are working with:

The Adaptor class has the get_all_text method, which you should be aware of by now. This method returns a TextHandler, of course. So, as you know here, if you did something like this >>> page.css_first('div::text').json() You will get an error because the div tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have text content at all. In this case, the get_all_text method comes to the rescue, so you can do something like that >>> page.css_first('div').get_all_text(ignore_tags=[]).json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} I used the ignore_tags argument here because the default value of it is ('script', 'style',), as you are aware. Another related behavior you should be aware of is the case while using any of the fetchers, which we will explain later. If you have a JSON response like this example: >>> page = Adaptor("""{"some_key": "some_value"}""") Because the Adaptor class is optimized to deal with HTML pages, it will deal with it as a broken HTML response and fix it, so if you used the html_content property, you get this >>> page.html_content '

{"some_key": "some_value"}

' Here, you can use json method directly, and it will work >>> page.json() {'some_key': 'some_value'} You might wonder how this happened while the html tag lacks direct text? Well, for these cases like JSON responses, I made the .json() method inside the Adaptor class to check if the current element doesn't have text content; it will use the get_all_text method directly. It might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions. Another handy method is .clean(), this will remove all white spaces and consecutive spaces for you and return a new TextHandler, wonderful >>> TextHandler('\n wonderful idea, \reh?').clean() 'wonderful idea, eh?' Another method that might be helpful in some cases is the .sort() method to sort the string for you as you do with lists >>> TextHandler('acb').sort() 'abc' Or do it in reverse: >>> TextHandler('acb').sort(reverse=True) 'cba' Other methods and properties will be added over time, but remember that this class is returned in place of strings nearly everywhere in the library. TextHandlers¶ You probably guessed it: This class is similar to Adaptors and Adaptor, but here it inherits the same logic and method as standard lists, with only re and re_first as new methods. The only difference is that the re_first method logic here does re on each TextHandler within and returns the first result it has or None. Nothing is new to explain here, but new methods will be added here with time. AttributesHandler¶ This is a read-only version of Python's standard dictionary or dict that's only used to store the attributes of each element or each Adaptor instance, in other words. >>> print(page.find('script').attrib) {'id': 'page-data', 'type': 'application/json'} >>> type(page.find('script').attrib).__name__ 'AttributesHandler' Because it's read-only, it will use fewer resources than the standard dictionary. Still, it has the same dictionary method/properties other than those allowing you to modify/override the data. It currently adds two extra simple methods: The search_values method In standard dictionaries, you can do dict.get("key_name") to check if a key exists. However, if you want to search by values instead of keys, it will take you some code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item. A simple example would be >>> for i in page.find('script').attrib.search_values('page-data'): print(i) {'id': 'page-data'} But this method provides the partial argument as well, which allows you to search by part of the value: >>> for i in page.find('script').attrib.search_values('page', partial=True): print(i) {'id': 'page-data'} These examples won't happen in the real world; most likely, a more real-world example would be using it with the find_all method to find all elements that have a specific value in their arguments: >>> page.find_all(lambda element: list(element.attrib.search_values('product'))) [, , ] All these elements have 'product' as a value for the attribute class. Hence, I used the list function here because search_values returns a generator, so it would be True for all elements. The json_string property This property converts current attributes to JSON string if the attributes are JSON serializable; otherwise, it throws an error >>> page.find('script').attrib.json_string b'{"id":"page-data","type":"application/json"}' Using automatch feature Introduction¶ Auto-matching is one of Scrapling's most powerful features. It allows your scraper to survive website changes by intelligently tracking and relocating elements. Let's say you are scraping a page with a structure like this: