#web-crawler #chunking #content

spider_transformations

Transformation utils to use for Spider Web Crawler

515 stable releases

new 2.36.6 Mar 30, 2025
2.31.4 Feb 28, 2025
2.23.3 Dec 31, 2024
2.13.84 Nov 30, 2024
0.0.3 Sep 21, 2024

#1711 in Command line utilities

Download history 4251/week @ 2024-12-08 913/week @ 2024-12-15 954/week @ 2024-12-22 1871/week @ 2024-12-29 2533/week @ 2025-01-05 690/week @ 2025-01-12 1825/week @ 2025-01-19 3315/week @ 2025-01-26 2572/week @ 2025-02-02 1684/week @ 2025-02-09 3680/week @ 2025-02-16 1324/week @ 2025-02-23 2171/week @ 2025-03-02 3007/week @ 2025-03-09 1741/week @ 2025-03-16 2370/week @ 2025-03-23

9,442 downloads per month
Used in spider_utils

MIT license

205KB
5K SLoC

spider_transformations

The Rust spider cloud transformation library built for performance, AI, and multiple locales. The library is used on Spider Cloud for data cleaning.

Usage

[dependencies]
spider_transformations = "2"
use spider_transformations::transformation::content;

fn main() {
    // page comes from the spider object when streaming.
    let mut conf = content::TransformConfig::default();
    conf.return_format = content::ReturnFormat::Markdown;
    let content = content::transform_content(&page, &conf, &None, &None);
}

Transfrom types

  1. Markdown
  2. Commonmark
  3. Text
  4. Markdown (Text Map) or HTML2Text
  5. WIP: HTML2XML

Enhancements

  1. Readability
  2. Encoding

Chunking

There are several chunking utils in the transformation mod.

This project has rewrites and forks of html2md, and html2text for performance and bug fixes.

License

MIT

Dependencies

~22–37MB
~642K SLoC