12 unstable releases (3 breaking)
0.3.0 | Dec 9, 2021 |
---|---|
0.2.0 | Dec 7, 2021 |
0.1.1 | Dec 5, 2021 |
0.0.9 | Dec 3, 2021 |
0.0.4 | Mar 12, 2017 |
#2447 in Parser implementations
205 downloads per month
115KB
3K
SLoC
xml_oxide
Rust XML parser implementation that parses any well-formed XML defined in the W3C Spec in a streaming way.
Features
- It uses constant-like memory for large XML files
- Fast enough for most use cases. It can parse a 1GB XML file(in memory) around 23 seconds
- Supports Namespaces in XML 1.0
- It only supports UTF-8 encoding
- It is a non-validating parser, it does important well-formedness checks
- Currently, it ignores well-formedness inside Processing Instructions, DTD/DOCTYPE and parses them as raw strings. It checks the general well-formedness including these entities. (It even parses comments inside DOCTYPE to achieve this)
- It can parse not-well-formed documents (please report as a bug)
- Entities that can be large are parsed as chunks to keep memory usage low: Character Data, CDATA Section, Comment, Whitespace
- Reading chunk size is currently default 8KB, not configurable. Internal buffer is 16KB. If you have an element tag or DOCTYPE declaration that is bigger than the buffer, it can backtrack and allocate more memory for the parsing operation. 1 byte chunk size is used for testing.
Unsafe usage
unsafe
is used for functionstd::str::from_utf8_unchecked
. It is used on a slice of bytes that is already checked to be a valid UTF8 string withstd::str::from_utf8
before. The performance saving is not tested though.- RefCell is not used anymore. Interestingly, just changing
RefCell<Vec<u8>>
tocircular::Buffer
passed Rust borrow checks. I'm leaving this note as a reference.RefCell
is used because Rust is too restrictive for using mutables in conditional loops. Hopefully, non-lexical lifetimes will get better over time.
To Do
- Because the namespace spec brings constraints around the usage of ":" in names. Provide
namespace-aware=false
option to parse otherwise valid XML 1.0 documents . - More tests
- Parsing every entity including DTD to be able to utilize conformance test suite.
Example Usage
In this example StartElement and EndElement events are counted. Note that you can find more examples under tests
directory.
StartElement
also include empty tags. Checked byis_empty
.- Reference entities like
&
or<
comes in its own event(Not inCharacters
). - Character/numerical and predefined entity references are resolved. Custom entity definitions are passed as raw.
- Check sax::Event to see all available event types
use std::fs::File;
use xml_oxide::{sax::parser::Parser, sax::Event};
fn main() {
println!("Starting...");
let mut counter: usize = 0;
let mut end_counter: usize = 0;
let now = std::time::Instant::now();
let f = File::open("./tests/xml_files/books.xml").unwrap();
let mut p = Parser::from_reader(f);
loop {
let res = p.read_event();
match res {
Ok(event) => match event {
Event::StartDocument => {}
Event::EndDocument => {
break;
}
Event::StartElement(el) => {
//You can differantiate between Starting Tag and Empty Element Tag
if !el.is_empty {
counter = counter + 1;
// print every 10000th element name
if counter % 10000 == 0 {
println!("%10000 start {}", el.name);
}
}
}
Event::EndElement(el) => {
end_counter += 1;
if el.name == "feed" {
break;
}
}
Event::Characters(_) => {}
Event::Reference(_) => {}
_ => {}
},
Err(err) => {
println!("{}", err);
break;
}
}
}
println!("Start event count:{}", counter);
println!("End event count:{}", end_counter);
let elapsed = now.elapsed();
println!("Time elapsed: {:.2?}", elapsed);
}
History & Credits
I tried to specify a push parser interface like the Java SAX library and implement it in 2017. The idea was to provide an interface that can have multiple implementations in the community. It was working(albeit slowly) but the main problem was that a push parser is not ergonomic in Rust. After thinking for a long time and learning more about Rust I decided to implement a pull parser. Currently, the SAX(pull) interface is just an enum
and its behavior(like the possibility of splitting characters for each call).
If you want to use xml_sax
interface to implement another parser we can discuss improving the interface. Currently, it is integrated into this crate.
Why a pull parser? section in pulldown-cmark
is a great explanation.
The current interface is inspired by quick-xml
, xml-rs
, and Java libraries.
nom
is a great library. It is just a crystallized & better version of what you would do naively at first try(I know). It also shows the power of composability in Rust.
Dependencies
~1–1.8MB
~36K SLoC