#logging #parser #compile #pt2

bin+lib tlparse

Parse TORCH_LOG logs produced by PyTorch torch.compile

33 releases

new 0.3.39 Mar 27, 2025
0.3.35 Feb 20, 2025
0.3.30 Dec 5, 2024
0.3.29 Nov 21, 2024
0.3.11 Mar 26, 2024

#184 in Machine learning

Download history 281/week @ 2024-12-06 153/week @ 2024-12-13 54/week @ 2024-12-20 33/week @ 2024-12-27 51/week @ 2025-01-03 43/week @ 2025-01-10 123/week @ 2025-01-17 533/week @ 2025-01-24 276/week @ 2025-01-31 198/week @ 2025-02-07 245/week @ 2025-02-14 197/week @ 2025-02-21 155/week @ 2025-02-28 305/week @ 2025-03-07 428/week @ 2025-03-14 237/week @ 2025-03-21

1,153 downloads per month

BSD-3-Clause

450KB
3K SLoC

Rust 3K SLoC // 0.0% comments JavaScript 344 SLoC // 0.2% comments

tlparse: Parse structured PT2 logs

tlparse parses structured torch trace logs and outputs HTML files analyzing data.

Quick start: Run PT2 with the TORCH_TRACE environment variable set:

TORCH_TRACE=/tmp/my_traced_log_dir example.py

Feed input into tlparse:

tlparse /tmp/my_traced_log_dir -o tl_out/

Adding custom parsers

You can extend tlparse with custom parsers which take existing structured log data and output any file. To do so, first implement StructuredLogParser with your own trait:

pub struct MyCustomParser;
impl StructuredLogParser for MyCustomParser {
    fn name(&self) -> &'static str {
        "my_custom_parser"
    }
    fn get_metadata<'e>(&self, e: &'e Envelope) -> Option<Metadata<'e>> {
        // Get required metadata from the Envelope.
        // You'll need to update Envelope with your custom Metadata if you need new types here
        ....
    }

    fn parse<'e>(&self,
        lineno: usize,
        metadata: Metadata<'e>,
        _rank: Option<u32>,
        compile_id: &Option<CompileId>,
        payload: &str
    ) -> anyhow::Result<ParserResult> {
       // Use the metadata and payload however you'd like
       // Return either a ParserOutput::File(filename, payload) or ParserOutput::Link(name, url)
    }
}

How to release

  1. Make a release commit by updating Cargo.toml and then running cargo update
  2. Push the release commit and a tag for it. This will trigger PyPI release
  3. cargo publish

Dependencies

~10–20MB
~257K SLoC