#file #chunk #chunking #process-file #concurrency

file-chunker

Efficiently process a file in (approximately) equally-sized parts

2 releases

0.1.1 Feb 10, 2022
0.1.0 Feb 9, 2022

#1205 in Filesystem

Download history 130/week @ 2024-11-18 221/week @ 2024-11-25 159/week @ 2024-12-02 119/week @ 2024-12-09 218/week @ 2024-12-16 115/week @ 2024-12-23 28/week @ 2024-12-30 46/week @ 2025-01-06 82/week @ 2025-01-13 103/week @ 2025-01-20 130/week @ 2025-01-27 148/week @ 2025-02-03 382/week @ 2025-02-10 157/week @ 2025-02-17 193/week @ 2025-02-24 155/week @ 2025-03-03

903 downloads per month
Used in krapslog

MIT license

17KB
222 lines

file-chunker

This crate provides the FileChunker type, which is useful for efficiently reading a file in (approximately) equally-sized parts.

The original use case was to process a log file in chunks, one thread per chunk, and to guarantee that each chunk ended with a full line of text.

Example

use file_chunker::FileChunker;
let file = std::fs::File::open("/path/to/file").unwrap();
let chunker = FileChunker::new(&file).unwrap();
chunker.chunks(1024, Some('\n'))
    .unwrap()
    .iter()
    .for_each(|chunk| {
        println!("{:?}", chunk);
    });

lib.rs:

This crate provides the FileChunker type, which is useful for efficiently reading a file in (approximately) equally-sized parts.

The original use case was to process a log file in chunks, one thread per chunk, and to guarantee that each chunk ended with a full line of text.

Example

use file_chunker::FileChunker;

let file = std::fs::File::open("/path/to/file").unwrap();
let chunker = FileChunker::new(&file).unwrap();
chunker.chunks(1024, Some('\n'))
    .unwrap()
    .iter()
    .for_each(|chunk| {
        println!("{:?}", chunk);
    });

Dependencies

~300KB