6 releases (stable)
1.2.2 | Jul 7, 2024 |
---|---|
1.2.1 | Feb 22, 2024 |
1.2.0 | Jan 4, 2024 |
1.1.0 | Dec 23, 2023 |
0.2.2 | Aug 20, 2023 |
#147 in Asynchronous
56 downloads per month
Used in ansi_parser_extended
74KB
1K
SLoC
get_chunk
About
get_chunk
is a library for creating file iterators or streams (asynchronous iterators),
specialized in efficient file chunking. The main task, the ability to retrieve chunk data especially from large files.
Key Features:
- File Chunking: Divide files, including large ones, into seamless chunks with each "Next" iteration.
- Modes: Choose between automatic tuning or manually set chunk size based on percentage or byte count.
- Automatic chunking: Each "Next" iteration dynamically determines an optimal chunk size, facilitating efficient handling of even large files.
⚠️ Important Notice:
The algorithm adjusts chunk sizes for optimal performance after the "Next" call, taking into account available RAM. However, crucially, this adjustment occurs only after the current chunk is sent and before the subsequent "Next" call.
It's important to note a potential scenario: Suppose a chunk is 15GB, and there's initially 16GB of free RAM. If, between the current and next "Next" calls, 2GB of RAM becomes unexpectedly occupied, the current 15GB chunk will still be processed. This situation introduces a risk, as the system might either reclaim resources (resulting in io::error) or lead to a code crash.
Iterators created by get_chunk
don't store the entire file in memory, especially for large datasets.
Their purpose is to fetch data from files in chunks, maintaining efficiency.
Key Points:
- Limited File Retention: Creating an iterator for a small file might result in fetching all data, OS-dependent. However, this doesn't guarantee file persistence after iterator creation.
- Deletion Warning: Deleting a file during iterator or stream iterations will result in an error. These structures don't track the last successful position.
- No File Restoration: Attempting to restore a deleted file during iterations is not supported. These structures don't keep track of the file's original state.
Iterator version
Example
use get_chunk::iterator::FileIter;
// Note: requires a `size_format` attribute.
use get_chunk::data_size_format::iec::IECUnit;
fn main() -> std::io::Result<()> {
let file_iter = FileIter::new("file.txt")?;
// or
// let file_iter = FileIter::try_from(File::open("file.txt")?)?;
// ...
for chunk in file_iter {
match chunk {
Ok(data) => {
// some calculations with chunk
//.....
println!("{}", IECUnit::auto(data.len() as f64));
}
Err(_) => break,
}
}
Ok(())
}
Example
use get_chunk::{
// Note: requires a `size_format` attribute.
data_size_format::iec::{IECSize, IECUnit},
iterator::FileIter,
ChunkSize,
};
fn main() -> std::io::Result<()> {
let file_iter = FileIter::new("file.txt")?
.include_available_swap()
.set_mode(ChunkSize::Bytes(40000))
.set_start_position_bytes(IECUnit::new(432.0, IECSize::Mebibyte).into());
Ok(())
}
Stream version
Example
// Note: requires the `size_format` and `stream` attributes.
use get_chunk::data_size_format::iec::IECUnit;
use get_chunk::stream::{FileStream, StreamExt};
#[tokio::main]
async fn main() -> std::io::Result<()> {
let mut file_stream = FileStream::new("file.txt").await?;
// or
// let mut file_stream = FileStream::try_from_data(File::open("file.txt").await?)?;
// ...
while let Ok(chunk) = file_stream.try_next().await {
match chunk {
Some(chunk) => {
// some calculations with chunk
},
None => break,
}
}
Ok(())
}
How it works
The calculate_chunk
function in the ChunkSize
enum determines the optimal chunk size based on various parameters. Here's a breakdown of how the size is calculated:
The variables prev
and now
represent the previous and current read time, respectively.
prev:
Definition: prev
represents the time taken to read a piece of data in the previous iteration.
now:
Definition: now
represents the current time taken to read the data fragment in the current iteration.
-
Auto Mode:
- If the previous read time (
prev
) is greater than zero:- If the current read time (
now
) is also greater than zero:- If
now
is less thanprev
, decrease the chunk size usingdecrease_chunk
method. - If
now
is greater than or equal toprev
, increase the chunk size usingincrease_chunk
method.
- If
- If
now
is zero or negative, maintain the previous chunk size (prev
).
- If the current read time (
- If the previous read time is zero or negative, use the default chunk size based on the file size and available RAM.
- If the previous read time (
-
Percent Mode:
- Calculate the chunk size as a percentage of the total file size using the
percentage_chunk
method. The percentage is capped between 0.1% and 100%.
- Calculate the chunk size as a percentage of the total file size using the
-
Bytes Mode:
- Calculate the chunk size based on the specified number of bytes using the
bytes_chunk
method. The size is capped by the file size and available RAM.
- Calculate the chunk size based on the specified number of bytes using the
Key Formulas:
- Increase Chunk Size:
(prev * (1.0 + ((now - prev) / prev).min(0.15))).min(ram_available * 0.85).min(f64::MAX)
- Decrease Chunk Size:
(prev * (1.0 - ((prev - now) / prev).min(0.45))).min(ram_available * 0.85).min(f64::MAX)
- Default Chunk Size:
(file_size * (0.1 / 100.0)).min(ram_available * 0.85).min(f64::MAX)
- Percentage Chunk Size:
(file_size * (percentage.min(100.0).max(0.1) / 100.0)).min(ram_available * 0.85)
- Bytes Chunk Size:
(bytes as f64).min(file_size).min(ram_available * 0.85)
License
Dependencies
~0.7–27MB
~395K SLoC