17 unstable releases (3 breaking)
new 0.8.11 | Nov 7, 2024 |
---|---|
0.8.5 | Nov 5, 2024 |
0.8.3 | Oct 29, 2024 |
0.7.3 | Oct 10, 2024 |
0.5.5 | Aug 30, 2024 |
#404 in Database interfaces
816 downloads per month
Used in adguard-flm-ffi
1MB
9K
SLoC
Filter list manager library core
Overview
This crate represents a library for managing AdGuard filter lists.
This library can:
- Fetch filter lists
- Store the downloaded filter list
- Perform filter lists updates
- ... and more
Filters analysis
List of meta tags that the library parses from filter content
! Title
- Name of the filter.! Description
- Detailed description of the filter.! Version
- Current version of the filter.! Expires
- Filter expiration period. Will be converted into seconds. See the tests for an example If this field is missing in the metadata, the global value from the configuration will be used. Before updating the filter, the value will be checked and aligned to the lower boundary (3600) if it is less than this value.! Homepage
- Filter website/homepage.! TimeUpdated
- When this filter was updated in registry. Format:2024-08-13T13:30:53+00:00
.! Last modified
- Alias forTimeUpdated
. Format:2024-08-13T12:01:26.703Z
. You can choose one format for both fields.! Diff-Path
- Differential updates information! License
- Link to filter license.! Checksum
- Filter's base64(md5-checksum). Before update/install filter, checksum will be calculated and compared. See the source here
List of filter preprocessor directives supported by the library
See AdGuard preprocessor directives
The library supports:
!#include file_path
- Includes contents of file into filter and process.file_path
must be:- Absolute url with the same origin as the parent filter.
- Relative url.
- File url (only if the parent filter's url has
file
scheme).
!#if/!#endif/!#else
- Condition compilation directives. They can be nested. Supported tokens:()
- parenthesestrue/false
- boolean values&& ||
- AND/OR operators!
- NOT operator- Literal compiler constant from configuration. For example,
windows
,mac
, etc... It works like this: if the constant encountered is in theconfiguration.compiler_conditional_constants
list, then the condition becomes true, false otherwise
See the tests for more information:
Usage
Create and setup configuration for library facade
// Every instance of FilterListManager must have its own configuration
let mut configuration = Configuration::default();
// Sets urls for filters indices.
configuration.metadata_url = "https://filters.adtidy.org/extension/safari/filters.json".to_string();
configuration.metadata_locales_url = "https://filters.adtidy.org/extension/safari/filters_i18n.json".to_string();
// Sets locale. Will be used for returning localized strings for filters,
// groups, and tags, where applicable.
configuration.locale = "pt_PT".to_string();
// Sets type of filters lists.
// By default, FilterListType::STANDARD will be selected.
configuration.filter_list_type = FilterListType::DNS;
// Creates facade instance
let flm = FilterListManagerImpl::new(configuration);
Example references
Configuration reference
FilterListManager reference
How to create and fill up standard filters database
// Creates and configures the database. Populates the database with information
// from the filter indexes (filters metadata), the paths to which are specified
// in the configuration.
// In addition, this method applies migrations that have not yet been applied.
// See the lift_up_database method for details on "lifting" a database.
flm.pull_metadata();
// Then, downloads the contents of the filters.
flm.update_filters(true, 0, true);
[!NOTE] By default, the application operates with a database located in the current working directory (cwd), and the database file name is generated based on the format
agflm_{configuration.filter_list_type.to_string()}
. For standard filters, the file path will be$CWD/agflm_standard.db
.
Database scheme updates
Database schema updates (migrations) are possible using the flm.lift_up_database()
method.
The method “raises” the state of the database to the working state.
If the database doesn't exist:
- Creates database
- Rolls up the schema
- Rolls migrations
- Performs bootstrap.
If the database is an empty file:
- Rolls the schema
- Rolls migrations
- Performs bootstrap.
... and so on.
Usage notes
Starting with version 0.7.1
the database is “uplifted” automatically when the filter_list_manager constructor is called.
To override this behavior you need to disable it in the configuration: configuration.auto_lift_up_database = false;
.
Storage notes. Important
Database lifting
If you have disabled automatic lifting, you must invoke it yourself after each library update if you don't want to miss a migration.
SQLITE_BUSY
Error
The library ensures that when using a single FLM instance for a single database file (also, by default, a database type) in a multithreaded environment, database queries will not return SQLITE_BUSY errors.
Operations with custom filters
The library categorizes all filters into three types:
- Index Filters - Filters created by parsing the index (registry).
- Custom Filters - Filters added (and edited) by the user using the library's methods.
- Special Filters - Custom filters preconfigured by the library's scripts.
You can refer to the db constants file to check the indicators for special and custom filters.
// Installs a custom filter.
let custom_filter = flm.install_custom_filter_list(
String::from("https://example.com/custom_filter.txt"),
true, // The filter list is marked as trusted.
Some(String::from("Custom title")),
Some(String::from("Custom description"))
).unwrap();
// Edit metadata.
flm.update_custom_filter_metadata(
custom_filter.id,
String::from("new title"),
false // The filter list is marked as not trusted.
).unwrap();
// Turn on this filter.
flm.enable_filter_lists(vec![custom_filter.id], true);
// Remove this filter.
flm.delete_custom_filter_lists(vec![custom_filter.id]);
Installing a custom filter from a string instead of downloading it
let string_contents = String::from(r"
! Checksum: ecbiyIyplBZKLeNzi64pGA
...
! JS API START
#%#var AG_onLoad=function(func){if(document.readyState==="complete"||document.readyState==="interactive")func();else
...
");
flm.install_custom_filter_from_string(
String::new(), // download url
1719505304i64, // last_download_time value. Explanation: Can we update filter? Answer: (filter.last_download_time + filter.expires < now())
true, // Enabled
true, // Trusted
string_contents, // Filter body
None, // Filter title - Option<String>
None // Filter description - Option<String>
);
Save operations for custom filters rules
// Saves the structure containing the filter rules.
flm.save_custom_filter_rules(/* FilterListRules */ rules_for_new_local_custom_filter);
// You can save only disabled rules for the filter list
flm.save_disabled_rules(filter.id, /* Vec<String> */ disabled_rules_list);
Example references
Get operations
// Retrieves all filters metadata from the database **with** theirs rules.
// Returns Vec<FullFilterList>.
flm.get_full_filter_lists();
// Retrieves a filter metadata by its ID from the database **with** its rules.
// Returns Optional<FullFilterList>.
flm.get_full_filter_list_by_id(id /* FilterId */);
// Retrieves all enabled filters as ActiveRulesInfo.
flm.get_active_filters();
// Retrieves all filters metadata from the database **without** theirs rules.
// Returns Vec<StoredFilterMetadata>
flm.get_stored_filters_metadata();
// Retrieves a filter metadata by its ID from the database **without** its rules.
// Returns Optional<StoredFilterMetadata>.
flm.get_stored_filter_metadata_by_id(id /* FilterId */);
// Retrieves a list of FilterListRulesRaw by IDs.
// This method acts in the same way as the `IN` database operator. Only found entities will be returned
flm.get_filter_rules_as_strings(ids /* Vec<FilterId> */);
// Reads the rule list for a specific filter in chunks, applying exceptions from the disabled_rules list on the fly.
// The default size of the read buffer is 1 megabyte. But this size can be exceeded if a longer string appears in the list of filter rules.
// The main purpose of this method is to reduce RAM consumption when reading large size filters.
flm.save_rules_to_file_blob(id /* FilterId */, file_path: /* String or AsRef<Path> */);
/// Returns lists of disabled rules by list of filter IDs as Vec<DisabledRulesRaw>
flm.get_disabled_rules(ids, /* Vec<FilterId> */);
This example references
FullFilterList reference
StoredFilterMetadata reference
ActiveRulesInfo reference
FilterListRulesRaw reference
DisabledRulesRaw reference
Other (All) operations
Dependencies
~37–55MB
~871K SLoC