#rss #feed #scraping #cinema

app cinema_feed

Scrape a cinema website and store the schedule into a DB, export it as RSS feed

2 releases

0.1.4 Jan 2, 2019
0.1.3 Jan 1, 2019

#49 in #scraping

GPL-2.0-or-later

17KB
310 lines

A cinema website scraper in Rust

A web crawler that gets the cinema movie list of a specific date, saves it into a DB and exposes an RSS feed for public consumption.

WARNING: this is not a crate for general use, it serves a very specific usecase: mine. At most it can be some sample code for newbies.

This is a little more than the usual Hello, world! project. It's a first contact with the Rust programming language, a.k.a I don't know what I'm doing, I just pulled it together applying a lot of google-fu and copy and paste ;-)

However, as any didactical project, it was really useful to learn a lot of things about Rust, its building toolchain, packaging, where to look for help and so on.

How does it work

This little application basically scrapes a web page, extracts some data using regular expressions, save the results into a SQLite3 DB and outputs anm RSS 2.0/Atom compliant XML file (that I will instruct my RSS feed reader to retrieve).

Limitations

HTML + regex. Enough said.

Installation and requirements

  • Tested with Rust 1.31 stable
  • Install cargo to make your life easier

Run

Example: CINEMA_URL="http://cinema.website.url" cargo run -- --date-from 2017-04-19 Optional parameters:

  • --purge-db delete (if any) local DB before starting
  • --feed-path custom RSS feed save path

Verbose debugging:

Application and every library: RUST_LOG=debug cargo run -- --date-from 2017-04-19

Applicatiion only: RUST_LOG=cinema_feed=debug cargo run -- --date-from 2017-04-19

DB module only: RUST_LOG=db=debug cargo run -- --date-from 2017-04-19

Dependencies

~60MB
~1M SLoC