r3bl_terminal_async

Why R3BL?

R3BL TUI library & suite of apps focused on developer productivity

We are working on building command line apps in Rust which have rich text user interfaces (TUI). We want to lean into the terminal as a place of productivity, and build all kinds of awesome apps for it.

1. 🔮 Instead of just building one app, we are building a library to enable any kind of rich TUI development w/ a twist: taking concepts that work really well for the frontend mobile and web development world and re-imagining them for TUI & Rust.

   • Taking inspiration from things like React, SolidJS, Elm, iced-rs, Jetpack Compose, JSX, CSS, but making everything async (so they can be run in parallel & concurrent via Tokio).
   • Even the thread running the main event loop doesn't block since it is async.
   • Using proc macros to create DSLs to implement something inspired by CSS & JSX.

2. 🌎 We are building apps to enhance developer productivity & workflows.

   • The idea here is not to rebuild tmux in Rust (separate processes mux'd onto a single terminal window). Rather it is to build a set of integrated "apps" (or "tasks") that run in the same process that renders to one terminal window.
   • Inside of this terminal window, we can implement things like "app" switching, routing, tiling layout, stacking layout, etc. so that we can manage a lot of TUI apps (which are tightly integrated) that are running in the same process, in the same window. So you can imagine that all these "app"s have shared application state. Each "app" may also have its own local application state.
   • Here are some examples of the types of "app"s we plan to build (for which this infrastructure acts as the open source engine):
     1. Multi user text editors w/ syntax highlighting.
     2. Integrations w/ github issues.
     3. Integrations w/ calendar, email, contacts APIs.

All the crates in the r3bl-open-core repo provide lots of useful functionality to help you build TUI (text user interface) apps, along w/ general niceties & ergonomics that all Rustaceans 🦀 can enjoy 🎉.

Table of contents

• Introduction
• Changelog
• Learn how these crates are built, provide feedback
• Features
  • Pause and resume support
  • Input Editing Behavior
• Examples
• How to use this crate
  • TerminalAsync::try_new, which is the main entry point for most use cases
  • Readline overview please see the docs for this struct for details
  • Spinner::try_start
• Build this crate with Naz on YouTube
• Why another async readline crate?
  • References for blocking and thread cancellation in Rust
  • Educational references for Linux TTY and async Rust

Introduction

The r3bl_terminal_async library lets your CLI program be asynchronous and interactive without blocking the main thread. Your spawned tasks can use it to concurrently write to the display output, pause and resume it. You can also display of colorful animated spinners ⌛🌈 for long running tasks. With it, you can create beautiful, powerful, and interactive REPLs (read execute print loops) with ease.

1. Because read_line() is blocking. And there is no way to terminate an OS thread that is blocking in Rust. To do this you have to exit the process (who's thread is blocked in read_line()).

   • There is no way to get read_line() unblocked once it is blocked.
   • You can use process::exit() or panic!() to kill the entire process. This is not appealing.
   • Even if that task is wrapped in a thread::spawn() or thread::spawn_blocking(), it isn't possible to cancel or abort that thread, without cooperatively asking it to exit. To see what this type of code looks like, take a look at this.

2. Another problem is that when a thread is blocked in read_line(), and you have to display output to stdout concurrently, this poses some challenges.

   • This is because the caret is moved by read_line() and it blocks.
   • When another thread / task writes to stdout concurrently, it assumes that the caret is at row 0 of a new line.
   • This results in output that doesn't look good since it clobbers the read_line() output, which assumes that no other output will be produced, while is blocking for user input, resulting in a bad user experience.

Here is a video of the terminal_async and spinner examples in this crate, in action:

Changelog

Please check out the changelog to see how the library has evolved over time.

Learn how these crates are built, provide feedback

To learn how we built this crate, please take a look at the following resources.

• If you like consuming video content, here's our YT channel. Please consider subscribing.
• If you like consuming written content, here's our developer site. Please consider subscribing to our newsletter.
• If you have questions, please join our discord server.

Features

1. Read user input from the terminal line by line, while your program concurrently writes lines to the same terminal. One Readline instance can be used to spawn many async stdout writers (r3bl_core::SharedWriter) that can write to the terminal concurrently. For most users the TerminalAsync struct is the simplest way to use this crate. You rarely have to access the underlying Readline or r3bl_core::SharedWriter directly. But you can if you need to. r3bl_core::SharedWriter can be cloned and is thread-safe. However, there is only one instance of Readline per TerminalAsync instance.

2. Generate a spinner (indeterminate progress indicator). This spinner works concurrently with the rest of your program. When the Spinner is active it automatically pauses output from all the r3bl_core::SharedWriter instances that are associated with one Readline instance. Typically a spawned task clones its own r3bl_core::SharedWriter to generate its output. This is useful when you want to show a spinner while waiting for a long-running task to complete. Please look at the example to see this in action, by running cargo run --example terminal_async. Then type starttask1, press Enter. Then type spinner, press Enter.

3. Use tokio tracing with support for concurrent stout writes. If you choose to log to stdout then the concurrent version (r3bl_core::SharedWriter) from this crate will be used. This ensures that the concurrent output is supported even for your tracing logs to stdout.

4. You can also plug in your own terminal, like stdout, or stderr, or any other terminal that implements SendRawTerminal trait for more details.

This crate can detect when your terminal is not in interactive mode. Eg: when you pipe the output of your program to another program. In this case, the readline feature is disabled. Both the TerminalAsync and Spinner support this functionality. So if you run the examples in this crate, and pipe something into them, they won't do anything.

Here's an example:

# This will work.
cargo run --examples terminal_async

# This won't do anything. Just exits with no error.
echo "hello" | cargo run --examples terminal_async

Pause and resume support

The pause and resume functionality is implemented using:

• LineState::is_paused - Used to check if the line state is paused and affects rendering and input.
• LineState::set_paused - Use to set the paused state via the r3bl_core::SharedWriter below. This can't be called directly (outside the crate itself).
• r3bl_core::SharedWriter::line_state_control_channel_sender - Mechanism used to manipulate the paused state.

The Readline::new or TerminalAsync::try_new create a line_channel to send and receive r3bl_core::LineStateControlSignal:

1. The sender end of this channel is moved to the r3bl_core::SharedWriter. So any r3bl_core::SharedWriter can be used to send r3bl_core::LineStateControlSignals to the channel, which will be processed in the task started, just for this, in Readline::new. This is the primary mechanism to switch between pause and resume. Some helper functions are provided in TerminalAsync::pause and TerminalAsync::resume, though you can just send the signals directly to the channel's sender via the r3bl_core::SharedWriter::line_state_control_channel_sender.
2. The receiver end of this tokio::sync::mpsc::channel is moved to the task that is spawned by Readline::new. This is where the actual work is done when signals are sent via the sender (described above).

While the [Readline] is suspended, no input is possible, and only Ctrl+C and Ctrl+D are allowed to make it through, the rest of the keypresses are ignored.

See [Readline] module docs for more implementation details on this.

Input Editing Behavior

While entering text, the user can edit and navigate through the current input line with the following key bindings:

• Works on all platforms supported by crossterm.
• Full Unicode Support (Including Grapheme Clusters).
• Multiline Editing.
• In-memory History.
• Left, Right: Move cursor left/right.
• Up, Down: Scroll through input history.
• Ctrl-W: Erase the input from the cursor to the previous whitespace.
• Ctrl-U: Erase the input before the cursor.
• Ctrl-L: Clear the screen.
• Ctrl-Left / Ctrl-Right: Move to previous/next whitespace.
• Home: Jump to the start of the line.
  • When the "emacs" feature (on by default) is enabled, Ctrl-A has the same effect.
• End: Jump to the end of the line.
  • When the "emacs" feature (on by default) is enabled, Ctrl-E has the same effect.
• Ctrl-C, Ctrl-D: Send an Eof event.
• Ctrl-C: Send an Interrupt event.
• Extensible design based on crossterm's event-stream feature.

Examples

cargo run --example terminal_async
cargo run --example spinner
cargo run --example shell_async

How to use this crate

[TerminalAsync::try_new()], which is the main entry point for most use cases

1. To read user input, call [TerminalAsync::get_readline_event()].
2. You can call [TerminalAsync::clone_shared_writer()] to get a r3bl_core::SharedWriter instance that you can use to write to stdout concurrently, using std::write! or std::writeln!.
3. If you use std::writeln! then there's no need to [TerminalAsync::flush()] because the \n will flush the buffer. When there's no \n in the buffer, or you are using std::write! then you might need to call [TerminalAsync::flush()].
4. You can use the TerminalAsync::println and TerminalAsync::println_prefixed methods to easily write concurrent output to the stdout (r3bl_core::SharedWriter).
5. You can also get access to the underlying Readline via the Readline::readline field. Details on this struct are listed below. For most use cases you won't need to do this.

Readline overview (please see the docs for this struct for details)

• Structure for reading lines of input from a terminal while lines are output to the terminal concurrently. It uses dependency injection, allowing you to supply resources that can be used to:

  1. Read input from the user, typically crossterm::event::EventStream.
  2. Generate output to the raw terminal, typically std::io::Stdout.

• Terminal input is retrieved by calling [Readline::readline()], which returns each complete line of input once the user presses Enter.

• Each Readline instance is associated with one or more r3bl_core::SharedWriter instances. Lines written to an associated r3bl_core::SharedWriter are output to the raw terminal.

• Call [Readline::new()] to create a Readline instance and associated r3bl_core::SharedWriter.

• Call [Readline::readline()] (most likely in a loop) to receive a line of input from the terminal. The user entering the line can edit their input using the key bindings listed under "Input Editing" below.

• After receiving a line from the user, if you wish to add it to the history (so that the user can retrieve it while editing a later line), call [Readline::add_history_entry()].

• Lines written to the associated r3bl_core::SharedWriter while readline() is in progress will be output to the screen above the input line.

• When done, call [crate::manage_shared_writer_output::flush_internal()] to ensure that all lines written to the r3bl_core::SharedWriter are output.

[Spinner::try_start()]

This displays an indeterminate spinner while waiting for a long-running task to complete. The intention with displaying this spinner is to give the user an indication that the program is still running and hasn't hung up or become unresponsive. When other tasks produce output concurrently, this spinner's output will not be clobbered. Neither will the spinner output clobber the output from other tasks. It suspends the output from all the r3bl_core::SharedWriter instances that are associated with one Readline instance. Both the terminal_async.rs and spinner.rs examples shows this (cargo run --example terminal_async and cargo run --example spinner).

Spinners also has cancellation support. Once a spinner is started, Ctrl+C and Ctrl+D are directed to the spinner, to cancel it. Spinners can also be checked for completion or cancellation by long running tasks, to ensure that they exit as a response to user cancellation. Take a look at the examples/terminal_async.rs file to get an understanding of how to use this API.

The third change is that [TerminalAsync::try_new()] now accepts prompts that can have ANSI escape sequences in them. Here's an example of this.

    let prompt = {
        let user = "naz";
        let prompt_seg_1 = "╭".magenta().on_dark_grey().to_string();
        let prompt_seg_2 = format!("┤{user}├").magenta().on_dark_grey().to_string();
        let prompt_seg_3 = "╮".magenta().on_dark_grey().to_string();
        format!("{}{}{} ", prompt_seg_1, prompt_seg_2, prompt_seg_3)
    };
    let maybe_terminal_async = TerminalAsync::try_new(prompt.as_str()).await?;
    let Some(mut terminal_async) = maybe_terminal_async else {
        return Err(miette::miette!("Failed to create terminal").into());
    };
    Ok(())

Build this crate with Naz on YouTube

Watch the following videos to learn more about how this crate was built:

• Part 1: Why?
• Part 2: What?
• Part 3: Do the refactor and rename the crate
• Part 4: Build the spinner
• Part 5: Add color gradient animation to spinner
• Part 6: Publish the crate and overview
• Testing playlist
  • Part 1: Intro
  • Part 2: Deep dive

The following playlists are relevant to this crate:

• Build with Naz, async readline and spinner for CLI in Rust
• Build with Naz : Explore Linux TTY, process, signals w/ Rust
• Build with Naz, testing in Rust

Why another async readline crate?

This crate & repo is forked from rustyline-async. However it has mostly been rewritten and re-architected. Here are some changes made to the code:

• Rearchitect the entire crate from the ground up to operate in a totally different manner than the original. All the underlying mental models are different, and simpler. The main event loop is redone. And a task is used to monitor the line channel for communication between multiple r3bl_core::SharedWriters and the Readline, to properly support pause and resume, and other control functions.
• Drop support for all async runtimes other than tokio. Rewrite all the code for this.
• Drop crates like pin-project, thingbuf in favor of tokio. Rewrite all the code for this.
• Drop simplelog and log dependencies. Add support for tokio-tracing. Rewrite all the code for this, and add tracing_setup.rs.
• Remove all examples and create new ones to mimic a real world CLI application.
• Add spinner_impl, readline_impl, and public_api modules.
• Add tests.

References for blocking and thread cancellation in Rust

• Docs: tokio's stdin
• Discussion: Stopping a thread in Rust
• Discussion: Support for Thread::cancel()
• Discussion: stdin, stdout redirection for spawned processes

Educational references for Linux TTY and async Rust

• Linux TTY and async Rust - Article on developerlife.com
• Linux TTY and async Rust - Playlist on developerlife.com YT channel

License: Apache-2.0