Introduction

Papyrus - A rust REPL and script running tool.

This project is no longer maintained.

For a similar REPL tool with a refreshing way to interact with data, checkout the ogma Project

See the rs docs and the guide. Look at progress and contribute on github.

papyrus=> 2+2
papyrus [out0]: 4

Overview

Papyrus creates a Rust REPL in your terminal. Code can be typed in, line by line with feedback on the evaluation, or code can be injected via stdin handles. Each code snippet is evaluated on an expression based system, so terminating with a semi-colon requires more input.

Example

[lib] papyrus=> 2+2
papyrus [out0]: 4
[lib] papyrus=> println!("Hello, world!");
[lib] papyrus.> out0 * out0
Hello, world!
papyrus [out1]: 16
[lib] papyrus=> :help
help -- prints the help messages
cancel | c -- returns to the root class
exit -- sends the exit signal to end the interactive loop
Classes:
  edit -- Edit previous input
  mod -- Handle modules
Actions:
  mut -- Begin a mutable block of code
[lib] papyrus=> :exit
[lib] papyrus=> Thanks for using papyrus!

Installation

Papyrus can be installed from crates.io or building from source on github. The default installation feature set requires a nightly toolchain, but stable can be used with fewer features enabled.

To install with all features:

rustup toolchain add nightly
cargo +nightly install papyrus

To install on stable without racer completion:

cargo +stable install papyrus --no-default-features --features="format,runnable"

Requirements

Features

Papyrus has features sets:

format: format code snippets using rustfmt
racer-completion: enable code completion using racer. Requires a nightly compiler
runnable: papyrus can be run, without needing to manually handle repl states and output

All features are enabled by default.

Cargo

Papyrus leverages installed binaries of both cargo and rustc. This requirement may lift in the future but for now, any user wanting to use Papyrus will need an installation of Rust.

API

papyrus offers a broad API to enable embedding and customisation. Each module which provides this functionality has documentation and there are examples that can be used as a guide.

Code

Source file and crate contents.

Input is parsed as Rust code using the syn crate. papyrus does not differentiate the myriad of classications for the input, rather it categorises them into Items, Statements, and CrateTypes.

papyrus will parse a string input into a Input, and these aggregate into a SourceCode structure, which flattens each input.

Examples

Building some source code.


#![allow(unused)]
fn main() {
extern crate papyrus;
use papyrus::code::*;

let mut src = SourceCode::default();
src.stmts.push(StmtGrp(vec![Statement {
    expr: String::from("let a = 1"),
        semi: true
    },
    Statement {
        expr: String::from("a"),
        semi: false
    }
]));
}

Crates have some more structure around them.


#![allow(unused)]
fn main() {
extern crate papyrus;
use papyrus::code::*;

let input = "extern crate a_crate as acrate;";
let cr = CrateType::parse_str(input).unwrap();

assert_eq!(&cr.src_line, input);
assert_eq!(&cr.cargo_name, "a-crate");
}

Cmds

Extendable commands for REPL.

The REPL makes use of the crate cmdtree to handle commands that can provide additional functionality over just a Rust REPL. A command is prefixed by a colon (:) and a number of defaults. To see the commands that are included, type :help.

Common Commands

There are three common commands, help, cancel or c, and exit, which can be invoked in any class.

cmd	action
`help`	displays help information for the current class
`cancel`	moves the class back to the command root
`exit`	quit the REPL

Other commands are context based off the command tree, they can be invoked with something similar to a nested command action syntax. There is also a 'verbatim' mode.

Verbatim Mode

'verbatim' mode can be used to read stdin directly without processing of tabs, newlines, and other keys which usually are interpreted to mean other things, such as completion or the end of an input. To enter verbatim mode, use Ctrl+o. Verbatim mode will read stdin until the break command is reached, which is Ctrl+d. Verbatim mode is especially useful for inputing multi-line strings and if injecting code into the REPL from another program using stdin.

Mutable Mode

The mut command will place the REPL into mutable mode, which makes access to app_data a &mut pointer. Mutable mode avoids having state change on each REPL cycle, rather, when in mutable mode, the expression will not be saved such that it will only be run once. Developers can use this mode to control how changes to app_data need to occur, especially by ensuring mutable access is harding to achieve.

Modules

The mod command allows more than just the lib module to exist in the REPL. Use mod to have different REPL sessions all sharing the same compilation cycle. This can be useful to switch contexts if need be.

There is also a clear command which can be used to clear the previous REPL inputs. It supports glob patterns matching module paths, for example :mod clear test/** will clear all inputs under the module path test/. :mod clear clears all previous REPL input in the current module.

Static Files

The static-files command allows the importing of file-system based rust documents into the REPL compilation. Rust files must be relative to the REPL working directory, and will be imported using a module path based off the relative file name. For example, :static-files add foo.rs will copy the contents of foo.rs into a file that is adjacent to the root library file, so can be access in the REPL through foo::*. The file foo/mod.rs would similarly be accessible through foo::*. If the file foo/bar.rs was imported, it would not be accessible unless there was also a foo static file, and the contents of that file would have to contain a pub mod bar;.

Static files can also reference crates. If a static file contains extern crate name; at the beginning of the file, these crates are added to the compilation and can be referenced.

To add static files, it is possible to use glob patterns to add multiple files in one go. For example to add all files in the working directory the command :static-files add *.rs can be used. To recursively add files **/*.rs can be used. This applies to removing static files using the rm command.

Extending Commands

Setup

This tutorial works through the example at papyrus/examples/custom-cmds.rs.

To begin, start a binary project with the following scaffolding in the main source code. We define a custom_cmds function that will be used to build our custom commands. To highlight the versatility of commands, the REPL is configured to have a persistent app data through a String. Notice also the method to alter the prompt name through the Builder::new method.

#[macro_use]
extern crate papyrus;

use papyrus::cmdtree::{Builder, BuilderChain};
use papyrus::cmds::CommandResult;

#[cfg(not(feature = "runnable"))]
fn main() {}

#[cfg(feature = "runnable")]
fn main() {
    // Build a REPL that will use a String as the persistent app_data.
    let mut repl = repl!(String);

    // Inject our custom commands.
    repl.data.with_cmdtree_builder(custom_cmds()).unwrap();

    // Create the persistent data.
    let mut app_data = String::new();

    // Run the REPL and collect all the output.
    let output = repl.run(papyrus::run::RunCallbacks::new(&mut app_data)).unwrap();

    // Print the output.
    println!("{}", output);
}

// Define our custom commands.
// The CommandResult takes the same type as the app_data,
// in this instance it is a String. We could define it as
// a generic type but then it loses resolution to interact with
// the app_data through commands.
fn custom_cmds() -> Builder<CommandResult<String>> {
    // The string defines the name and the prompt that will be used.
    Builder::new("custom-cmds-app")
}

Echo

Let's begin with a simple echo command. This command takes the data after the command and prints it to screen. All these commands will be additions to the Builder::new. Adding the following action with add_action method, the arguments are written to the Writeable writer. The REPL provides the writer and so captures the output. args is passed through as a slice of string slices, cmdtree provides this, and are always split on word boundaries. Finally, CommandResult::Empty is returned which papyrus further processes. Empty won't do anything but the API provides alternatives.


#![allow(unused)]
fn main() {
extern crate papyrus;
use papyrus::cmdtree::BuilderChain;
use papyrus::cmds::CommandResult;
type Builder = papyrus::cmdtree::Builder<CommandResult<String>>;
Builder::new("custom-cmds-app")
    .add_action("echo", "repeat back input after command", |writer, args| {
    writeln!(writer, "{}", args.join(" ")).ok();
    CommandResult::Empty
    })
    .unwrap()
;
}

Now when the binary is run the REPL runs as usual. If :help is entered you should see the following output.

[lib] custom-cmds-app=> :help
help -- prints the help messages
cancel | c -- returns to the root class
exit -- sends the exit signal to end the interactive loop
Classes:
    edit -- Edit previous input
    mod -- Handle modules
Actions:
    echo -- repeat back input after command
    mut -- Begin a mutable block of code
[lib] custom-cmds-app=>

The echo command exists as a root level action, with the help message displayed. Try calling :echo Hello, world! and see what it does!

Alter app data

To extend what the commands can do, lets create a command set that can convert the persistent app data case. The actual actions are nested under a 'class' named case. This means to invoke the action, one would call it through :case upper or :case lower.


#![allow(unused)]
fn main() {
extern crate papyrus;
use papyrus::cmdtree::BuilderChain;
use papyrus::cmds::CommandResult;
type Builder = papyrus::cmdtree::Builder<CommandResult<String>>;
Builder::new("custom-cmds-app")
    .add_action("echo", "repeat back input after command", |writer, args| {
    writeln!(writer, "{}", args.join(" ")).ok();
    CommandResult::Empty
    })
    .begin_class("case", "change case of app_data")
    .add_action("upper", "make app_data uppercase", |_, _|
    CommandResult::<String>::app_data_fn(|app_data, _repldata, _| {
        *app_data = app_data.to_uppercase();
        String::new()
        })
    )
        .add_action("lower", "make app_data lowercase", |_, _|
    CommandResult::<String>::app_data_fn(|app_data, _repldata, _| {
        *app_data = app_data.to_lowercase();
        String::new()
        })
    )
    .end_class()
    .unwrap()
;
}

An example output is below. To inject some data into the persistent app data, a mutable code block must be entered first.

[lib] papyrus=> :mut
beginning mut block
[lib] custom-cmds-app-mut=> app_data.push_str("Hello, world!")
finished mutating block: ()
[lib] custom-cmds-app=> app_data.as_str()
custom-cmds-app [out0]: "Hello, world!"
[lib] custom-cmds-app=> :case upper
[lib] custom-cmds-app=> app_data.as_str()
custom-cmds-app [out1]: "HELLO, WORLD!"
[lib] custom-cmds-app=> :case lower
[lib] custom-cmds-app=> app_data.as_str()
custom-cmds-app [out2]: "hello, world!"

Output

Reading and writing output.

The Repl provides mechanisms to handle the output that is produced when operating. An Output instance is maintained in the Repl. This output is vector of lines of output. There are also mechanisms on the Repl to listen to changes on the Output, such that changes can be synchronised without needing to diff the output. This is useful in longer running operations where displaying the output progressively is required.

Examples

Stdio

Stdio could be construed as the more simple case, but actually entails some more complexity due to the dual nature of input and output in the terminal, whereas if input is separated from the output rendering the separated rendering example can be implemented.

This tutorial works through the example at papyrus/examples/output-stdio.rs.

First start an empty binary project with two extra dependencies: term_cursor and term_size.

#[macro_use]
extern crate papyrus;
extern crate term_cursor;
extern crate term_size;

use papyrus::output::{OutputChange, Receiver};
use papyrus::repl::{EvalResult, ReadResult, Signal};
use std::io::{stdin, stdout, StdoutLock, Write};

Before defining the main loop, lets define some functions that will be used. The first one is one which erases the current console line. It does this by moving to the first column in the terminal, writing a line of spaces (' '), and then moving again to the first column. This is where the dependencies are required.

/// Erases the current console line by moving to first column,
/// writing a row of spaces ' ', then setting cursor back
/// to first column.
fn erase_console_line(stdout: &mut StdoutLock) {
    use term_cursor as cursor;
    let lineidx = cursor::get_pos().map(|x| x.1).unwrap_or_default();

    let width = term_size::dimensions().map(|(w, _)| w).unwrap_or_default();

    cursor::set_pos(0, lineidx).unwrap();
    (0..width).for_each(|_| write!(stdout, " ").unwrap());
    cursor::set_pos(0, lineidx).unwrap();
}

Next we define a simple fuction to read a line from stdin and return a string.

/// Waits for a line input from stdin.
fn read_line() -> String {
    let mut buf = String::new();
    stdin().read_line(&mut buf).unwrap();
    buf
}

And finally define a function that handles the line changes. This function does its work on a separate thread to run asynchronously while the repl is evaluating. It receives each line change and writes it to stdout, erasing a current line, or writing a new line if required.

/// This handles the writing of output changes to stdout.
fn write_output_to_stdout(rx: Receiver) -> std::thread::JoinHandle<()> {
    std::thread::spawn(move || {
        let mut stdout = std::io::stdout();

        for chg in rx.iter() {
            match chg {
                OutputChange::CurrentLine(line) => {
                    let mut lock = stdout.lock();
                    erase_console_line(&mut lock);
                    write!(&mut lock, "{}", line).unwrap();
                    lock.flush().unwrap();
                }
                OutputChange::NewLine => writeln!(&mut stdout, "").unwrap(),
            }
        }
    })
}

These functions can be used in a main function that handles the repl states.

// build the repl
let repl = repl!();

// alias the state in the variable name.
let mut read = repl;

loop {
    // write the prompt, erase line first.
    {
        let stdout = stdout();
        let mut lock = stdout.lock();
        erase_console_line(&mut lock);
        write!(&mut lock, "{}", read.prompt()).unwrap();
        lock.flush().unwrap();
    }

    // read line from stdin
    let line_input = read_line();
        
    // set this as the repl input
    read.line_input(&line_input);
    
    // handle the input and get a result from it
    let read_res = read.read();

    match read_res {
        ReadResult::Read(repl) => {
            // The repl is still in a read state, continue reading
            read = repl;
        }
        ReadResult::Eval(mut eval) => {
           // The repl is ready for evaluating
        
           // as we want to update as input comes in,
           // we need to listen to output changes
           let rx = eval.output_listen();

           // start the output thread
           let output_thread_jh = write_output_to_stdout(rx);

           // evaluate using a unit value for data
           let EvalResult { repl, signal } = eval.eval(&mut ());

           // handle the signal, other values are elided but would be
           // handled in a more complete implementation
           match signal {
               Signal::Exit => break,
               _ => (),
           }

           let (mut repl, _) = repl.print();

           // we have printed everything it is time to close the channel
           // it is worth testing what happens if you don't, and it should
           // highlight the reason for requiring the listening channels.
           repl.close_channel();

           // we wait for the output thread to finish to let it write out
           // the remaining lines
           output_thread_jh.join().unwrap();

           read = repl;
        }
    }
}

Separated Rendering

To reduce the complexity of rendering, the output listener and listen to all changes, which includes any input changes, it will get updated on calls to Repl.line_input(). This example shows a naive implementation which writes to a file as it goes.

This tutorial works through the example at papyrus/examples/output-file.rs.

First start an empty binary project:

#[macro_use]
extern crate papyrus;

use papyrus::output::{OutputChange, Receiver};
use papyrus::repl::{EvalResult, ReadResult, Signal};
use std::io::{stdin, StdoutLock, Write};

As before there is a read_line function, and we alter the output function somewhat.

/// Waits for a line input from stdin.
fn read_line() -> String {
    let mut buf = String::new();
    stdin().read_line(&mut buf).unwrap();
    buf
}

/// Write output to a file as you go. Not a very efficient implementation.
fn write_outoput_to_file(rx: Receiver) -> std::thread::JoinHandle<()> {
    std::thread::spawn(move || {
        let mut output = String::new();
        let mut pos = 0;

        for chg in rx.iter() {
            match chg {
                OutputChange::CurrentLine(line) => {
                    output.truncate(pos);
                    output.push_str(&line);
                    std::fs::write("repl-output.txt", &output).unwrap();
                }
                OutputChange::NewLine => {
                    output.push('\n');
                    pos = output.len();
                }
            }
        }
    })
}

These functions can be used in a main function that handles the repl states.

// build the repl
let repl = repl!();

// alias the state in the variable name.
let mut read = repl;

// as we want to update as input comes in,
// we need to listen to output changes
let rx = read.output_listen();

// start the output thread
write_outoput_to_file(rx);

loop {
    // read line from stdin
    let line_input = read_line();

    // set this as the repl input
    read.line_input(&line_input);

    // handle the input and get a result from it
    let read_res = read.read();

    read = match read_res {
        ReadResult::Read(repl) => {
            // The repl is still in a read state, continue reading
            repl
        }
        ReadResult::Eval(eval) => {
            // The repl is ready for evaluating

            // evaluate using a unit value for data
            let EvalResult { repl, signal } = eval.eval(&mut ());

            // handle the signal, other values are elided but would be
            // handled in a more complete implementation
            match signal {
                Signal::Exit => break,
                _ => (),
            }

            let (repl, _) = repl.print();
            repl
        }
    }
}

Linking

Linking an external crate and sharing data.

When running a REPL you might want to link an external crate. The specific use case is a developer wants to link the crate they are working on into the REPL for the user to be able to use. A developer might also want to make data available to the REPL. Papyrus has this functionality but makes some assumptions that the developer will need to be aware of, detailed below.

Worked Example

A REPL instance should always be created by invoking the macro repl!(). In the examples below this will be elided for as the documentation won't compile with the macros. The macro accepts a type ascription (such as u32, String, MyStruct, etc.) which defines the generic data constraint of the REPL. When an evaluation call is made, a mutable reference of the same type will be required to be passed through. Papyrus uses this data to pass it (across an FFI boundary) for the REPL to access.

To show the functionality of linking, let's work on a crate called some-lib.

File Setup

main.rs:

#[macro_use]
extern crate papyrus;

use papyrus::prelude::*;

#[cfg(not(feature = "runnable"))]
fn main() {}

#[cfg(feature = "runnable")]
fn main() {
  let mut repl = repl!();

  let d = &mut ();

  repl.run(papyrus::run::RunCallbacks::new(d));
}

lib.rs:


#![allow(unused)]
fn main() {
pub struct MyStruct {
  pub a: i32,
  pub b: i32,
}

impl MyStruct {
  pub fn new(a: i32, b: i32) -> Self {
    MyStruct { a, b }
  }

  pub fn add_contents(&self) -> i32 {
    self.a + self.b
  }
}
}

Cargo.toml:

[package]
name = "some-lib"

...

[lib]
name = "some_lib"
crate-type = ["rlib" ]
path = "src/lib.rs" # you may need path to the library

[dependencies]
papyrus = { version = "*", crate-type = [ "rlib" ] }
...

Notice that you will have to specify the library with a certain crate-type. Papyrus links using an rlib file, but it is shown that you can also build multiple library files. If you build this project you should find a libsome_lib.rlib sitting in your build directory. Papyrus uses this to link when compiling. The papyrus dependency also requires a crate-type specification. If not specified, references to papyrus in the library will cause compilation errors when running the REPL.

REPL

Run this project (cargo run). It should spool up fine and prompt you with papyrus=>. Now you can try to use the linked crate.

papyrus=> some_lib::MyStruct::new(20, 30).add_contents()
papyrus [out0]: 50

Behind the scenes

Papyrus takes the crate name you specify and will add this as extern crate CRATE_NAME; to the source file.
When setting the external crate name, the rlib library is found and copied into the compilation directory.
- Papyrus uses std::env::current_exe() to find the executing folder, and searches for the rlib file in that folder (libCRATE_NAME.rlib)
- Specify the path to the rlib library if it is located in a different folder
When compiling the REPL code, a rustc flag is set, linking the rlib such that extern crate CRATE_NAME; works.

Passing `MyStruct` data through

Keep the example before, but alter the main.rs file.

main.rs:

#[macro_use]
extern crate papyrus;
extern crate some_lib;

use some_lib::MyStruct;

#[cfg(not(feature = "runnable"))]
fn main() {}

#[cfg(feature = "runnable")]
fn main() {
  let mut app_data = MyStruct::new(20, 10);

  let mut repl = repl!(some_lib::MyStruct);

  repl.data = repl
    .data
    .with_extern_crate("some_lib", None)
    .expect("failed creating repl data");

  repl.run(&mut app_data);
}

Run this project (cargo run). It should spool up fine and prompt you with papyrus=>. Now you can try to use the linked data. The linked data is in a variable app_data. It is borrowed or mutably borrowed depending on the REPL state.

papyrus=> app_data.add_contents()
papyrus [out0]: 50

Notes

Panics

To avoid crashing the application on a panic, catch_unwind is employed. This function requires data that crosses the boundary be UnwindSafe, making & and &mut not valid data types. Papyrus uses AssertUnwindSafe wrappers to make this work, however it makes app_data vulnerable to breaking invariant states if a panic is triggered.

The developer should keep this in mind when implementing a linked REPL. Some guidelines:

Keep the app_data that is being transfered simple.
Develop wrappers that only pass through a clone of the data.

Dependency Duplication

When linking an external library, the deps folder is linked to ensure that the dependencies that the library is built with link properly. There are specific use cases where the rust compiler will be unable to determine what dependencies to use. This happens when:

The library has a dependency depx
The REPL is asked to use a dependency depx
The library and REPL both use the exact same dependency structure for depx
- This means that depx is the same version, and has the same feature set enabled
The library and REPL both use the dependency in code

As an example, the use of the rand crate might cause compilation issues to arise if the linked external library also relies of rand. The exact cause is having both crates in the dependency graph that rustc cannot discern between. The compilation error is however a good indication that the external library needs to be supplying these transitive dependencies for the REPL's use, as the REPL is really using the external library as a dependency (just in an indirect manner). Usually an error message such as error[E0523]: found two different crates with name rand that are not distinguished by differing -C metadata. This will result in symbol conflicts between the two. would be encountered.

To solve this issue, any REPL dependency that could overlap with a library dependency be exposed by the library itself. This can be done by using pub use depx; or pub extern crate depx; in the root of the library source. Then, alter the persistent_module_code on the linking configuration to include a statement such as use external_lib::depx; where the external lib is your library name. If you library had the name awesome and you wanted to expose the rand crate you would add use awesome::rand; to the persistent_module_code (make sure to test for whitespace and add if necessary). There is access to the persistent_module_code through the ReplData.

Adding this code effectively aliases the library dependency as if it was a root dependency of the REPL. This trick is especially important if one is linking a library that makes use of the kserd crate and has implemented ToKserd so data types can automatically be transferred across the REPL boundary. The REPL needs to not use the kserd dependency it is using and use the kserd dependency from the external library. Using use external_lib::kserd; will manage this.

This is also important as then if the user of the REPL wants to implement ToKserd on REPL types, it will still be using the consistent kserd dependency, although an astute user might try to implement ::kserd::ToKserd which would break! At least at this point it is easy to back out changes in the temporary REPL session.

papyrus book