View on GitHub

mock_shootout

Comparison of several Rust mocking libraries

Rust Mock Shootout!

Rust has several different mocking libraries. None is clearly superior to all
of the others. This project compares their feature sets, using an
apples-to-apples approach. Each of several dozen features is implemented with
multiple mocking libraries, if possible. The test results then show which
features work with which library.

The Contenders

Mockers
This is the oldest library in our shootout. Inspired by
GoogleMock, it has an elegant syntax, powerful sequence matching, and a rich
variety of helper methods. It requires nightly Rust, and is very sensitive to
exact compiler versions. Each release of Mockers typically only works with a
single version of rustc.
Mock_Derive
This was the first Rust mocking library that supported
automatically derive ing the Mock object from the target trait. That saves a
lot of typing! Mock_Derive is still very easy to use, though it lacks any
ability to validate method arguments. It also can’t work with generic traits,
traits with generic methods, traits defined in external crates, or multiple
traits (like T: A + B). However, it does have a few rare features, like the
ability to mock foreign free functions, or traits with static methods.
Unfortunately, it’s maintenance has fallen behind and it no longer compiles on
recent toolchains.
Galvanic-mock
This is part of a suite of testing libraries. Along with
galvanic-test and galvanic-assert, it provides a comprehensive set of testing
functionality for Rust programs. Galvanic-mock itself takes a behavior-driven
approach to testing. It tries to separate the specification from what a mock
does, from how that mock is expected to be used. It’s a good all-around
library: good feature set, good documentation, good ergonomics.
Pseudo
All of the previous libraries had one thing in common: they all
require the nightly compiler. That allows them to do some pretty cool stuff,
like tweak the language’s syntax, but it’s also inherently unstable. No code
that relies on nightly can be guaranteed to work with future compilers. Pseudo
is different. It eschews nightly-dependent features like derive so that it
can work on stable Rust. Unfortunately, that also makes it pretty verbose.
Double
Like Pseudo, Double runs on stable Rust. However, it uses a few
macros to reduce the verbosity. The feature set is pretty similar. In fact,
the entire API is eerily similar. I think one of these crates must’ve copied
from the other (perhaps they both did).
Simulacrum
This is a bit of a different beast. Whereas other mock
libraries try to provide a clean API, Simulacrum actually provides 3 different
APIs. That can be confusing at first, but the result is great power. Unusual
requirements, impossible to meet with the highest-level and most conveient API,
can be satisfied (at greater effort) with the lower-level APIs. Simulacrum also
runs on stable Rust, and manages to do it with less verbosity than Double.
Optional, nightly-dependent, support for derive is a work-in-progress.
Mock-it
Mock-it is Rust’s newest mocking library, and probably its
simplest. Its chief advantage is that its simplicity allows it to run on stable
Rust, though that’s fast becoming a less unique feature. However, the lack of a
high-level API also gives Mock-it some of the power of Simulacrum; it can mock a
struct, for example. Overall, Mock-it has few practical advantages, but it’s
a good starting point for someone looking to build something bigger.
Moctopus
Mocktopus is a bit of a different beast. Whereas every other
library focuses on mocking trait s, Mocktopus focuses on free functions. In
fact, the only way that it can mock a trait is by manually creating an
implementation of that trait, and then mocking every single function.
Mocktopus also has only the most rudimentary support for expectations. Its
advantages are that it requires very little boiler plate, it can mock struct
s and free functions, and it handles generic functions well.

Features

I evaluated about three dozen features evaluated for each library. The first
group are the essential features. These determine the library’s overall
capabilities. While they may not be important to all users, they’re considered
“essential” because a user can’t implement any of these if the library doesn’t
intrinsically support them.

The second group are the convenience features. These are features that a user
can implement in terms of other essential features, for example, “Match
constant” can be implemented by matching with a method). The lack of any of
these features shouldn’t preclude the use of a certain library, but may cause
some annoyance.

The third group of “features” are really more informational in nature:

Feature Matrix

Double Galvanic-mock Mock-it Mockers Mocktopus Pseudo Simulacrum
Essential Features
Associated types no yes yes yes no yes yes
Checkpoints no no no yes no yes yes
Consume parameters no no no yes yes no no
Consume self no yes yes yes yes yes yes
Doctest yes yes yes yes yes yes yes
External traits yes no yes yes no yes yes
Foreign no no no yes no no no
Generic methods no no no no yes no yes
Generic return no no no yes yes no yes
Generic traits no yes yes no yes yes yes
Inherited traits yes yes yes yes no yes yes
Match function yes yes no yes yes yes yes
Structs no no yes no yes yes yes
Traits yes yes yes yes no yes yes
Multiple traits no yes yes yes no yes yes
Return call with args yes yes no yes yes yes yes
Return lifetime no yes no no yes no no
Return owned no no no yes no no no
Return parameters no no no yes yes no yes
Send no no yes no no yes no
Sequence single method no no multi object no single method multi method
Static methods no no yes yes yes yes no
Times range yes no no yes no no no
Convenience Features
Derive no yes no yes yes no no
Fallback no no no no yes no no
Match combinations yes no no yes no no no
Match constant yes no yes yes no yes yes
Match operator yes yes no yes no no yes
Match pattern no no no yes no no no
Match range no no no yes no no no
Match wildcard yes yes yes yes no yes yes
Mock modules no no no no yes no no
Return call no no no no no no no
Return a constant yes yes yes yes no yes no
Return default yes no yes yes no yes no
Return panic no yes no yes no no no
Times once yes yes no yes no no yes
Times any yes yes no yes no no yes
Times n yes yes no yes no yes yes
Times never yes yes no yes no yes yes
Other
Maximum arguments 12 >= 16 12 4 >= 16 12 9
Rustc stable nightly stable nightly nightly stable stable
First release Dec-12-2017 Aug-13-2017 Mar-11-2018 Apr-6-2016 Sep-5-2017 Mar-23-2017 Dec-17-2017
Tested version 0.2.2 0.1.3 0.2.1 0.13.0 0.5.0 0.2.0 0.3.1
Current version

Feature Definitions

Associated types
Can the library mock a trait with associated types, like
Iterator?
Checkpoints
When validating sequences of method calls, can the library
create checkpoints (aka Eras)? A checkpoint divides expectations
chronologically. All expectations created before the checkpoint must be
satisfied before it, and all expectations created afterwards must be satisfied
after it.
Consume parameters
Can a mock method consume its parameters, passing them
by value to an arbitrary function? This is important, for example, to keep the
parameters from drop ping after the mocked method call.
Consume self
Can the library mock a consuming method? A consuming method
is one that takes the self parameter by value, rather than by reference.
into_* methods are a common example.
Doctest
Can the library be used in doc tests? The key difference here is
that doc tests are compiled with cfg=false.
External traits
Can the library mock a trait defined in another module or
crate?
Fallback
Can a mock object proxy certain method calls to a real object?
Foreign
Can the library mock static external functions?
Generic methods
Can the library mock traits with generic methods that
have parameterized arguments, and set expectations for those methods? For
example, a method like fn foo<T>(&self, t: T) -> u32.
Generic return
Can the library mock traits with generic methods that have
parameterized return values, and set expectations for those methods? For
example, a method like fn foo<T>(&self, x: u32) -> T.
Generic traits
Can the library mock generic traits, like
std::sync::Mutex<T>?
Inherited traits
Can the library mock inherited traits like pub trait B: A?
Match function
Can an expectation validate arguments with an arbitrary
function?
Structs
Can the library mock a concrete struct instead of just
trait s? This requires altering the module’s namespace during unit tests.
The test_double crate can do that.
Modules
Can the library mock every function in a module?
Multiple traits
Can the library create a mock that satisfies multiple traits,
so it can be passed to a function like fn foo<T: A + B>(t: T)?
Traits
Can the library create a mock object that implements a trait?
Return call with args
Can a mocked method return a value computed from the
arguments by an arbitrary function?
Return lifetime
Can a mocked method return a reference with a
non-'static lifetime?
Return owned
Can a mocked method return ownership of a value that does not
implement Clone?
Return parameters
Can a mocked method modify method arguments provided as
mutable references?
Sequence
Can the library assert that methods are called in a particular
order? This feature is implemented to different degrees by different libraries.
One library only supports validating call order on a method-by-method basis.
Another can validate the call order of different methods of the same mock
object. Only one library can validate the call order of methods of different
objects.
Send
Are mock objects Send? If not, then the library cannot mock a
trait that is.
Static methods
Can the library mock a trait that has a static method? A
static method, also called an “associated function” is one that does not receive
any form of self as a parameter. In Rust, they must be called using
Trait::function() syntax, rather than object.function(). That makes it
impossible for a mock library to set an expectation on such a method. However,
mocking such a trait is still useful for setting expectations on other methods.
Times range
Can the library expect a method to be called a variable number
of times, bounded by a range?
Derive
Can the library automatically generate the Mock struct by
derive ing on the trait? This feature can save a lot of typing, but it
invariably requires nightly Rust. It also can’t be used when mocking external
traits.
Match combinattions
Expectations can match boolean combinations of other
validators.
Match constant
Can an expectation match arguments with constants?
Match operator
Can an expectation match arguments with common operators,
like eq , gt , etc?
Match pattern
Can an expectation match enum arguments using patterns?
Match range
Can an expectation match arguments with ranges?
Match wildcard
Can an expectation match any argument?
Return call
Can an expectation return a value computed from an arbitrary
function?
Return constant
Can an expectation return a constant?
Return default
Can mocked methods automatically return the
Default::default() value?
Return panic
Can expectations panic instead of returning?
Times once
Can an expectation assert that it’s called exactly once?
Times any
Can a mocked method be called any number of times?
Times n
Can an expectation assert that it’s called an arbitrary number of
times?
Times never
Can a mocked method expect to never be called?
Maximum arguments
The maximum number of arguments for a mocked method.
Rustc
Minimum required compiler version. None of these six crates
guarantee a specific version, just “stable”, or “nightly”. Typically, the
crates that require “nightly” will only work with a narrow range of nightly
versions.
First release
Date of the first release on crates.io .

Conclusion

The best mocking library is … none of them. No one library is clearly
superior to all of the others. Every project will need to choose a different
mocking library with the features needed for that particular project. Some
projects may event need to use multiple libraries in combination. But a few
things stand out:

Discussion

Tear me a new one at the Rust Forums