Rust has three reference types!

Instead of a garbage collector, Rust offers manual control over exactly how values are stored, and when they are destroyed. This has benefits to predictability, performance, and even correctness (since it is a part of what allows Rust to avoid mutable aliasing), but the price is the complexity of that control, and the multiplicity of storage types.

In a language like, say, Python, there is only one type for storing a value: the garbage collected reference. References are always copyable and movable, always transferable between threads, etc.: there are no choices to be made. In Rust, you have options, and they each carry a tradeoff. For example, if you use an Rc<T>, you can’t send it across threads or mutate it, but if you use Box<T> you can’t clone it, and so on. And in any case, they are all incompatible: an interface expecting an Rc<T> or Box<T> can’t be easily passed an Arc<T>.

This proliferation of storage types is not unique to Rust: C++ has the same problem with its numerous ways of storing a value (T, shared_ptr<T>, unique_ptr<T>, weak_ptr<T>,T*, and yet more being proposed for standardization, such as indirect<T> and polymorphic<T>). It is obviously not reasonable to handle every single possible type. But as Herb Sutter notes in GotW #91, if you accept a reference, your code will work with every caller, because every caller can convert whatever they’re using to a reference. References are a “narrow waist”. And just as they are in C++, so they are in Rust.

This strategy of using references anywhere you can works fairly well for C++. Everything in C++ can bottom out at one of three different reference types: T&, const T&, or the rvalue reference T&&. (Okay, const T&& exists, but you can ignore it for the same reason I’m also ignoring volatile.) If your interface accepts these, it will work for ~anything, stored ~anywhere. Or even stored “nowhere”, as in a temporary.

Rust tries to bottom out at just two reference types: &T and &mut T. That is, it’s expected that no matter how you have your object, you can get a &T, or for mutable objects also a &mut T. For example, if you want to implement the indexing operator, myvalue[x], then there are exactly two traits: Index and IndexMut, which take &self and &mut self respectively. It is expected that any object you have can be reduced to one of these reference types. There is no third trait for a third reference type.

It’s really too bad, there is a third reference type: Pin<&mut T>.

Pinning for the Fjords

The short, short version: a Pin<&mut T> is an alternative to &mut T for types which must not move in memory – for example, types like struct MyWeirdStruct {a: ..., b: *mut ...}, where b might point to a. If we moved the struct, the b pointer would then be invalidated, and we don’t want to allow that in safe code. (This example is very similar to some versions of the “small string optimization”.) Syntactically, Pin<&mut T> looks like a wrapper around a &mut T, but interconversions are unsafe and unsound, and it must be used in place of &mut T. Pin also brings with it a whole parallel world of pinned pointer types: a Pin<Box<T>> is to Pin<&mut T> as Box<T> is to &mut T. (This is, by the way, incredibly cleverly designed! In a good way!)

Pin<&mut T> is not baked into the language, and as a consequence, everything wonderful about &mut T is absent for Pin<&mut T>.

Suppose I have a pinned array type, PinnedArray<T, const N: usize>. This is like a regular [T; N], except that it’s structurally pinned: given a Pin<&mut PinnedArray<T, N>> , you can obtain a Pin<&mut T> for one of the elements, because the “pinned-ness” of the array itself propagates to its elements. This is much like how, for a regular &mut [T; N], one can obtain a &mut T referring to an element. (The C++ analogue to the PinnedArray<T, N> type is std::array<T, N>, which natively supports pinning.)

pub struct PinnedArray<T, const N: usize>([T; N]);

What does the API for this new array type look like in Rust?

Well, it doesn’t look a lot like [T; N]! The Pin<&mut T> type is a parallel reference type to &mut T. You cannot convert between the two (safely, soundly). And so every language feature and every trait which supports &mut T fundamentally does not work with pinned references:

• DerefMut only supports &mut T, and mutable deref coercion fundamentally does not apply. For example, if a caller has a Box<[T; N]>, they are able to call mutating methods on it, like get_mut(). In fact, the get_mut method is actually implemented on [T], since arrays dereference to slices. However, for Pin<Box<PinnedArray<T, N>>, there is no automatic dereferencing. Callers must explicitly use x.as_mut().get_mut(). And get_mut must be implemented separately on pinned slices and pinned arrays.
• In turn, the syntactic magic around automatic reborrowing only works for &mut T (and &T), Even if x were already a pinned reference, one must still explicitly reborrow it whenever invoking methods, calling x.as_mut().get_mut(). If you fail to do this, you will discover that your pinned reference was not reborrowed, it was moved into the method call.
• As alluded to earlier, IndexMut only supports &mut T, and so we cannot use the [] operator to implement array access for our type. This is a general problem: any methods or functions you’d ever heard of that support &mut T will not support your pinned reference. Sometimes this is desirable, as with std::mem::swap – that’s the whole point of pinning! But with traits and operator overloading, probably not.
• There is a whole crate dedicated to reimplementing field projection for pinned references. There are multiple crates dedicated to safe creation of a pinned local variable. (This one is mine!)

There’s no way out of this. Rust does not bottom out at only the two reference types &T and &mut T, because we cannot reduce a pinned reference to either of these cases for any of the above use cases. Instead, Rust bottoms out at three reference types: shared references, mutable references, and pinned references. Unfortunately, the language only special-cases the first two, so pinned references do not get access to many language features you’d normally expect, often requiring substantial and tricky third-party code to get things the way you want them.

This is better than it sounds, because, really, Pin is not common. However, I work on C++/Rust interop (Crubit). In C++, every nontrivial type is pinned¹. If interop works well, and a codebase spends a lot of time talking back and forth with C++, Pin<&mut T> (or something like it) may be as common as &mut T. Ouch! Victims of our own success!

If &mut T had the ergonomics that Pin<&mut T> does, we’d fix &mut T. No language can survive its core datatypes being this inconvenient. It’s really only an acceptable interface to the extent that it’s rare. And so if Pin isn’t rare, at least in some codebases – if it really is a third core reference type – then it should be fixed.

Is this solvable?

Enumerated above are specific pain points with Pin, and each can be solved in a relatively targeted way:

1. DerefMut, and its syntactic sugar counterparts (deref coercion, automatic reborrowing, reference projection), do not support Pin<&mut T>
2. Library traits more generally accept only &mut T, with no equivalent for Pin<&mut T>

The “most obvious” fix is to actually add Pin into the language itself: instead of bottoming out pins at Pin<&mut T>, bottom out at &pin T. Add &pin T versions of every trait, including Deref. Support it in the reference syntaxes.

However, this approach does not seem to generalize. After all, aren’t there other reference-like types we’d like to support? This post is focused on pinned references, but, consider: for C++/Rust interop, some people (me! me!) will want to support, say, rvalue references, and they’ll have many of the same problems as Pin<&mut T> by virtue of not being a built-in type. And let’s not forget aliasing references, which are worse than Pin<&mut T> - for example see this proposal for a CppRef<T> and the related RFC.

The less obvious approach, then, is to actually generalize DerefMut, so that anyone defining a new reference-like type can participate in the same syntax sugar that &mut does, and even be accessible via the same traits like IndexMut. But if you ask me how that would work, I have no idea.

The least obvious approach of all – and one which I’m ashamed to admit I kind of like – is straight up adding all the reference types you could want into Rust. Sure, you want an ergonomic pinned type, so let’s add &pin. Sure, you want C++ rvalue references? Let’s add &cpp_rvalue. You want aliasing references too? Makes sense, let’s add &alias and &alias mut. This is ad hoc in the extreme (especially rvalue references, which have no justification at all except for C++ interoperability), but if it’s stupid and it works, it ain’t stupid. Right?

class MySelfReferential {
    int a_;
    // points to a_ or to the heap
    int* b_;

  public:
    // Constructs a copy at a new address.
    MySelfReferential(const MySelfReferential& other) :
        a_(other.a_),
        b_(other.b_ == &other.a_ ? &a_ : other.b_) {}
    
    // Cleans up the memory for b_ if it was a heap pointer.
    ~MySelfReferential() {if (b_ != &a_) delete b_;}
    
    ...
};