In my continuing efforts to get caught up on discussing my work, I am now commencing a roundup for 2019. I think I am going to structure this one slightly differently from the last one: I am going to try to segment this roundup by project.
Here is an index of all the entries in this series:
- Part 1 - Porting the DLL Interceptor to AArch64 (this post)
Porting the DLL Interceptor to AArch64
During early 2019, Mozilla was working to port Firefox to run on the new AArch64 builds of Windows. At our December 2018 all-hands, I brought up the necessity of including the DLL Interceptor in our porting efforts. Since no deed goes unpunished, I was put in charge of doing the work! [I’m actually kidding here; this project was right up my alley and I was happy to do it! – Aaron]
Before continuing, you might want to review my previous entry describing the Great Interceptor Refactoring of 2018, as this post revisits some of the concepts introduced there.
Let us review some DLL Interceptor terminology:
- The target function is the function we want to hook (Note that this is a distinct concept from a branch target, which is also discussed in this post);
- The hook function is our function that we want the intercepted target function to invoke;
- The trampoline is a small chunk of executable code generated by the DLL interceptor that facilitates calling the target function’s original implementation.
On more than one occasion I had to field questions about why this work was even necessary for AArch64: there aren’t going to be many injected DLLs in a Win32 ecosystem running on a shiny new processor architecture! In fact, the DLL Interceptor is used for more than just facilitating the blocking of injected DLLs; we also use it for other purposes.
Not all of this work was done in one bug: some tasks were more urgent than others. I began this project by enumerating our extant uses of the interceptor to determine which instances were relevant to the new AArch64 port. I threw a record of each instance into a colour-coded spreadsheet, which proved to be very useful for tracking progress: Reds were “must fix” instances, yellows were “nice to have” instances, and greens were “fixed” instances. Coordinating with the milestones laid out by program management, I was able to assign each instance to a bucket which would help determine a total ordering for the various fixes. I landed the first set of changes in bug 1526383, and the second set in bug 1532470.
It was now time to sit down, download some AArch64 programming manuals, and take a look at what I was dealing with. While I have been messing around with x86 assembly since I was a teenager, my first exposure to RISC architectures was via the DLX architecture introduced by Hennessy and Patterson in their textbooks. While DLX was crafted specifically for educational purposes, it served for me as a great point of reference. When I was a student taking CS 241 at the University of Waterloo, we had to write a toy compiler that generated DLX code. That experience ended up saving me a lot of time when looking into AArch64! While the latter is definitely more sophisticated, I could clearly recognize analogs between the two architectures.
In some ways, targeting a RISC architecture greatly simplifies things: The DLL Interceptor only needs to concern itself with a small subset of the AArch64 instruction set: loads and branches. In fact, the DLL Interceptor’s AArch64 disassembler only looks for nine distinct instructions! As a bonus, since the instruction length is fixed, we can easily copy over verbatim any instructions that are not loads or branches!
On the other hand, one thing that increased complexity of the port is that some branch instructions to relative addresses have maximum offsets. If we must branch farther than that maximum, we must take alternate measures. For example, in AArch64, an unconditional branch with an immediate offset must land in the range of ±128 MiB from the current program counter.
Why is this a problem, you ask? Well, Detours-style interception must overwrite
the first several instructions of the target function. To write an absolute jump,
we require at least 16 bytes: 4 for an
LDR instruction, 4 for a
instruction, and another 8 for the 64-bit absolute branch target address.
Unfortunately, target functions may be really short! Some of the target functions that we need to patch consist only of a single 4-byte instruction!
In this case, our only option for patching the target is to use an immediate
instruction, but that only works if our hook function falls within that ±128MiB
limit. If it does not, we need to construct a veneer. A veneer is a special
trampoline whose location falls within the target range of a branch instruction.
Its sole purpose is to provide an unconditional jump to the “real” desired
branch target that lies outside of the range of the original branch. Using
veneers, we can successfully hook a target function even if it is only one
instruction (ie, 4 bytes) in length, and the hook function lies more than 128MiB
away from it. The AArch64 Procedure Call Standard specifies
X16 as a volatile
register that is explicitly intended for use by veneers: veneers load an
absolute target address into
X16 (without needing to worry about whether or
not they’re clobbering anything), and then unconditionally jump to it.
Measuring Target Function Instruction Length
To determine how many instructions the target function has for us to work with, we make two passes over the target function’s code. The first pass simply counts how many instructions are available for patching (up to the 4 instruction maximum needed for absolute branches; we don’t really care beyond that).
The second pass actually populates the trampoline, builds the veneer (if necessary), and patches the target function.
Since the DLL interceptor is already well-equipped to build trampolines, it did not take much effort to add support for constructing veneers. However, where to write out a veneer is just as important as what to write to a veneer.
Recall that we need our veneer to reside within ±128 MiB of an immediate branch. Therefore, we need to be able to exercise some control over where the trampoline memory for veneers is allocated. Until this point, our trampoline allocator had no need to care about this; I had to add this capability.
Adding Range-Aware VM Allocation
Firstly, I needed to make the
MMPolicy classes range-aware: we need to be able
to allocate trampoline space within acceptable distances from branch instructions.
Consider that, as described above, a branch instruction may have limits on the extents of its target. As data, this is easily formatted as a pivot (ie, the PC at the location where the branch instruction is encoutered), and a maximum distance in either direction from that pivot.
On the other hand, range-constrained memory allocation tends to work in terms
of lower and upper bounds. I wrote a conversion method,
to convert between the two formats. In addition to format conversion, this method
also constrains resulting bounds such that they are above the 1MiB mark of the
process’ address space (to avoid reserving memory in VM regions that are
sensitive to compatiblity concerns), as well as below the maximum allowable
user-mode VM address.
Another issue with range-aware VM allocation is determining the location, within
the allowable range, for the actual VM reservation. Ideally we would like the
kernel’s memory manager to choose the best location for us: its holistic view of
existing VM layout (not to mention ASLR) across all processes will provide
superior VM reservations. On the other hand, the Win32 APIs that facilitate this
are specific to Windows 10. When available,
When we’re running on Windows versions where those APIs are not yet available,
we need to fall back to finding and reserving our own range. The
MMPolicyBase::FindRegion method handles this for us.
All of this logic is wrapped up in the
MMPolicyBase::Reserve method. In
addition to the desired VM size and range, the method also accepts two functors
that wrap the OS APIs for reserving VM.
Reserve uses those functors when
available, otherwise it falls back to
FindRegion to manually locate a suitable
Now that our memory management primatives were range-aware, I needed to shift my focus over to our VM sharing policies.
One impetus for the Great Interceptor Refactoring was to enable separate
Interceptor instances to share a unified pool of VM for trampoline memory.
To make this range-aware, I needed to make some additional changes to
VMSharingPolicyShared. It would no longer be sufficient to assume that we
could just share a single block of trampoline VM — we now needed to make the
shared VM policy capable of potentially allocating multiple blocks of VM.
VMSharingPolicyShared now contains a mapping of ranges to VM blocks. If we
request a reservation which an existing block satisfies, we re-use that block.
On the other hand, if we require a range that is yet unsatisfied, then we need to
allocate a new one. I admit that I kind of half-assed the implementation of the
data structure we use for the mapping; I was too lazy to implement a fully-fledged
interval tree. The current implementation is probably “good enough,” however
it’s probably worth fixing at some point.
Finally, I added a new generic class,
TrampolinePool, that acts as an
abstraction of a reserved block of VM address space. The main interceptor code
requests a pool by calling the VM sharing policy’s
Reserve method, then it
uses the pool to retrieve new
Trampoline instances to be populated.
It is much simpler to generate trampolines for AArch64 than it is for x86(-64).
The most noteworthy addition to the
Trampoline class is the
method, which writes an absolute address into the trampoline’s literal pool,
followed by writing an
LDR instruction referencing that literal into the
Thanks for reading! Coming up next time: My Untrusted Modules Opus.