A well-known problem at Mozilla is that, while most of our desktop users run Windows, most of Mozilla’s developers do not. There are a lot of problems that result from that, but one of the most frustrating to me is that sometimes those of us that actually use Windows for development find ourselves at a disadvantage when it comes to tooling or other productivity enhancers.
In many ways this problem is also a Catch-22: People don’t want to use Windows for many reasons, but tooling is big part of the problem. OTOH, nobody is motivated to improve the tooling situation if nobody is actually going to use them.
A couple of weeks ago my frustrations with the situation boiled over when I
learned that our
Cpp unit test suite could not log symbolicated call stacks,
resulting in my filing of bug 1238305 and bug 1240605. Not only could we
not log those stacks, in many situations we could not view them in a debugger
Due to the fact that PDB files consume a large amount of disk space, we don’t keep those when building from integration or try repositories. Unfortunately they are be quite useful to have when there is a build failure. Most of our integration builds, however, do include breakpad symbols. Developers may also explicitly request symbols for their try builds.
A couple of years ago I had begun working on a WinDbg debugger extension that was tailored to Mozilla development. It had mostly bitrotted over time, but I decided to resurrect it for a new purpose: to help WinDbg* grok breakpad.
mozdbgext is the result. This extension
adds a few commands that makes Win32 debugging with breakpad a little bit easier.
The original plan was that I wanted
mozdbgext to load breakpad symbols and then
insert them into the debugger’s symbol table via the
API. Unfortunately the design of this API is not well equipped for bulk loading
of synthetic symbols: each individual symbol insertion causes the debugger to
re-sort its entire symbol table. Since
xul.dll’s quantity of symbols is in the
six-figure range, using this API to load that quantity of symbols is
prohibitively expensive. I tweeted a Microsoft PM who works on Debugging Tools
for Windows, asking if there would be any improvements there, but it sounds like
this is not going to be happening any time soon.
My original plan would have been ideal from a UX perspective: the breakpad symbols would look just like any other symbols in the debugger and could be accessed and manipulated using the same set of commands. Since synthetic symbols would not work for me in this case, I went for “Plan B:” Extension commands that are separate from, but analagous to, regular WinDbg commands.
I plan to continuously improve the commands that are available. Until I have a proper README checked in, I’ll introduce the commands here.
Loading the Extension
- Use the
Loading the Breakpad Symbols
- Extract the breakpad symbols into a directory.
- In the debugger, enter
- Note that this command will take some time to load all the relevant symbols.
Working with Breakpad Symbols
Note: You must have successfully run the
!bploadsyms command first!
As a general guide, I am attempting to name each breakpad command similarly to
the native WinDbg command, except that the command name is prefixed by
- Stack trace:
- Find nearest symbol to address:
!bpln <address>where address is specified as a hexadecimal value.
Note that there are several other commands that are “roughed-in” at this point and do not work correctly yet. Please stick to the documented commands at this time.
* When I write “WinDbg”, I am really
referring to any debugger in the Debugging Tools for Windows package,