I’m enormously pleased about this because for the last two and a half years I’ve been working as a Software Development Engineer in Test (SDET) on the Windows Fundamentals Reliability team. I joined the team (and Microsoft) right as Windows 7 development was getting underway and have been immersed in our work on it ever since.

Throughout the development cycle the Reliability team was responsible for driving reliability improvements across Windows as a whole. We tackled this with a range of approaches, one of which was implementing new reliability-related features within Windows. I owned the testing responsibilities for three of these. All three are low-level infrastructure features that won’t be mentioned in an ad campaign, but each has been given some level of exposure as Windows 7 architecture has been  discussed and reviewed in the run-up to its release.

RADAR – Automatic memory leak detection

RADAR automatically detects memory leaks in processes running on a Windows 7 system. Most of the public documentation about this feature is an interview that three of my colleagues and I gave to MSDN’s Channel 9. We got to dive into RADAR’s architecture and the impact it has had on Windows 7 throughout its development. The C9 page for the interview has a nice summary of RADAR:

RADAR is a memory leak detection technology built into Windows 7 and integrated with Watson (error reporting) and AutoBug (automatic bug filing). It allows Microsoft product teams and third parties to discover and fix memory leaks early in the product cycle and after release. Since RADAR runs on customer machines, leaks can be caught during public betas, after release, and by third parties, thus ridding the entire ecosystem of memory leaks. RADAR-shipped components are highly optimized to have no appreciable performance impact.

The talk is archived here: Inside Windows 7: RADAR – Windows Automatic Memory Leak Detection.

Fault Tolerant Heap (FTH)

Fault Tolerant Heap (FTH) was, in a lot of ways, the most fascinating of all three features to work on. FTH monitors application crashes on a Windows 7 system. When it thinks it can help FTH applies mitigations on a per-application basis to try to prevent future crashes. The development lead for the project, Silviu Calinoiu, did a great interview with C9 that went into detail about how exactly FTH goes about its crash prevention.

Silviu’s talk is archived here: Silviu Calinoiu: Inside Windows 7 – Fault Tolerant Heap.

One potential complication for FTH is that developers who are working on a new app may not want FTH’s help – they want their test app to crash so that they can find the problem and fix it. MSDN has a page that discusses how devs can check whether FTH has been enabled for a particular app and, if they choose, disable FTH entirely on a test machine. See here: Fault Tolerant Heap (Windows).

Process Reflection

Process reflection is a way for a monitor application to clone, or reflect, some target process. Once the operation is complete the clone, or reflection, process has a complete copy of the original target process’s address space which can then be analyzed or dumped to disk for later analysis. Even better, the target process doesn’t have to be suspended or terminated to be reflected which makes process reflection tremendously useful for gathering data from long-running processes like background services or the user shell. Mark Russinovich briefly discusses process reflection and shows a nice demo of it in action in another C9 talk that just went up today: Mark Russinovich- Inside Windows 7 Redux. He starts talking about process reflection at around the 31:00 mark. He also updated his SysInternals procdump tool to optionally use reflection for creating dumps. The updated procdump is available here.

It’s been a tremendously satisfying and humbling two and a half years working on Windows 7 together with a great set of colleagues from all over the world. I can’t wait to see what the next two and a half years will bring!

I attended several days’ worth of presentations and roundtables across multiple tracks. While I learned a lot in the presentations, the thing I’ve spent the most time thinking about was actually a free handout I got on the morning of the first day.

The forum event staff printed up three different styles of notepad to distribute to attendees—one each for developers, testers, and program managers. Each notepad page was bordered by a series of short sayings related to the target discipline. As a tester I picked up the test notepad and then spent the first several minute of the morning session distractedly reading around the edges of the page. I don’t know where the sayings came from originally, or who wrote them, but the sayings on the notepad have proven to be a lot of fun to ponder.

Each saying is short and highly domain-specific. I think of them as test aphorisms—pithy sayings to be unpacked and thought about from many directions. They each make some sort of claim about the right way to test, but sometimes it can be hard to see what justifies the claim, or even what the claim is to begin with.

The first four test aphorisms are as follows. I have interspersed some initial questions to help direct my thinking for each in the future.

1. Trust, but verify.
   1. What are we trusting?
   2. What constitutes verification?
2. Test functionality and the scenario.
   1. What constitutes testing functionality or testing a scenario in this context?
   2. What distinguishes one from the other?
   3. Why is that distinction important?
3. Test the design.
   1. How does one test a design?
   2. Is it the same thing as assessing the testability of a design?
4. It’s a model all the way down.
   1. Is the object of the test the model?
   2. Is the test itself the model?
   3. Is it both—a model of a model?

Some more general questions:

1. What model of testing do these aphorisms suggest?
2. What is the justification for the claims implicit in each of them?
3. What are the weaknesses in these claims?
4. What actions do these aphorisms entail for the practice of testing?
5. Can we test the practical value of each of these aphorisms? How?

No one knows what reliability is.

Or, to put it more accurately, no one agrees on the definition of reliability as it applies to software. Is it

1. Predictable behavior? The software should just work—it should respond in the way I expect within a consistent timeframe. This also implies that at a bare minimum it should never crash unexpectedly or corrupt the data I’m working on.
2. Durability? The software should require very little maintenance over time—or perhaps none at all.
3. Fault tolerance? The software should continue running even in the presence of multiple unexpected errors.
4. Diagnosis and repair? If the software does fail for some reason, it should provide convenient tools or resources for recovering from that failure and resuming work.
5. Efficient use of resources? The software should not use resources in a way that negatively impacts overall system performance.

At different times I’ve spoken with individuals who each valued one of the above characterizations of reliability above the others. I think there is a case to be made for each of them.

Generalizing from this multiplicity illuminates what I think is a key aspect of how to think about the quality—the definition of reliability is often domain-specific. Taking the first definition above, for example, who can say what constitutes predictable behavior without also discussing the particulars of the behavior that is expected?

As a tester working on system reliability in general, rather than reliability within the context of a specific component, this presents a dilemma—how can one generically test the reliability of a system if as a quality it is inherently domain-specific?

I’ll end with some of the open-ended questions I’ve been thinking and writing about as I attempt to resolve this dilemma and make progress in my work:

1. How can we model any of the definitions above?
2. Can we model the environmental factors that induce the behavior we wish to observe?
3. How can we measure the response in a useful, quantitative manner?
4. What is our failure model? Is it too coarse? Too fine-grained?
5. What effect does our failure model have on the measurements we make?
6. How can we look for these concerns when the code is still at the design stage?
7. What would be a useful test tool for each of the reliability definitions above? Does it have to be domain-specific or is it possible to think of something generic?

Traditional continuation implementation techniques depend on the ability to modify the area of memory holding the runtime call stack, but the CLR and similar VMs such as the JVM prevent this for security reasons. Nonetheless, it is still possible to implement continuations through a variety of means.

To explore these alternative implementations I built a model Scheme interpreter in C#. I then presented four separate methods of implementing continuations, each with their own set of tradeoffs.

The four methods are:

1. Exceptions. The exceptions that currently exist in the CLR and JVM can be viewed as one-shot escaping continuations. As such they can provide the basis for first-class continuations that also have the same behavior. Last I checked the JScheme Java Scheme interpreter used this method for its continuations.
2. Heap. Instead of using the CLR stack an interpreter can maintain its own stack in heap memory. This allows the interpreter to implement reusable first-class continuations. However it also effectively requires the interpreter to act as a separate VM running within the context of the parent instance of the CLR or JVM. SISC is an example of an interpreter that uses this method.
3. Stack reconstruction. With the right kind of artful manipulation an interpreter can use the exception mechanism in the CLR to inspect the current runtime stack in order to build a logical model of its contents and, when necessary, clear the current stack and replace it with a new stack built from one of the models. The method comes from papers by Sekiguchi, Sakamoto, and Yonezawa and Pettyjohn, Clements, Marshall, Krishnamurthi, and Felleisen. At the time I wrote my thesis there were no publicly-available interpreters I could find that used it.
4. Threads. The thread method for implementing continuations also revolves around viewing an existing runtime facility in a novel context. In this method threads are treated as stack segment containers. When a continuation is captured the current thread is suspended and a new thread is created and started to continue execution. Invoking a continuation discards the current thread and restarts the thread that was suspended when the continuation was captured. The method comes from Feeley. The architectural limitations this implies for the continuations that result are similar to the exception method.

Ultimately I argued that the stack reconstruction method was superior, since it was the only method that could provide unlimited continuations while still using the CLR runtime call stack to maintain execution context.

The PDF for the document is now available here at lorelli.info. The appendices contain the complete C# code for the four variations of the Scheme interpreter described above.

The way the music recontextualizes the game footage astonishes me each time I watch it.

The original photo galleries are still MIA. Eventually I’d like to get them back up at the new web host. I’ll probably change the format significantly though. I used to export pages straight out of iWeb, which worked well enough, and before that I used Gallery. Now with the change in hosts I’d like to see what new photo gallery web apps are available. I’m particularly interested in Zenphoto - the demo galleries make a good first impression.