Open Testware Reviews
Holodeck Enterprise Edition (Trial Version)
Copyright 2003 by Tejas Software Consulting
- All rights reserved.
Contents
Overview -- Maturity
-- Project activity -- Platforms -- Support -- Documentation -- Installation
-- Implementation -- Performance -- Similar
tools -- Limitations -- Observations -- Appendix: Exported
system call log
Overview
Reviewed: 2003-July-23
Version reviewed: 2.0.173,
2003-June-3
Maintainer: Security Innovation
URL: http://sisecure.com/holodeck/
Testingfaqs.org category:
Test Implementation Tools
License: custom (closed-source,
binary-only)
User interface: GUI,
command line
Holodeck is a fault injection tool that you can use to test the robustness
of your Windows applications. "Fault injection" may be an intimidating term,
but the faults that Holodeck deals with are easy to understand - running out
of memory, file not found errors, network disconnects, etc. The power of
a fault injection tool is that you can simulate these errors without actually
filling the memory, corrupting your disk, or yanking your network cable.
If you haven't used a fault injection tool on your application before, there's
a high likelihood that it will root out some nasty and otherwise hard-to-find
bugs.
According to the IEEE, robustness is "The degree to which a system or component
can function correctly in the presence of invalid inputs or stressful environmental
conditions." What it doesn't mention is that robustness testing is fun,
but also disconcerting because you'll see how badly your application reacts
to everyday stresses. Oddly enough, the concept of robustness is mentioned
only in one obscure place on the Security Innovation web site.
The free trial version of Holodeck just barely meets my definition of freeware.
Because the trial does not expire, and because just enough features are enabled
for the tool to be useful, I'm going to count it as freeware. Security Innovation
did not really intend to create a freeware tool; they want to convince you
to buy the full version. In fact, there's a decent chance that they will implement
a time limitation in the trial version in the future that will bring it outside
my definition of freeware, though there are some other channels for getting
earlier versions of Holodeck.
The full version of Holodeck has some very interesting features that are
worth considering purchasing. But staying within the scope of Open Testware
Reviews, I will only review functionality that's available in the trial version.
Here's a screen shot of Holodeck testing another instantiation of itself:
Maturity
3 - Alpha (on a scale
of 1-5)
Performance is the main issue dragging Holodeck down to alpha status.
Earlier versions of Holodeck had much better performance, even on slow hardware
platforms. Also, Holodeck is vulnerable to some of the same faults that it
injects into your applications, which is surprising. I anticipate that Holodeck's
maturity will improve noticeably within the next few releases.
I have looked at a preview of the 2.1 release of Holodeck, and it has some
substantial performance improvements, so it might rate as Beta on the maturity
scale.
Project activity
unknown
Because of the closed-source nature
of this product, I won't try to guess at project activity. Security Innovation
is actively working on the tool. However, I don't see a visible user community.
Platforms
Holodeck runs only on Windows 2000 and XP. 256
MB RAM is recommended, 128 MB is the minimum supported. The tool requires
90 MB of disk space according to the documentation (actually more than 96
MB on my XP system). A minimum 450 MHz CPU is recommended, Pentium III or
Celeron.
The help text states that Holodeck runs only on Windows XP, but the web
page says that Windows 2000 is also supported. I was able to run Holodeck
on Windows 2000, which was its original development platform.
Support
Security Innovation is unlikely to provide much support unless you express
an interest in evaluating the tool for purchase. They did not intend for
the trial version to be touted as freeware.
Holodeck has a "Report a bug" option under its Help menu, which takes you
to a Security Innovation web form by embedding Internet Explorer into Holodeck.
I used this form a few times to report serious bugs and didn't get any response.
The contents of the bug database is not publically available, so save your
bug data before you submit it. There is also a support@sisecure.com address
that you can use if the form doesn't work. I was able to confirm that my
two bugs were received when I contacted the support staff via this email
address. Except for not getting a confirmation of the bugs I submitted until
I asked, the support staff was very responsive.
I am not aware of any online forums where users discuss usage of the tool.
Being a closed source tool, there is no public change log or configuration
management.
Documentation
The online help text serves as the tool's documentation. The help text
is fairly thorough, though it could use some copyediting. There are a few
places where jumping-off points into other pages in the help text show up
at surprising places. There is a nice reference of Windows API calls, error
codes, and exception codes. It doesn't say which version of Windows that
these references apply to. The help text does not document which features
are disabled in the trial version, so you have to use trial and error to
find out.
The "Release Notes.doc" file that installs with the tool documents 13 known
issues. It's good to see Security Innovation being forthcoming with this
information.
The book How to Break Software, which I've posted a review
of, is a good introduction to robustness testing. The author, James Whittaker,
was the project lead on the initial development of Holodeck, and is now
the Chief Scientist of Security Innovation.
Installation
Holodeck Enterprise Edition installs from a 31 MB InstallShield executable.
You can download it from the Security Innovation web site. There is a simple
form requesting your name and email address, both of which are optional.
There is a check box to request more information, and I had a response within
minutes when I used it.
I had to reboot once after uninstalling, though I didn't have to reboot
after another uninstall. Installing Holodeck did not require a reboot. Uninstalling
Holodeck Enterprise Edition will break Holodeck Lite if you have it installed,
probably because of a shared HeatDll file (more information in the Similar tools section).
You'll have to run the installer twice if another version of Holodeck is
already installed, once to uninstall, and again to actually install. The
installer just exits after uninstalling the old version, with no explanation.
Implementation
Because Holodeck is a closed source tool, it doesn't really help to know
the implementation details. You'll find evidence of the .NET components if
you see a crash.
Performance
I experienced very slow performance with Holodeck. Because of the extensive
hooks that it has in the applications that it runs, I expected that applications
running under Holodeck would potentially run very slowly. But some core Holodeck
features are inordinately slow, much slower than the application I'm testing.
I thought my 450 MHz AMD K6-2 met the minimum requirements, but it turns
out that the generation of the CPU is as important as the clock speed. So
technically my computer isn't fast enough to run Holodeck, though thankfully
the tool runs anyway.
It took about 33 seconds to start up the "Create A New Project Wizard,"
between clicking the button and getting the first dialog. It took more than
two minutes to start a test project for the Windows Notepad. It took more
than 45 seconds to bring up the first dialog for the "Create A Network Corruption
Fault Wizard." It would be hard to tolerate the slow response time even running
four times faster.
The support folks at Security Innovation tell me that they are working on
big performance improvements for the next Holodeck release. Also, they explained
that Holodeck does a lot of caching when it starts up, which explains why
the performance of the application under test is fairly good after the slow
startup. I clocked the three cases mentioned above on a beta version of the
2.1 release (build 2.1.198), and got these numbers on average - 4 seconds
to start the "Create A New Project Wizard," 37 seconds start start a test
project, and 52 seconds to open the "Create A Network Corruption Fault Wizard."
So the first two are noticeably improved in the new version, and would likely
be tolerable on a much faster computer.
Similar tools
The Florida Institute of Technology (FIT) distributes earlier incarnations
of Holodeck - Holodeck Lite and Canned HEAT. These tools were initially
developed under the auspices of FIT. They have moved the tools to the "boneyard"
section of their web page, which sometimes requires that you do some sleuthing
to fix up the URLs. I have a feeling that they may delete all traces of them
from the web site without warning. You can also get these two tools from the
CD that comes with James Whittaker's book, How to Break Software. I
hear that similar tools also come with his new book, How to Break Software
Security.
Holodeck Lite, with
an install package of less than a megabyte, can safely be installed on the
same system as Holodeck Enterprise Edition except for the problem mentioned
previously about uninstalling. Holodeck Lite can use the newer HeatDll file
that comes with Enterprise Edition. Holodeck Lite has drastically better
performance than the Enterprise Edition on my XP box. It also allows two
each of the network, disk, and memory faults, compared to one of each for
the newer Holodeck. However, it doesn't support the scheduled test and network
corruption fault features of the new trial version. It's worth a look if
you can get it.
An earlier cousin of Holodeck is Canned HEAT.
The downloads
page
is a bit hard to find. Though it offers fewer overall features than any
of the unrestricted Holodeck releases, Canned HEAT has no restrictions at
all on which of its injectable faults you can use, so it offers many more
fault injection options than the free incarnations of Holodeck. The user
interface is buggy and somewhat difficult to use (for example, I have to
restart Canned HEAT each time I want to restart the application under test),
but because of its richer set of faults compared to the free versions of
Holodeck, you might want to give it a try.
There is a syscalltrack
tool for Linux that seems to have a similar design as Holodeck. It uses
a command line interface rather than a GUI, but it seems to be as powerful
and perhaps more flexible than Holodeck. Similar tools for other operating
systems are likely to be lurking out there somewhere.
Limitations
The greatest limitation with the version of Holodeck that I reviewed is
that certain important aspects of the tool run very slowly, especially if
you have slow hardware.
Also, while the Holodeck trial version is usually able to trip across
robustness bugs, I would prefer to target faults to specific individual system
calls in the application code. This would help ensure that you have thorough
coverage of many of the possible things that could go wrong, rather than
just hitting an unknown point in the code when I enable a fault. The "Code
Coverage Test Generation" feature of the full version of Holodeck looks like
it might meet this need.
Other limitations I noted -
- I encountered two Holodeck crashes. One was an unhandled exception
apparently related to an out of bounds data element, and the other was a
disk space exception (I'm pretty sure I was not out of disk space at the
time). I did not reproduce either of these crashes during normal use of the
tool. However, I did use Canned HEAT to show that Holodeck does indeed crash
if it tries to write to a full disk. It would be nice if Holodeck itself
were robust against the kinds of errors that it's trying to root out of our
applications.
- There are some worrisome bugs mentioned in the release notes, including
one that crashes the OS (actually, this one is related to the nature of this
kind of tool and isn't terribly surprising), plus three catastrophic failures
and some serious uninstall bugs. I also encountered a minor bug mentioned
in the release notes - I tried to install Holodeck while Windows was in safe
mode - the install screen was larger than 640x480 and I couldn't access the
buttons.
- The status of the application under test is reported only in a status
line at the bottom of the Log pane (paused, active, terminated, etc.). This
status line was not always visible in the Log pane during my testing, and
it was never visible when I was using the Faults pane.
- The only way to restart an application is to close and reopen its
project file. This is not clearly documented.
- The "Create New Project Wizard" dialog is somewhat confusing - you
have to type the name of your project at the end of the path or click "Browse."
- I couldn't figure out how to toggle the "Pause Application on Start"
flag after creating a project, or even to see the state of the flag. I had
to create a new project to change the flag.
- The way Holodeck embeds Internet Explorer produces some strange results.
For example, there's no way to go back to where you came from when viewing
the API documentation. It's possible to get to a search engine and thus
view practically any web site within Holodeck, which isn't really bug, but
an unintended feature.
- Holodeck can't launch itself as the application to test (it logs system
calls for it but never shows the GUI), but it can usually attach to another
running instance of Holodeck.
Observations
The trial version of Holodeck Enterprise Edition teases us with a small
subset of a powerful array of fault injection features. When it wraps its
tentacles around an unsuspecting application, Holodeck can force a file not
found error, an insufficient memory error, or a network disconnect error from
all system calls that can return these types of errors. You can toggle these
faults on and off at will. When you turn one of them on, any eligible system
call that the application under test uses will get the fault.
You can also get to some of the more advanced features - network corruption
faults and "scheduled tests," both of which offer some fairly sophisticated
methods of corrupting data in ways that you can specify. In the trial version
you may only use one of the scheduled tests for a single system call, or a
network corruption fault, during a run of the application. I used the network
corruption fault feature in concert with Netscape to create some very amusing
effects. I got garbage across random parts of the web pages I browsed, and
I reproduced an incarnation of what James Whittaker calls his favorite error
message of all time:
Another feature of Holodeck is its ability to log native Windows and .NET
functions, so you get a system call trace. You can filter the trace so you
only track calls that you're interested in seeing. See the Appendix for a
sample. Holodeck will show you the string and integer parameters, but more
complex data structures seem to come through only as a raw pointer. I've found
system call trace tools to be very useful in determining exactly what an
application is trying to do. You'll need to have some knowledge of Windows
and/or .NET system calls in order to thoroughly understand the information
in the log. However, some educated guessing can go a long way, and you can
look up functions via links in the help text.
Holodeck has a per-thread mode that separate both the faults and the logs
by thread. This can allow you to hone in on a particular thread, though you
may have difficulty knowing the ID of thread you want to target.
There's no application that I've tested with Holodeck that didn't exhibit
robustness problems. I have used it on a client project to find bugs in a
large commercial application. I can easily crash Netscape and Notepad with
Holodeck. The most benign response I saw was in the Windows pinball game,
which stopped displaying its informational messages when under adverse conditions.
So I tried pinball again, this time pausing it on startup - and when I turned
all the faults on right at the beginning, it died silently before bringing
up the GUI.
It's interesting to note that Security Innovation markets Holodeck as a
security test tool. It's certainly sobering to know that any Windows API
call can be intercepted with a tool like Holodeck. The full version of Holodeck
allows you to write code that replaces functions and returns any value you
want to. While security testing is one aspect of what Holodeck can help with,
I recommend that you use it for general robustness testing. You can't do much
security testing with the trial version.
Security Innovation is taking a university project and turning into an industrial-grade
product. Holodeck is going through some growing pains now, such that you might
consider trying to previous releases of the tool in the short term, or check
to see if a newer version has been released. The trial version doesn't make
it easy to do exhaustive robustness testing of each system call in your application.
However, the fact that in its current state it's very likely to find serious
flaws in your application is very compelling, especially knowing that these
flaws would be difficult to find without a tool like Holodeck.
Appendix: Exported system call log
Here's the beginning of a Holodeck log, with a few tab corrections for readability.
TimeStamp ThreadID Category DllName Function ReturnValue ErrorCode Exception Parameter 0 Parameter 1 Parameter 2 Parameter 3 Parameter 4 Parameter 5 Parameter 6 Parameter 7 Parameter 8 Parameter 9 Parameter 10 Parameter 11
7/17/2003 21: 4:21:250 1012 FILE kernel32.dll GetFileAttributesW 4294967295 2 C:\WINDOWS\system32\mscoree.dll.local
7/17/2003 21: 4:21:260 1012 FILE kernel32.dll CreateFileW 4294967295 2 C:\Program Files\Security Innovation\Holodeck Enterprise Edition\HolodeckGui.exe.config 2147483648 1 0 3 128 0
7/17/2003 21: 4:21:250 1012 REGISTRY advapi32.dll RegOpenKeyExW 2 0 2147483649 Software\Microsoft\.NETFramework 0 131097 0
7/17/2003 21: 4:21:250 1012 REGISTRY advapi32.dll RegOpenKeyExW 0 0 2147483650 Software\Microsoft\.NETFramework 0 131097 72
7/17/2003 21: 4:21:260 1012 REGISTRY advapi32.dll RegQueryValueExW 0 0 72 InstallRoot 0 1243376 0 1243392
7/17/2003 21: 4:21:260 1012 REGISTRY advapi32.dll RegQueryValueExW 0 0 72 InstallRoot 0 0 1607312 1243392
7/17/2003 21: 4:21:260 1012 REGISTRY advapi32.dll RegCloseKey 0 0 72
...