Edward,
Testing firewalls is a very complex undertaking. Being thorough is
perhaps the most valuable asset you will have.
There are some basic kinds of questions which need to answered.
o Are you trying to establish a firewall test lab or do you just want
to find out which firewall is the best for your company?
The answer to this question determines the type of testing that you
will be performing. Testing for a firewall test lab goes way beyond
what a someone will do for their company. When someone is looking
for a firewall for their company, they will start a sifting process
which should take a couple of months (gathering data, verifying claims,
etc.). Much of the initial sifting process can be done on paper without
having to take firewalls apart. When you have a set of 3-5 firewalls
which meet your basic criteria, then start take the firewalls out for
a test drive. This will eliminate another one or two. Then start the
testing process.
o "Joe at Company A uses brand X firewall. He likes it and recommends it
very highly. I should use it too, right? Maybe, maybe not. First,
Joe's comments are hearsay. Second, Joe's experience with firewalls
may be limited. Third, and most important: What works well for Joe,
may be a complete disaster for your company.
Every company has unique business and security requirements. A firewall
is an implementation of a security policy which is based on these
requirements. Putting the NSA's security posture into a university will
bankrupt the university very quickly. Putting a university's security
posture (of an open environment) into the NSA is a recipe for a national
security disaster. Determine in advance what you need and choose your
firewall accordingly. Choose wisely.
o A firewall is an implementation of a security policy. Having the policy
will help define the firewall's rules as well as deal with legal and
non-compliance issues.
o How much time are you ready to spend testing?
A "network scan" of a firewall for vulnerabilities can take as little
as 5-15 minutes using commonly available commercial products which
were mentioned in other's postings.
A thorough firewall test takes 1-2 months (minimum). It is extremely
time-consuming to test a firewall and do it right. (And we haven't
even gotten to the report-writing) 8^(
As in the testing of CPU chips, complete testing coverage isn't practical
or even feasible. You have to do the best you can in the time allowed.
o What is your methodology?
Before you start testing, you should first map out your firewall test
methodology. If you are looking for a starting point, you might check
out www.fortified.com which has a Free Firewall Evaluation Checklist.
The Checklist is available via HTTP only. While it is primarily
designed to help people who are evaluating firewalls, it may give
you an idea of some things you might want to test.
o What should I test for?
o Vulnerabilities Most people test for vulnerabilities. If it passes
all of the tests, then it must be OK. Right? Not really. Testing
of a firewall should be *very* comprehensive and go way beyond looking
for vulnerabilities. A firewall's ability to pass vulnerability tests
may or may not be a good indicator of how robust the firewall really
is. It could mean that the firewall has a very robust architecture
and it is not vulnerable against the attacks you tried. It could also
mean that the firewall's architecture is not quite up to speed and
that
the vendor is very fast in generating patches for their product. Both
appear to produce the same results. Looking at the firewall in
detail will help determine what is really going on.
o Functionality - does the firewall do the things it is supposed to do?
o Gotchas - does the firewall do the things that it is not supposed to do?
o Verification of claims - does the firewall really do all of the
things that the vendor says it can? This different than the Gotchas
or Functionality testing mentioned above
o Documentation
o How easy/difficult it is to configure the rules
o Tech Support
o History of the company
o Etc., etc.
o What about firewall "certification"?
Some organizations will wave a scanning tool across the firewall and
"certify" it if it passes all of the tests. One in particular comes
to mind. In this particular case, I am not aware of any firewalls
which failed to be certified. Most of the "certified" firewalls would
have not made it past the initial sifting process of evaluating firewalls.
This doesn't mean that they may be bad firewalls. It only means that I
don't consider them robust enough to recommend or use for my purposes.
YMMV, of course.
I've discovered problems in every firewall I ever tested. So have
other professionals on this list. Although most problems are minor,
some have been rather severe ("show-stoppers"). (Please don't bother
to ask which ones I have tested, or which ones have had problems.)
There is no such thing as a perfect firewall. Some are better than
others in different areas. You really have to look at the whole
picture. As vendors tend to leapfrog each other in terms of
technology, the test criteria get updated frequently.
Also, there is no "one size fits all" when it comes to firewalls.
What works for you may not necessarily work for someone else.
My personal opinion is that firewall certification says nothing and
proves nothing. It is a nice marketing tool and tends to make a lot
of money for those who are performing the testing. Let me give you a
couple of examples:
o Suppose you have a firewall which passes every test you can think of?
What about the tests that you haven't thought of, (but the hackers will
or have)?
o Hypothetically speaking, suppose you have a bullet-proof firewall
which is impervious to every possible vulnerability. Unfortunately,
when the firewall is installed, it should is installed incorrectly.
Instead of being protected from the risks of the Internet, the company
now has more exposures than before - perhaps enough to bankrupt the
company.
o Who has failed, and for what problems? If no firewalls ever failed
the testing, then how valid is the testing methodology really?
o What about the legal liabilities if a "certified" firewall is
penetrated by an attacker? If the tester is going to certify
something, they should also be capable of backing up their claims
that the product performs as it should. What are the legal
liabilities for the tester if the firewall is known to be vulnerable
to certain types of attacks & the tester passes it anyway?
For all of the reasons above and more, I'll never certify firewalls or
other security products.
I hope the above has been of some help to you.
Best Regards,
Frank
The opinions of the author of this mail may not necessarily be
representative of the opinions of Fortifed Networks, Inc.
Fortified Networks, Inc. - http://www.fortified.com/
Expert (vendor-neutral) Computer and Network Security Consulting
Phone: (317) 573-0800 Fax: (317) 573-0817
Any sufficiently advanced bug is indistinguishable from a feature.
-- Rich Kulawiec
|
|
