firewall weirdness

[Mark Phillips <mbp@geomtech.com>]

Hi everybody.  I've been having some strange problems related to my
firewall, which is running RedHat 5.2.  I don't know if this is a
Linux issue or not, but since I'm running Linux on the firewall I
thought I'd ask here.  This note is rather long so if you don't know
or care about firewall issues, you might want to stop reading now.

Here's the setup:
                        ____________
                        |    my    |
       ISP    <------> eth0      eth1 <----->  my internal
      router            | firewall |             network
                        |__________|

The firewall normally lets through things that I specifically allow,
such as ssh.  It has a set of filtering rules that are loaded at boot
time; nothing on it (as far as I know) modifies the filtering rules as
it runs.

Yet twice in the last week I've enountered situations where suddenly
the internal network is unreachable.  Ssh connections that I had open
on external machines elsewhere on the internet, going to hosts in the
internal network, were dropped with the error message "no route to
host".  Neither ping nor ssh nor http produced nary a response from
any machine on the internal network (actually, there's only 1 internal
host, but it has multiple IP addresses).

I could get to the firewall itself fine, and its route table and
ifconfig configuration seemed normal, but the internal network wasn't
any more accessible from the firewall than it was from the outside.
So I rebooted the firewall, and then everything was fine, at least
until a few days later when the problem happened a second time.
The internal machine has been fine all along.

This firewall ran for seven months without a glitch, and these
problems started happening a few days after we moved it to a different
router at the ISP (and a different physical location); so the outside
interface got a new IP address, and some routing tables on the ISP
routers got changed to route our internal network to the firewall's new
location.  As a result of the move I edited the firewall's filter
rules to reflect the new IP address, and I also changed the machine's
hostname at that time.  These are the only recent changes.

What's got me baffled is that it works most of the time, and
then just out of the blue seems to drop the internal network.
The times when it has dropped the internal network have
both been in the middle of the night, but not the same time
both times, and neither time coincides with any cron jobs
or other regular things that happen on any of our machines.

Could this be:

  1. A flakey ethernet card on eth1?  This was my first
     thought, but the fact that rebooting the firewall
     fixes the problem seems strange in this case.  Unless
     rebooting actually power cycles the card.  Anyone know
     about this?

  2. A problem on my ISP's router?  If it's dropping routes
     to my internal network that could explain why the
     outside world can't see the internal network, but
     it doesn't explain why the firewall itself can't, right?

  3. Something else.  I'm open to suggestions.  Any ideas?

Thanks,

--Mark

Mark Phillips @ Geometry Technologies, Inc.
Suite 260, 400 Sibley St., St. Paul  MN  55101
651-223-2884, Fax 651-291-2402  
mbp@geomtech.com       http://www.geomtech.com