Sooo, I mentioned that we did a few *firsts* at eBay. One of them was to put Windows Server 2008 R2 in the DMZ and try to get SharePoint 2010 to work. Needless to say, after setting everything up properly, it didn't. After enabling 500 failed request tracing in the web servers and doing some crazy port analyzer stuff with good ole WireShark, I determined that there was some kind of firewall issue occuring between IIS and the domain controller in the DMZ.
Everything was setup properly in the firewall, with IIS and with SharePoint, so we all were very confused by what was occuring. It got so bad, that we made a preimer support call to see if someone could help us figure out what the hell was going on. Turns out we got one of the top IIS guys in Redmond on the phone and he was able to very quickly tell us we were having an RPC problem. We were all very confused, why are we having an RPC problem? He quickly told us about NDR64. NDR64 is a new protocol for RPC communicaiton in windows server 2008 and windows server 2008 R2. This protocol was not exactly written by Microsoft nor did Microsoft want to implement it (hint it was a MS partner that drove them to do it, but NDA keeps me from saying who and why). here's the details of NDR64:
So what does that mean? Well, let me explain it…turns out that today's firewalls are able to recognize RPC packets and conversations and can dynamically open the ports between servers based on these conversations. The new NDR64 is a 64bit version of the older 32bit version of the RPC protocol and MOST firewalls today, DO NOT know how to handle or interrogate the traffic. SO what does that mean? It means that you better be ready to open a whole bunch of high level ports to ensure that your authentication traffic will work when running SharePoint 2010 in your DMZ on Server 2008 and Server 2008 R2.
Until all the firewall vendors write a new bios/software to support his new RPC protocol, you will have to keep from implementing a DMZ 2010 environment, or open a lot of ports until your firewall vendor figures it out.
I'd start calling your firewall vendor NOW and see when you can get the latest firmware/software version.
Enjoy,
Chris