Sunday, May 27, 2007

Windows Media Services vs SonicWall Firewall

So today’s problem dealt with Windows Media Services. I have a friend that was looking at hosting and broadcasting video streams over the Internet and silly me said, “Hey I’ve done that and I even have a server at a collocation.”

Over the years, one of the Microsoft products I really enjoyed playing with as a hobby was Windows Media Services. It all started with a friend living oversees who wanted to watch “American” TV shows. So to help him out I started learning how to capture a TV show, encode it using Windows Media Encoder, and post it to a Windows Media Server.

A fascinating part was over time learning how to tweak the encoder properties to improve the quality. One of the best features is encoding multiple rate streams in the same file and allowing the server and player to negotiate the best stream. Towards the end, I was encoding content at multiple rates at resolutions from 1Mb/s down to 33.6Kb/s in 100Kb/s increments. This allowed devices as fast as broadband computers to cellular PDAs to automagically watch the streams that matched the network’s and device’s capability.

So here I have my old streams, a server, and I need to get a demo up and running to show what can “easily” be done. Stupid me for using the word “easily”. With my server at a co location site behind a SonicWall Firewall and a Windows Media Player 11 on Vista (hey, I like bleeding edge) behind a Cisco 501 PIX firewall at my house, the first test went great. The player opened the stream and everything appeared to be fine. The next test was to force the player to request a slower rate stream.

From within the options in Windows Media Player you can select different maximum bandwidths, forcing a server and client to negotiate a slower stream. Restarting the video with the new options, did not go so great with a rather bland error of “Windows Media Player encountered a problem while playing the file.” If I selected Web Help on the error message, slightly more information was display and root error was listed as “80070057, one or more arguments are invalid”. So the question now was, “Why do the slower streams not work?”

Step one, simplify the problem by removing devices that could be the problem. First went the PIX at my house. The PIX was performing Network Address Translation of the IP addresses entering my house network and since NAT can give audio and video streaming protocol fits, it seemed logical to try the client without the PIX. No change, error message still being displayed, so probably not the PIX or client side NAT.

The next test was to capture a packet trace when the Media Player worked and compare it to a packet trace when the media player did not work. This helped a little. In the trace that worked, the stream was being negotiated and transmitted via TCP in the trace that did not work, the stream was being negotiated via UDP and a few UDP packets were showing up in the trace. The odd part was during the negotiation, the client was replying back with a FIN flag on a TCP conversation, which kills the conversation.

Hmm, so the client is negotiating with the server and suddenly the client ends everything. I started wondering if the Windows Firewall was blocking the UDP. Disabling Windows Firewall proved that theory wrong when the problem persisted.

Moving foward, I wanted to test if it was UDP related, so changed the bandwidth setting in Windows Media Option to auto detect, and unchecked or disabled TCP. Quick try and the same error code, so UDP seemed to be the problem. Conversely, I lowered the maximum bandwidth, disabled UDP and enabled TCP, and the stream came through fine. So we are definitely dealing with a UDP problem.

Next check, does it work on the same network as the Windows Media Server. I remote consoled into the media server at the colocation site, opened Windows Media Player, changed the options to disable TCP, and played the stream. No error message! The slightly bad news was this introduced a new variable into the mix a bit. At home I was using Windows Media Player 11, but at the remote site I had version 10.

So now I am wondering if this is a bug in Windows Media Player 10 (running on Windows 2003) vs Windows Media Player 11 (running on Vista). Ran some more test from the house and it didn’t matter if I used version 11 or version 10, same error code. So this ruled out it was related to the version of Windows Media Player.

Since local worked and remote did not, I was left with the network path as the potential cause. At the collocation site, the server was firewalled from the Internet with a SonicWall firewall which was doing one to one NAT. I started looking through the firewall rule set to see if I had missed some UDP port, but everything here checked out.

So now I am left with just checking everything on the SonicWall Firewall. While going through the setup screens on the SonicWall, I found parameters for how Voice over IP was handled. Specifically there was a setting which enabled H.323 Transformations. H.323 is a standard for handling video conferencing , but both H.323 and RTSP (Real Time Streaming Protocol) use RTP (Realtime Transport Protocol) to deliver the content. Since Windows Media uses RTSP, it made we wonder if the SonicWall was mistaking the underlying RTP as H.323 and manipulating the packets.

A quick uncheck, apply, and another test was made. Holy Crap, it worked! Re-enabling the H.323 transformation on the SonicWall and the UDP streams to the media player client were broken again, confirming this was the magic parameter that needed to be changed. Since I did not have any video conferencing needs, I disabled the H.323 transformations on the SonicWall, saved the configuration, and called it quits on this problem. Beer me!