iPhone HTTP Live Streaming - Between a rock (Microsoft) and a Hard Place (Apple)

We were asked by a customer to create a live streaming system for the iPhone.  No problem, I know the iPhone 3.x software supports live streaming, and we've worked with streaming video on other platforms before...

After some quick googling I discover that the iPhone only supports an emerging standard known as HTTP live streaming.  As far as I can tell it is only supported by Apple software (Safari & Quicktime X).

In a nutshell HTTP live streaming is accomplished by creating small (10 second or so) segments of a mpeg-ts stream (usually h.264/aac or mp3) and then creating a playlist containing pointers to the URL's of those segments.  The playlist is continually updated as new segments are available, and older segments fall out.  The client device (iPhone in this case) polls the playlist periodically looking for segments to stream.

The trick in creating a HTTP live stream is to correctly create the segments, which should be split on key frame boundaries so the client can put them back to together correctly.

Apple provides a piece of software to do this called mediastreamsegmenter.  Of course it only runs on OSX.  I don't have a Mac.

Ok, off to google again.   There it is.  A piece of open source software to the rescue:  iPhone Windowed HTTP Live Streaming Server

This is a great piece of software that uses some ruby scripting, a small C program and FFMPEG to do the segmenting.  I'm saved.

Well, not quite yet.  What I am actually trying to do is re-stream an existing video stream to the iPhone.  The existing stream is streamed via MMS and uses the Microsoft VC-1 codec.  My first thought is that this is not going to be an issue, FFMPEG can certainly transcode VC-1/WMA to H.264/MP3.  But what about MMS?

Nope.  FFMPEG doesn't seem to support the MMS streaming protocol.  I'll have to figure something else out.

I figure I can modify the configuration and/or ruby script for the open-source segmenter to use VLC.  VLC supports the MMS protocol.

Next problem.  When I try to use VLC to view (or transcode) the MMS stream it seems to hang for a few minutes before it begins playing.  That's odd.  I wonder whats going on.  I find this link: http://forum.videolan.org/viewtopic.php?f=14&t=45844&p=237757

It talks about other people seeming to have the same problem (and it's from a couple of years ago).  It also points to a VLC bug report that is still mark as unresolved.

Looking through the thread I notice a post from carver (thanks carver!) who ran a TCP sniffer on the VLC/MMS communication.  I quickly notice that VLC is sending out a request using HTTP/1.0, but the MMS server is responding with "Content-Length" and "Keep-Alive" headers which should only be used in response to an HTTP/1.1 request.  Keep-Alive means the server is not going to shut down the connection, but rely on the client to realize the request is finished when it hits Content-Length.  I'm guessing VLC (since it is expecting a HTTP1.0 response which doesn't use this mechanism) is just sitting there waiting for more data even after content-length is reached.  Maybe I can fix this...

Download the latest VLC from git, make sure the problem still exists (it does), and take a quick look at the code.  Yup, looks like the above could quite possibly be the problem.  Try to figure out the code a bit, make some changes, and see if it works.  Yup!  Another roadblock overcome.

I'm posting the MMSH patch for VLC in a seperate post to make it easier for people just wanting to use the patch:
VLC MMS start delay (keep-alive) patch

So now I give it a shot.  I modify the configuration files & ruby script having VLC do the receiving/transcoding from the MMS stream to mpeg-ts/H.264/mp3 and pass it to the open source segmenter to re-segment the stream.  It seems like it's working, but the resulting HTTP live streaming feed is choppy with audio sync issues.  Ok, I'll try just using VLC to receive the MMS stream and have ffmpeg do all the transcoding/remuxing.  I really think this should work.  But it doesn't.  I'm not sure why.  Still audio/video issues.  It's really close, but no cigar.

I spend a day playing with combinations of VLC/ffmpeg and the open source segmenter.  I can't get it to work.

Now I'm sure someone can get the above to work.  It should work, but I'm tired of playing around.  What occurs to me is what I am trying to do is really silly.  I'm trying to get VLC to demux, FFMPEG to transcode/remux, and then the open source segmenter to again demux/remux into segments.  This is alot of remuxing and most of it is pointless.  I start to think, hey, I've already started modifying the VLC code, why can't I just implement HTTP Live Streaming directly as a module (VLC is based on modules) for VLC.  It can't be much harder than fooling around like I've been doing and it's certainly going to be more fun.

So that's what I did.  I wrote a new VLC module (livehttp) so that VLC itself can do all of the transcoding/muxing/segmenting.  Really went relatively smoothly, especially considering I had never looked at the VLC codebase before and only had a working knowledge of the basics of trancscoding and muxing.

I've put together a seperate post with the livehttp streaming patch for VLC:
VLC HTTP Live Streaming module (patch)

And then it was done.  I have a Windows Server streaming MMS VC-1/WMA video to a Linux server running my patched version of VLC, which can then be streamed live via HTTP live streaming to an iPhone.