Wednesday, March 24, 2010

iPhone HTTP Live Streaming - Between a rock (Microsoft) and a Hard Place (Apple)

We were asked by a customer to create a live streaming system for the iPhone.  No problem, I know the iPhone 3.x software supports live streaming, and we've worked with streaming video on other platforms before...

After some quick googling I discover that the iPhone only supports an emerging standard known as HTTP live streaming.  As far as I can tell it is only supported by Apple software (Safari & Quicktime X).

In a nutshell HTTP live streaming is accomplished by creating small (10 second or so) segments of a mpeg-ts stream (usually h.264/aac or mp3) and then creating a playlist containing pointers to the URL's of those segments.  The playlist is continually updated as new segments are available, and older segments fall out.  The client device (iPhone in this case) polls the playlist periodically looking for segments to stream.

The trick in creating a HTTP live stream is to correctly create the segments, which should be split on key frame boundaries so the client can put them back to together correctly.

Apple provides a piece of software to do this called mediastreamsegmenter.  Of course it only runs on OSX.  I don't have a Mac.

Ok, off to google again.   There it is.  A piece of open source software to the rescue:  iPhone Windowed HTTP Live Streaming Server

This is a great piece of software that uses some ruby scripting, a small C program and FFMPEG to do the segmenting.  I'm saved.

Well, not quite yet.  What I am actually trying to do is re-stream an existing video stream to the iPhone.  The existing stream is streamed via MMS and uses the Microsoft VC-1 codec.  My first thought is that this is not going to be an issue, FFMPEG can certainly transcode VC-1/WMA to H.264/MP3.  But what about MMS?

Nope.  FFMPEG doesn't seem to support the MMS streaming protocol.  I'll have to figure something else out.

I figure I can modify the configuration and/or ruby script for the open-source segmenter to use VLC.  VLC supports the MMS protocol.

Next problem.  When I try to use VLC to view (or transcode) the MMS stream it seems to hang for a few minutes before it begins playing.  That's odd.  I wonder whats going on.  I find this link: http://forum.videolan.org/viewtopic.php?f=14&t=45844&p=237757

It talks about other people seeming to have the same problem (and it's from a couple of years ago).  It also points to a VLC bug report that is still mark as unresolved.

Looking through the thread I notice a post from carver (thanks carver!) who ran a TCP sniffer on the VLC/MMS communication.  I quickly notice that VLC is sending out a request using HTTP/1.0, but the MMS server is responding with "Content-Length" and "Keep-Alive" headers which should only be used in response to an HTTP/1.1 request.  Keep-Alive means the server is not going to shut down the connection, but rely on the client to realize the request is finished when it hits Content-Length.  I'm guessing VLC (since it is expecting a HTTP1.0 response which doesn't use this mechanism) is just sitting there waiting for more data even after content-length is reached.  Maybe I can fix this...

Download the latest VLC from git, make sure the problem still exists (it does), and take a quick look at the code.  Yup, looks like the above could quite possibly be the problem.  Try to figure out the code a bit, make some changes, and see if it works.  Yup!  Another roadblock overcome.

I'm posting the MMSH patch for VLC in a seperate post to make it easier for people just wanting to use the patch:
VLC MMS start delay (keep-alive) patch

So now I give it a shot.  I modify the configuration files & ruby script having VLC do the receiving/transcoding from the MMS stream to mpeg-ts/H.264/mp3 and pass it to the open source segmenter to re-segment the stream.  It seems like it's working, but the resulting HTTP live streaming feed is choppy with audio sync issues.  Ok, I'll try just using VLC to receive the MMS stream and have ffmpeg do all the transcoding/remuxing.  I really think this should work.  But it doesn't.  I'm not sure why.  Still audio/video issues.  It's really close, but no cigar.

I spend a day playing with combinations of VLC/ffmpeg and the open source segmenter.  I can't get it to work.

Now I'm sure someone can get the above to work.  It should work, but I'm tired of playing around.  What occurs to me is what I am trying to do is really silly.  I'm trying to get VLC to demux, FFMPEG to transcode/remux, and then the open source segmenter to again demux/remux into segments.  This is alot of remuxing and most of it is pointless.  I start to think, hey, I've already started modifying the VLC code, why can't I just implement HTTP Live Streaming directly as a module (VLC is based on modules) for VLC.  It can't be much harder than fooling around like I've been doing and it's certainly going to be more fun.

So that's what I did.  I wrote a new VLC module (livehttp) so that VLC itself can do all of the transcoding/muxing/segmenting.  Really went relatively smoothly, especially considering I had never looked at the VLC codebase before and only had a working knowledge of the basics of trancscoding and muxing.

I've put together a seperate post with the livehttp streaming patch for VLC:
VLC HTTP Live Streaming module (patch)

And then it was done.  I have a Windows Server streaming MMS VC-1/WMA video to a Linux server running my patched version of VLC, which can then be streamed live via HTTP live streaming to an iPhone.

16 comments:

  1. Really good work!!! I spent a lot of time in the last month on HTTP live streaming and your approach is really interesting!

    ReplyDelete
  2. Very interesting way to solve the problem! I am curious though, did you ever find a PC client to receive the http live streaming? Particularly for testing outside of an iPhone and/or mac?

    ReplyDelete
  3. great work
    I'm trying to do he same thing, please tell me did you install the VLS?

    ReplyDelete
  4. Awesome work! I'm trying to learn about HTTP Live streaming and starting to try to contribute to related open source projects, do you have any recommendations ?

    ReplyDelete
  5. Thanks a lot for doing this. I did not find the livehttp module in the VLC1.1.4/1.1.3/1.1.2 or 1.1.1 distributions for OSX 10.6 ( Intel 64-bit). So, I decided to compile the sources and enable livehttp, but I am stuck at the point where VLC build is failing because it can't find libX11.la files ( which I believe are no longer distributed with OSX 10.6). Looks like this is work cut out for someone who is familiar with building VLC with all its myriad settings.

    I need some help either a) building this on Mac or b) finding a Mac version of VLC distro with livehttp module that works.

    Thanks a lot.

    ReplyDelete
  6. hello,
    it looks that the module doesn't work with audio only live streams.
    Try with any mms:// link with only audio to segment this to mp3 chunks and you would get an error access_mms error . but with video works.
    Any ideas ?

    ReplyDelete
  7. Can anyone guide me with some sample code in iOS for playing live streaming video. My requirement is similar to the below link
    http://www.youtube.com/watch?v=gWpJRfGS8cc

    ReplyDelete

Note: Only a member of this blog may post a comment.