iPhone HTTP Live Streaming - Between a rock (Microsoft) and a Hard Place (Apple)

We were asked by a customer to create a live streaming system for the iPhone.  No problem, I know the iPhone 3.x software supports live streaming, and we've worked with streaming video on other platforms before...

After some quick googling I discover that the iPhone only supports an emerging standard known as HTTP live streaming.  As far as I can tell it is only supported by Apple software (Safari & Quicktime X).

In a nutshell HTTP live streaming is accomplished by creating small (10 second or so) segments of a mpeg-ts stream (usually h.264/aac or mp3) and then creating a playlist containing pointers to the URL's of those segments.  The playlist is continually updated as new segments are available, and older segments fall out.  The client device (iPhone in this case) polls the playlist periodically looking for segments to stream.

The trick in creating a HTTP live stream is to correctly create the segments, which should be split on key frame boundaries so the client can put them back to together correctly.

Apple provides a piece of software to do this called mediastreamsegmenter.  Of course it only runs on OSX.  I don't have a Mac.

Ok, off to google again.   There it is.  A piece of open source software to the rescue:  iPhone Windowed HTTP Live Streaming Server

This is a great piece of software that uses some ruby scripting, a small C program and FFMPEG to do the segmenting.  I'm saved.

Well, not quite yet.  What I am actually trying to do is re-stream an existing video stream to the iPhone.  The existing stream is streamed via MMS and uses the Microsoft VC-1 codec.  My first thought is that this is not going to be an issue, FFMPEG can certainly transcode VC-1/WMA to H.264/MP3.  But what about MMS?

Nope.  FFMPEG doesn't seem to support the MMS streaming protocol.  I'll have to figure something else out.

I figure I can modify the configuration and/or ruby script for the open-source segmenter to use VLC.  VLC supports the MMS protocol.

Next problem.  When I try to use VLC to view (or transcode) the MMS stream it seems to hang for a few minutes before it begins playing.  That's odd.  I wonder whats going on.  I find this link: http://forum.videolan.org/viewtopic.php?f=14&t=45844&p=237757

It talks about other people seeming to have the same problem (and it's from a couple of years ago).  It also points to a VLC bug report that is still mark as unresolved.

Looking through the thread I notice a post from carver (thanks carver!) who ran a TCP sniffer on the VLC/MMS communication.  I quickly notice that VLC is sending out a request using HTTP/1.0, but the MMS server is responding with "Content-Length" and "Keep-Alive" headers which should only be used in response to an HTTP/1.1 request.  Keep-Alive means the server is not going to shut down the connection, but rely on the client to realize the request is finished when it hits Content-Length.  I'm guessing VLC (since it is expecting a HTTP1.0 response which doesn't use this mechanism) is just sitting there waiting for more data even after content-length is reached.  Maybe I can fix this...

Download the latest VLC from git, make sure the problem still exists (it does), and take a quick look at the code.  Yup, looks like the above could quite possibly be the problem.  Try to figure out the code a bit, make some changes, and see if it works.  Yup!  Another roadblock overcome.

I'm posting the MMSH patch for VLC in a seperate post to make it easier for people just wanting to use the patch:
VLC MMS start delay (keep-alive) patch

So now I give it a shot.  I modify the configuration files & ruby script having VLC do the receiving/transcoding from the MMS stream to mpeg-ts/H.264/mp3 and pass it to the open source segmenter to re-segment the stream.  It seems like it's working, but the resulting HTTP live streaming feed is choppy with audio sync issues.  Ok, I'll try just using VLC to receive the MMS stream and have ffmpeg do all the transcoding/remuxing.  I really think this should work.  But it doesn't.  I'm not sure why.  Still audio/video issues.  It's really close, but no cigar.

I spend a day playing with combinations of VLC/ffmpeg and the open source segmenter.  I can't get it to work.

Now I'm sure someone can get the above to work.  It should work, but I'm tired of playing around.  What occurs to me is what I am trying to do is really silly.  I'm trying to get VLC to demux, FFMPEG to transcode/remux, and then the open source segmenter to again demux/remux into segments.  This is alot of remuxing and most of it is pointless.  I start to think, hey, I've already started modifying the VLC code, why can't I just implement HTTP Live Streaming directly as a module (VLC is based on modules) for VLC.  It can't be much harder than fooling around like I've been doing and it's certainly going to be more fun.

So that's what I did.  I wrote a new VLC module (livehttp) so that VLC itself can do all of the transcoding/muxing/segmenting.  Really went relatively smoothly, especially considering I had never looked at the VLC codebase before and only had a working knowledge of the basics of trancscoding and muxing.

I've put together a seperate post with the livehttp streaming patch for VLC:
VLC HTTP Live Streaming module (patch)

And then it was done.  I have a Windows Server streaming MMS VC-1/WMA video to a Linux server running my patched version of VLC, which can then be streamed live via HTTP live streaming to an iPhone.

VLC HTTP Live Streaming module patch (iPhone streaming support)

As noted in the post iPhone HTTP Live Streaming - Between a rock (Microsoft) and a Hard Place (Apple) I created an access module for VLC to allow VLC to support the Apple/iPhone HTTP Live Streaming protocol. There is also a basic outline of how HTTP Live streaming works in that post.

Thanks go out to ioncannon.net's post on his FFMPEG based HTTP live streamer which gave me a tool to play around with/learn from before attempting this patch for VLC.  It is also a good place to look for more  information on HTTP live streaming.

Here is the patch:

This patch should no longer be needed, as livehttp is included in the latest VLC git trunk, and should also be included in the nightly builds.

vlc-livehttp.patch



Notes:

• This has only been tested using H.264 w/MP3 or AAC audio using mux=ts, and raw MP3 using mux=raw
• I've been mostly an FFMPEG guy till now, so forgive me if my VLC understanding/terminology is somewhat off. 
• This plug-in should support both Live and non-live HTTP Live streaming feeds, depending on the options passed to the module.

Instructions:

The name of the module is livehttp, and is specified by specifying "access=livehttp"

Options:
splitanywhere= (default: false)
Tells livehttp to split the stream anywhere, not just on video keyframes.  Currently required to be set to true for audio-only streams and not recommended (probably won't work) for video streams.

seglen= (default: 10)  
How many seconds of audio/video each segment should contain.  Apple recommends 10, I have been using 5.

numsegs=(default: 0)
  The number of segments to keep in the index file.  The default of 0 keeps all segments in the index (which you would want for non-live streaming).  For live streaming the specification require at least 3.

delsegs= (default: true)
Delete segments as they are no longer needed.   If numsegs=0 this parameter is ignored (as all segments are assumed to be needed)

dst=
This is actually an option to the access std module.  The path of the segment files to write.  The # characters get replaced with the segment number.  So a path of "seg-###.ts" will end up with files called "seg-001.ts, seg-002.ts, seg-003.ts" etc.

index=
The path of the index file to write, which will contain the "playlist" of video/audio segments to stream.  Recommended to end in .m3u8 by specifications.  This is the file the <video> tag should be pointed to.

index-url=
This is the URL that corresponds to the dst above (how a browser would access the dst file).  The # characters get replaced same as in the dst parameter.  Note:  The filename portion of this URL will most likely need to be in the exact same format as the dst parameter.  So for example if dst=/www/seg-##.ts then the index-url should be something like index-url=http://mydomain.com/streams/seg-##.ts  (Note the same number of # characters)

rate-control=(default: false)
If set to false the there is no rate control (the muxer sends the data as fast/slow as it can to the streamer).  If set to true the muxer should do rate-control to control the speed to muxed audio/video is sent to the streamer.  I'm a little unsure what the "best" setting for this parameter is when re-streaming an existing video stream.  I've been leaving it at false...

Examples:
All examples assume the following:

• The Web Server root directory is /var/www
• The domain name of the web server is mydomain.com
• The stream segments &amp; index files will be written into /var/www/streaming/ and will be accessed via http://mydomain.com/streaming/...
• The destination stream name index file will be called "mystream.m3u8"

The following HTML will allow you to view the video based on the above on an iPhone:

<html>
  <head>
    <title>Video Test</title>
    <meta name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;"/>
  </head>
  <body style="background-color:#FFFFFF; ">
    Audio/Video Stream:<br/>
    <center>
      <video width='150' height='150' src="http://mydomain.com/streaming/mystream.m3u8" />
    </center>
  </body>
</html>

Re-stream a live video feed:
vlc -I dummy --mms-caching 0 http://www.nasa.gov/55644main_NASATV_Windows.asx vlc://quit --sout='#transcode{threads=2,width=320,height=240,fps=25,vcodec=h264,vb=256,venc=x264{aud,profile=baseline,level=30,keyint=30,bframes=0,ref=1,nocabac},acodec=mp3,ab=96}:duplicate{dst=std{access=livehttp{seglen=10,delsegs=true,numsegs=5,index=/var/www/streaming/mystream.m3u8,index-url=http://mydomain.com/streaming/mystream-########.ts},mux=ts{use-key-frames},dst=/var/www/streaming/mystream-########.ts}}'
Create a VOD stream:
(Non-live.  When this command finishes, all the segments should have been created and the index file contain pointers to all of them)
vlc -I dummy /var/myvideos/video.mpg vlc://quit --sout='#transcode{threads=2,width=320,height=240,fps=25,vcodec=h264,vb=256,venc=x264{aud,profile=baseline,level=30,keyint=30,bframes=0,ref=1,nocabac},acodec=mp3,ab=96}:duplicate{dst=std{access=livehttp{seglen=10,delsegs=false,numsegs=0,index=/var/www/streaming/mystream.m3u8,index-url=http://mydomain.com/streaming/mystream-########.ts},mux=ts{use-key-frames},dst=/var/www/streaming/mystream-########.ts}}'
Re-stream a live audio feed:
vlc -I dummy --mms-caching 0 http://www.nasa.gov/55644main_NASATV_Windows.asx vlc://quit --sout='#transcode{acodec=mp3,ab=96}:duplicate{dst=std{access=livehttp{seglen=10,delsegs=true,numsegs=5,index=/var/www/streaming/mystream.m3u8,index-url=http://mydomain.com/streaming/mystream-########.mp3},mux=raw,dst=/var/www/streaming/mystream-########.mp3},select=novideo}'



 Possible improvements/fixes:

• Have the module auto-detect audio only streams, so the splitanywhere option is not required.
• I'm not sure I am doing the right thing with the Win32 rename function.  Linux (I believe) allows me to rename a file over an existing file, even if the existing file is in use.  Win32 is not so friendly.  This ability is useful for updating the index file at same time it may be currently being read by the HTTP server serving the files.
• Break the dst= and index= parameter into seperate filename/directory entries, so you only need to specify the filename format once. (instead of once for the dst= parameter, and once for the index-url= parameter)
  

VLC MMS start delay (keep-alive) patch

As noted in the post iPhone HTTP Live Streaming - Between a rock (Microsoft) and a Hard Place (Apple) I had occasion to use VLC to stream an MMSH stream.  It seemed to delay for a few minutes before starting to play the stream and then would play fine.

This problem has been noted for a while with some MMS streams, particular in this thread:
http://forum.videolan.org/viewtopic.php?f=14&t=45844&p=237757
and this bug report:
http://trac.videolan.org/vlc/ticket/2533


Looking through the thread above I noticed that "carver" had done some TCP sniffing and posted the results.  It looked to me like VLC was wanting to use HTTP 1.0 and the server was replying as if it supported HTTP 1.1.  The server was including Content-Length and keep-alive headers which are only supported by HTTP 1.1.  This is really a bug in the server, but it's fixable in VLC by making VLC support enough (keep-alive, content-length) of the HTTP 1.1 standard to work with these servers.

Here is a patch to VLC that seems to fix the issue by making VLC look for the Content-Length header and only reading that number of bytes from the server (instead of waiting for more until it times out after a few minutes) when reading from an MMSH stream:

This patch should no longer be needed, as is included in the latest VLC git trunk, and should also be included in the nightly builds.

vlcmms.patch