Github Twitter 2 Delicious Linkedin Stackoverflow Google plus new 32px Rss

Video Encoding with Uploadify, CarrierWave and Zencoder

For a recent Rails 3.1 project one of the features requested was to allow users to painlessly upload videos and have them all look consistent. We choose Zencoder for our video encoding service and Rackspace Cloud Files for storage. We were already using the gem CarrierWave for attachments and the jQuery plugin Uploadify for uploading files.

Having large files uploaded through Uploadify works really well. As soon as a file is selected, it starts to upload; this is a much better experience than making a user wait while a video uploads over a standard request. This post will outline all the steps to get a video uploaded, encoded, and displayed to your users.

This is a short rundown of what will happen:

  1. User uploads video using Uploadify
  2. Carrierwave uploads the video to Rackspace
  3. After upload is complete a video encoding job is submitted to Zencoder
  4. Zencoder retrieves the uploaded video file from Rackspace
  5. Zencoder encodes the video and replaces the uploaded file on Rackspace
  6. Zencoder calls back to the server to notify that the job has completed
  7. Encoded video is now available to user
The complete source code for this blog post is available on my Github account. You can just update it with your own Zencoder and Rackspace account credentials and it should work.

https://github.com/nick-desteffen/carrierwave_zencoder_example

Create project

Start out with a fresh Rails 3.1 project.

$ rails new carrierwave_zencoder_example
$ rm carrierwave_zencoder_example/public/index.html

Add Javascript and Gem dependencies

Lets get all the dependencies out of the way first.

  1. Download Uploadify and extract it. The version we used was 2.1.4. Put the extracted folder under /vendor/assets/javascripts
  2. Update your application.css manifest file to include the Uploadify stylesheet
    /app/assets/stylesheets/application.css
            *= require_self
            *= require_tree . 
            *= require jquery.uploadify-v2.1.4/uploadify
          
  3. Update your application.js manifest file to include all the Uploadify javascript files
    /app/assets/javascripts/application.js
            //= require jquery
            //= require jquery_ujs
            //= require_tree .
            //= require jquery.uploadify-v2.1.4/jquery.uploadify.v2.1.4.min
            //= require jquery.uploadify-v2.1.4/swfobject
          
  4. Add the following gems to your Gemfile
    /Gemfile
            gem "carrierwave"
            gem "fog"
            gem "zencoder"
          
  5. Run:
    $ bundle install

Configure Rackspace & CarrierWave

Now that we have CarrierWave installed we need to configure it to use Rackspace to store uploaded videos. CarrierWave uses the gem called Fog to manage remote file storage. This allows CarrierWave to use any storage backend. Fog has support for Amazon S3 and Google Storage for Developers as well, either of those options can easily be substituted.

I'm going to assume you have already signed up for Rackspace Cloud Files:

  1. Log in and click on Your Account and then API Access and make note of your API key.
  2. Add a container called blog.uploads (or whatever your in the mood for).
  3. Click on the container you created in the listing and you should be able to change some properties:
    1. Check the box Publish to CDN.
    2. There should now be a CDN URL listed, make note of that.
  4. CarrierWave is best configured using a Rails initalizer. Under /config/initializers create a new file named carrierwave.rb and put the following code into it.
/config/initalizers/carrierwave.rb
    CarrierWave.configure do |config|
      if Rails.env.test?
        config.storage = :file
        config.enable_processing = true
      else
        config.storage = :fog
        config.fog_credentials = {
          :provider           => 'Rackspace',
          :rackspace_username => 'YOUR_RACKSPACE_USERNAME',
          :rackspace_api_key  => 'YOUR_RACKSPACE_API_KEY'
        }
        config.fog_directory = 'blog.uploads'
        config.fog_host = 'YOUR_RACKSPACE_CDN_URL'
      end
    end
  
(Update it with the username you used when signing up for Rackspace Cloud Files, the API key fro your account. Under config.fog_host put the CDN URL.)

If you want files served over SSL just update the URL's second subdomain to be "ssl".
Example:
non secure: http://c293023.r87.cf2.rackcdn.com
ssl: https://c744687.r87.ssl.rackcdn.com

Configure Zencoder

I'm going to assume you have a Zencoder account.

  1. Login and click API. Take note of the key.
  2. Create a Rails initalizer for zencoder.
/config/initalizers/zencoder.rb
    Zencoder.api_key = 'YOUR_API_KEY'
  

Add Rack middleware to allow file uploads through flash

Rails has built in cross site request forgery protection. When a form is submitted a token is sent along with it. Since Uploadify is flash based we need to create a Rack middleware that tacks this parameter on to the request from flash.

I followed the instructions in this post by Damian Galarza on how to do this.
He gives a very thorough explanation of uploading to Rails using Uploadify, I'm gonna shorten it a bit to just include all the code that's needed to get it up and running.

  1. Create a a middleware folder under /app and a new file named flash_session_cookie_middleware.rb
  2. Here is the middleware code:
    /app/middleware/flash_session_cookie_middleware.rb
            require 'rack/utils'
    
            class FlashSessionCookieMiddleware
              def initialize(app, session_key = '_session_id')
                @app = app
                @session_key = session_key
              end
    
              def call(env)
                if env['HTTP_USER_AGENT'] =~ /^(Adobe|Shockwave) Flash/
                  req = Rack::Request.new(env)
                  env['HTTP_COOKIE'] = [ @session_key,
                                         req.params[@session_key] ].join('=').freeze unless req.params[@session_key].nil?
                  env['HTTP_ACCEPT'] = "#{req.params['_http_accept']}".freeze unless req.params['_http_accept'].nil?
                end
    
                @app.call(env)
              end
            end
          
  3. Update your config/application.rb and tell it to autoload the middleware.
    /config/application.rb
            config.autoload_paths += %W(#{config.root}/app/middleware/**/*)
          
  4. Edit your session_store initializer to include middleware.
    /config/initalizers/session_store.rb
            Rails.application.config.middleware.insert_before(
              ActionDispatch::Session::CookieStore,
              FlashSessionCookieMiddleware,
              Rails.application.config.session_options[:key]
            )
          

Create a Video model

Create a model to store our uploaded video information. It will have an attachment column for the filename used by CarrierWave, a column to put the output id from Zencoder and a processed boolean flag.

  1. $ bundle exec rails g model video attachment:string zencoder_output_id:string processed:boolean
  2. Migrate your database:
    $ bundle exec rake db:migrate
  3. In your video.rb model add a method to update the processed flag.
    /app/models/video.rb
            class Video < ActiveRecord::Base  
    
              attr_accessible :attachment
    
              validates_presence_of :attachment
    
              def processed!
                update_attribute(:processed, true)
              end
    
            end
          

Create Zencoder callback controller

This controller will handle the callback from Zencoder when an encoding job is complete. Zencoder sends data as JSON.

  1. $ bundle exec rails g controller zencoder_callback
  2. Add a route to the create action for the callback:
    /config/routes.rb
            post "zencoder-callback" => "zencoder_callback#create", :as => "zencoder_callback"
          
  3. Put the following code in your controller:
    /app/controllers/zencoder_callback_controller.rb
            class ZencoderCallbackController < ApplicationController
    
              skip_before_filter :verify_authenticity_token
    
              def create
                zencoder_response = ''
                sanitized_params = sanitize_params(params)
                sanitized_params.each do |key, value|
                  zencoder_response = key.gsub('\"', '"')
                end
    
                json = JSON.parse(zencoder_response)
                output_id = json["output"]["id"]
                job_state = json["output"]["state"]
    
                video = Video.find_by_zencoder_output_id(output_id)
                if job_state == "finished" && video
                  video.processed!
                end
    
                render :nothing => true
              end
    
              private
    
              def sanitize_params(params)
                params.delete(:action)
                params.delete(:controller)
                params
              end
    
            end
          

    Whats going on in the controller?
    1. Since Zencoder will be sending a POST request and won't have the authenticity token we should skip checking it.
    2. Remove the controller and action parameters from the params hash since they aren't needed, this will leave just the JSON from Zencoder.
    3. We had to unescape the string prior to parsing it with JSON.
    4. After parsing it you should be able to pull the output id and state. If the state is finished and a video was found via the output id then flag it as processed using the method we created on the video model.

Create CarrierWave uploader

One of the things I really like about CarrierWave is that it pushes all the attachment processing code off into it's own reusable class called an uploader. Next up is creating an uploader to handle videos.

  1. $ bundle exec rails g uploader video
  2. Update the generated uploader, make sure you remove the storage :file line, we configured this in the CarrierWave initalizer
  3. Update the extension whitelist to include video formats you accept
  4. Uploaders have callbacks, similar to ActiveRecord models. This is where we'll tell Zencoder the file has been uploaded.
  5. Creating a Zencoder job:
    1. Input should be the location of the video that was uploaded
    2. Outputs should be an array. You can tell Zencoder to encode multiple formats, just label each hash of options appropriately (web, mobile, etc.)
    3. After a job is submitted Zencoder will respond with an array of jobs that it has created. We need to loop over the array of jobs from Zencoder and grab the output id and update the model.
    4. The notifications option is where we'll tell Zencoder to we want to receive the callback. This is the controller we created earlier.
      In order to receive the callback Zencoder must be able to connect to your server, so it needs to be on the open internet
      /config/application.rb
                  config.action_mailer.default_url_options = {:host => "your_ip_address"}
                
    5. Include the Rails.application.routs.url_helpers module, since routes aren't available in uploaders.
    6. Set the default url to be the host of your callback, if you want to just use the same host as your email just add: Rails.application.routes.default_url_options = ActionMailer::Base.default_url_options
    7. Use the ssl protocol in the callback url if your site is running ssl.
/app/uploaders/video_uploader.rb
    class VideoUploader < CarrierWave::Uploader::Base
      include Rails.application.routes.url_helpers

      Rails.application.routes.default_url_options = ActionMailer::Base.default_url_options

      after :store, :zencode

      def store_dir
        "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
      end

      def extension_white_list
        %w(mov avi mp4 mkv wmv mpg)
      end

      def filename
        "video.mp4" if original_filename
      end

      private 

      def zencode(args)
        input = "cf://rackspace_username:rackspace_api_key@blog.uploads/uploads/video/attachment/#{@model.id}/video.mp4"
        base_url = "cf://rackspace_username:rackspace_api_key@blog.uploads/uploads/video/attachment/#{@model.id}"
        
        zencoder_response = Zencoder::Job.create({
          :input => input,
          :output => [{
            :base_url => base_url,
            :filename => "video.mp4",
            :label => "web",
            :notifications => [zencoder_callback_url(:protocol => 'http')],
            :video_codec => "h264",
            :audio_codec => "aac",
            :quality => 3,
            :width => 854,
            :height => 480,
            :format => "mp4",
            :aspect_mode => "preserve",
            :public => 1
          }]
        })

        zencoder_response.body["outputs"].each do |output|
          if output["label"] == "web"
            @model.zencoder_output_id = output["id"]
            @model.processed = false
            @model.save(:validate => false)
          end
        end
      end

    end
  
Mount the uploader on the video model to the attachment column. Put this line somewhere in your model.
/app/models/video.rb
      mount_uploader :attachment, VideoUploader
  

Create Videos controller

We can now create a controller to tie it all together

  1. $ bundle exec rails g controller videos index show new
  2. Update your routes:
    /config/routes.rb
            resources :videos
            root :to => "videos#index"
          
  3. Your controller is pretty basic, the create action just needs to return success or failure on uploading the video. The Uploadify script has success and failure functions that we'll use.
    /app/controllers/videos_controller.rb
            class VideosController < ApplicationController
    
              def index
              end
    
              def show
                @video = Video.find(params[:id])
              end
    
              def new
                @video = Video.new
              end
    
              def create
                @video = Video.new(:attachment => params[:attachment])
                if @video.save
                  render :nothing => true
                else
                  render :nothing => true, :status => 400
                end
              end
    
            end
          
  4. /app/views/videos/index.html.erb
            <%= link_to "New Video", new_video_path %>

    <% Video.all.each do |video| %> <%= link_to "Video ##{video.id}", video_path(video) %>
    <% end %>
  5. The show view will either show a processing message or the video.
    /app/views/videos/show.html.erb
            <% if @video.attachment_url.present? && @video.processed? %>
              
            <% else %>
              

    Please be patient while we process this video

    <% end %>
    For a more full featured video player with Flash based fallback see http://videojs.com/

  6. Your new view doesn't need a form, just an element in the DOM to attach the flash uploader to. Use the asset_path helper for the uploadify.swf and cancel.png paths, this is used to correctly reference the files in the asset pipeline.
    /app/videos/new.html.erb
            

    Upload your Video

    <%= link_to "Video Listing", videos_path %> <script type="text/javascript"> $(function() { <% session_key = Rails.application.config.session_options[:key] %> var uploadify_script_data = {}; var csrf_token = $('meta[name=csrf-token]').attr('content'); var csrf_param = $('meta[name=csrf-param]').attr('content'); uploadify_script_data[csrf_param] = encodeURI(encodeURIComponent(csrf_token)); uploadify_script_data['<%= session_key %>'] = '<%= cookies[session_key] %>'; $('#video_attachment').uploadify({ uploader : '<%= asset_path("jquery.uploadify-v2.1.4/uploadify.swf") %>', script : '<%= videos_path %>', wmode : 'transparent', cancelImg : '<%= asset_path("jquery.uploadify-v2.1.4/cancel.png") %>', fileDataName : 'attachment', scriptData : uploadify_script_data, auto : true, buttonText : 'Browse', onAllComplete : function(event, data){ alert("Success! Please be patient while the video processes."); }, onError: function(event, ID, fileObj, errorObj){ alert("There was an error with the file you tried uploading.\n Please verify that it is the correct type.") } }); }); </script>

Try it out!

Start your server and try uploading a video!

The full source code for this is located at:
https://github.com/nick-desteffen/carrierwave_zencoder_example

(The video format in this post only works in Chrome with the HTML5 Video tag used)
Good luck!

Related Links

Comments

heff
heff says:
09/20/2011 12:12pm

Nice! Great example. Thanks for publishing this.

ahmy
ahmy says:
09/21/2011 05:36am

Does uplodify support direct upload to S3?

Nick DeSteffen
09/21/2011 06:45pm

I personally haven't tried uploading directly to S3. You could take a look at this blog post, they seem to have had some success.

Dave Preston
03/05/2012 12:17am

This blog post was extremely helpful thank you, but I still have one question. zencoder can create thumbnail images with only a slight change to your above code, but I'm not clear on how to get the thumbnail url into the attachment model since it's not a "true" model. My attempt so far (adding version code) just ends up prepending thumb_ to the video but leaves .mp4 as the extension. Have you run into this and/or have any suggestions?

BTW, adapting your code to use s3 was not difficult. Just had to update the fog_credentials for AWS thanks again.

Nick DeSteffen
03/10/2012 12:37pm

Dave,

Glad it was helpful to you. I didn't really do anything with the thumbnails. I believe CarrierWave supports multiple attachments per model. What you could do is create another column and mount a Thumbnail uploader to that. Then put some logic in place to retrieve the thumbnail from Zencoder after the encoding process is complete. I haven't tried this, but it's worth a shot.

Good luck, and let me know if it works!

Nick

Morgan
Morgan says:
03/25/2012 11:46pm

Hi Nick,

Firstly, thanks for the tutorial its great to find one specifically for cloudfiles. I have gone trough the tutorial countless times over the past week and I keep getting stuck at the zencoder response. Do you think you could please have a look at my log file? http://cl.ly/3Z3w2p1W3K3D0J0a1F2J

Thanks again.

Morgan
Morgan says:
03/26/2012 05:23am

I found that the problem was because of postgres so after trying some of the suggested fixes I decided to just use mysql instead but now I get the following error:

http://cl.ly/0z143E0y1m1X2p0O2f1Q

Barry Owen
03/26/2012 08:32pm

@Dave Preston - I had the same problem, found a solution here:

https://gist.github.com/1429975

Which I tweaked for my needs to be something like this:

## Create different versions of your uploaded files:
version :videothumb, :if => :is_video? do
  def filename
    File.basename(originalfilename, '.*') +'.png' if originalfilename
  end
end

Hope that helps.

Morgan
Morgan says:
03/27/2012 05:26pm

Hi Barry, Sorry for the newb question but where did you put the above snippet? did you end up getting the thumbs working? I worked out how to send the request fot the thumbs to zencoder which I can see have been created but I havent worked out how to have sent back.

swapnilp
swapnilp says:
08/21/2012 07:07am

Great Tutorial :)

Brad says:
10/15/2012 10:26pm

Sigh I'm just trying to get a plain file uploaded to Rackspace. No fancy video stuff. So I skipped Zencoder and Uploadify. But all I get is Fog::Storage::Rackspace::NotFound in AuthorController#update. Can't figure out why.

Nick DeSteffen
10/16/2012 09:39am

@brad -- I'd double check the Configure Rackspace & Carrierwave step. This person seemed to have the same problem as you.

Nick

Brad says:
10/17/2012 08:26am

@nick - Thanks for commenting promptly. But I had already found that issue when I did a search for the Fog::Storage::Rackspace::NotFound error and that's not the problem I have. I posted my code at http://stackoverflow.com/questions/12922980/fogstoragerackspacenotfound-error-when-using-carrierwave-to-upload-to-rack if you're interested in taking a look. I'm surprised I couldn't find any step by step instructions for fog/carrierwave/rackspace for noobs who just simply uploading files to Rackspace.

Brad says:
10/17/2012 10:48am

I took

def store_dir
  "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end

out of my code because rackspace cloud hosting doesn't recognize directories, only containers. But I still get the same error.

Brad says:
10/24/2012 08:34am

This still isn't working for me. Rackspace says I need to use an auth url, not a CDN url. Do you know what they are talking about?

Shrikant
Shrikant says:
03/05/2014 06:04am

I followed all your steps. But my browse button is not working. I tried to find the solution for it. I got "NetworkError: 404 Not Found - http://localhost:3000/videos/uploadify.swf?preventswfcaching=1394020919388" error in the firebug. How can I resolve it. Thanks in advance.

Nick DeSteffen
03/05/2014 07:41am

@Shrikant - Did you have the problem with the demo application or your own application? Please checkout the demo and see if that works and then start debugging your own using that as a starting point, what you have given me isn't much to go off of, it might be an issue with your flash install.

jose
jose says:
03/20/2014 03:38pm

Great tutorial @Nick, thank you so much, but i am getting a problem VIDEOVIEWDEMO.MP4 (1.36MB) - HTTP ERROR. every time i try to upload the my video. Thank you

Format using Markdown