Advanced Video Processing for Ruby on Rails with FFmpeg Filters and Frei0r Plugin Effects

This is the second article in my series on video processing with Rails. Here I’m going to describe more advanced things like video and audio filters, reading video metadata, taking screenshots for each second of video, and monitoring video processing progress in real time with Action Cable. All these features are going to be added to the application I developed in my first article.

I’ll put the results of my video processing experiments in my third article.

You can find the full Video Manipulator project in this GitHub repository.

Project Setup

1. Install FFmpeg and all necessary software: the Frei0r plugin, mongoDB and redis, ruby, rails, and rvm.

If you’re running macOS, you can use homebrew for installation.

brew install ffmpeg --with-fdk-aac --with-frei0r --with-libvo-aacenc --with-libvorbis --with-libvpx --with-opencore-amr --with-openjpeg --with-opus --with-schroedinger --with-theora --with-tools

Even though you might not need all of these options for the current project, I’ve tried to include as many options as possible to avoid any issues.

2. Fix frei0r.

The first time I installed ffmpeg with Frei0r, its effects didn’t work well. To check if you’re having such an issue, run ffmpeg with frei0r effects:

ffmpeg -v debug -i 1.mp4 -vf frei0r=glow:0.5 output.mpg

You might see something like this:

[Parsed_frei0r_0 @ 0x7fe7834196a0] Looking for frei0r effect in '/Users/user/.frei0r-1/lib/glow.dylib' [Parsed_frei0r_0 @ 0x7fe7834196a0] Looking for frei0r effect in '/usr/local/lib/frei0r-1/glow.dylib' [Parsed_frei0r_0 @ 0x7fe7834196a0] Looking for frei0r effect in '/usr/lib/frei0r-1/glow.dylib'

As you can see, FFmpeg can’t find the necessary libraries. I can't blame it, since these libraries are stored with a different extension. If you try ls -l /usr/local/lib/frei0r-1/, you’ll see that the plugins are installed with the *.so extension.

On my machine (macOS 10.12.5 running ffmpeg version 3.3.2 with frei0r-1.6.1), I used the following line of code to solve the problem:

for file in /usr/local/lib/frei0r-1/*.so ; do cp $file "${file%.*}.dylib" ; done

I also had to set this environment variable with a path to the folder where the .dylib files are stored:

​export FREI0R_PATH=/usr/local/Cellar/frei0r/1.6.1/lib/frei0r-1

This solution looks like some strange hack, but finally I got Frei0r working well.

Development

1. Video and Audio Filters

Let's first define all effects that can be used with this app inside the EncodingConstants module:

module EncodingConstants

 #  ...



 VIDEO_EFFECTS = {

   sepia:

     %w[

       -filter_complex colorchannelmixer=.393:.769:.189:0:.349:.686:.168:0:.272:.534:.131

       -c:a

       copy

     ],

   black_and_white: %w[-vf hue=s=0 -c:a copy],

   vertigo: %w[-vf frei0r=vertigo:0.2 -c:a copy],

   vignette: %w[-vf frei0r=vignette -c:a copy],

   sobel: %w[-vf frei0r=sobel -c:a copy],

   pixelizor: %w[-vf frei0r=pixeliz0r -c:a copy],

   invertor: %w[-vf frei0r=invert0r -c:a copy],

   rgbnoise: %w[-vf frei0r=rgbnoise:0.2 -c:a copy],

   distorter: %w[-vf frei0r=distort0r:0.05|0.0000001 -c:a copy],

   iirblur: %w[-vf frei0r=iirblur -c:a copy],

   nervous: %w[-vf frei0r=nervous -c:a copy],

   glow: %w[-vf frei0r=glow:1 -c:a copy],

   reverse: %w[-vf reverse -af areverse],

   slow_down: %w[-filter:v setpts=2.0*PTS -filter:a atempo=0.5],

   speed_up: %w[-filter:v setpts=0.5*PTS -filter:a atempo=2.0]

 }.freeze



 AUDIO_EFFECTS = {

   echo: %w[-map 0 -c:v copy -af aecho=0.8:0.9:1000|500:0.7|0.5],

   tremolo: %w[-map 0 -c:v copy -af tremolo=f=10.0:d=0.7],

   vibrato: %w[-map 0 -c:v copy -af vibrato=f=7.0:d=0.5],

   chorus: %w[-map 0 -c:v copy -af chorus=0.5:0.9:50|60|40:0.4|0.32|0.3:0.25|0.4|0.3:2|2.3|1.3]

 }.freeze



 EFFECT_PARAMS = VIDEO_EFFECTS.merge(AUDIO_EFFECTS).freeze



 ALLOWED_EFFECTS = EFFECT_PARAMS.keys.map(&:to_s).freeze



 # ...

end

Suggested improvement: All these filters here are defined as constants for the sake of simplicity since this is a demo application. In production, however, I’d prefer creating a hierarchy of classes for these filters. This could let us set all filter parameters from the frontend and validate them on the backend.

In my demo application, I made it possible for a user to specify multiple effects for video processing. Let's add an array attribute to the Video model that will contain all effects selected by the user:

 # All user-applied effects are stored as array

 field :effects, type: Array, default: []

We should also make sure that a user can select only available effects:

 validate :effects_allowed_check

 private

 def effects_allowed_check

   effects.each do |effect|

     unless ::VideoUploader::ALLOWED_EFFECTS.include?(effect)

       errors.add(:effects, :not_allowed, effect: effect.humanize)

     end

   end

 end

In the create/update video form, we have the following as a set of checkboxes (the materialize-sass gem is used for styling):

<div class="row">

   <span>Effects</span>

   <% ::VideoUploader::ALLOWED_EFFECTS.each do |effect| %>

     <div class="input-field">

       <%= f.check_box :effects, {id: "effects_#{effect}", multiple: true, checked: f.object.effects.include?(effect) }, effect, nil %>

       <%= f.label :effects, effect.humanize, id: "effects_#{effect}", for: "effects_#{effect}" %>

     </div>

   <% end %>

 </div>

Next, in the VideoUploader, we have to include our EncodingConstants module and add an encoding step for effects processing after a normalization step. The logic for the normalization_step was defined in my first article –  it just converts an input video to the mp4 format.

 # ...

 include ::EncodingConstants

 process encode: [:mp4, PROCESSED_DEFAULTS]

 def encode(format, opts = {})

   normalization_step(format, opts)

   apply_effect_steps(format, opts)

 end

 private

 ​ def audio_effects

   model.effects & AUDIO_EFFECTS.keys.map(&:to_s)

 end

 def video_effects

   model.effects & VIDEO_EFFECTS.keys.map(&:to_s)

 end



 def ordered_effects

   audio_effects + video_effects

 end



 def apply_effect_steps(format, opts)

   ordered_effects.each do |effect|

     encode_video(

       format,

       opts.merge(

         processing_metadata: { step: "apply_#{effect}_effect" }

       )

     ) do |_, params|

       params[:custom] = EFFECT_PARAMS[effect.to_sym]

     end

   end

 end



 # ...

You can find the full video uploader code here.

All the effects selected by a user are processed one by one. Some combinations of effects can be applied simultaneously during one transcoding operation, while others are not compatible with each other at all. Due to this, we’ll process each separately to make sure that everything is processed properly.

Of course, this is slower than transcoding all at once, but it’s more reliable. Mind you, audio effects are usually applied before any video effects. I’ve had some compatibility issues when applying audio effects after video effects. So this time I changed the order, putting audio effects processing first in the ordered_effects method. The :custom option from the carrierwave-video gem is used to provide filtering parameters to FFmpeg.

Suggested improvement: It’s better to have the ability to chain these effects in a similar way to the Rails ActiveRecord scopes. Chaining lets you define the order of effects and their quantity, which is especially important if these effects have the ability to accept some user-provided filtering parameters.

2. Video watermark

This feature was described in my first article. In this sample project, it’s been moved to a separate processing step in the VideoUploader:

def encode(format, opts = {})

   normalization_step(format, opts)

   apply_effect_steps(format, opts)

   apply_watermark_step(format, opts) if model.watermark_image.path.present?

   # ...

 end

 private

  def apply_watermark_step(format, opts)

   encode_video(

     format,

     opts.merge(processing_metadata: { step: 'apply_watermark' })

   ) do |_, params|

     params[:watermark] ||= {}

     params[:watermark][:path] = model.watermark_image.path

   end

 end

3. Reading video metadata

Now let's add a hash attribute to the Video model; this attribute will be stored together with video metadata. Also, we’ll need to process the callback method that will then save metadata in the Video model. The file_duration will be updated here as well (in the save_metadata method, to be precise).

 # File ffmpeg metadata is stored in hash

 field :metadata, type: Hash, default: {}

 # Callback method

 def save_metadata(new_metadata)

   set(

     metadata: new_metadata,

     file_duration: new_metadata[:format][:duration]

   )

 end

Next, we need to create the ::CarrierWave::Extensions::VideoMetadata module, which implements some metadata reading/extraction logic.

::CarrierWave::Extensions::VideoMetadata has a parameter called read_video_metadata; one of the options of this parameter is save_metadata_method, which contains the name of a callback in a model that’s called with extracted metadata. Its code was made to be compatible with the carrierwave-video gem, which is why it has similar logic to the encode_video method.

Unlike other operations (such as video processing or uploading, for instance), read_video_metadata can’t return progress information (there’s no progress at all – the data reading procedure happens very quickly). But even in this case, we should inform our user when the operation has successfully completed; this way we can maintain a consistent app architecture. To do so, we’ll simply send “1” to indicate 100% progress to the progress callback method at once.

module CarrierWave

 module Extensions

   module VideoMetadata

     def read_video_metadata(format, opts = {})

       # move upload to local cache

       cache_stored_file! unless cached?



       @options = CarrierWave::Video::FfmpegOptions.new(format, opts)



       file = ::FFMPEG::Movie.new(current_path)



       if opts[:save_metadata_method]

         model.send(opts[:save_metadata_method], file.metadata)

       end



       progress = @options.progress(model)



       with_trancoding_callbacks do

         # Here it happens instantly, so we provide value for 100%

         progress&.call(1.0)

       end

     end

   end

 end

End

In the VideoUploader, we should apply this logic to the encode method after the watermarking step that we took earlier:

 def encode(format, opts = {})

   # ...

   apply_watermark_step(format, opts) if model.watermark_image.path.present?

   read_video_metadata_step(format, opts)

   # ...

 end



 private



 def read_video_metadata_step(format, opts)

   read_video_metadata(

     format,

     opts.merge(

       save_metadata_method: :save_metadata,

       processing_metadata: { step: 'read_video_metadata' }

     )

   )

 end

4. Generating thumbnails

Now we’re going to add the Thumbnail model, which is embedded in the Video model. Each Thumbnail record will be stored as one thumbnail image:

class Thumbnail

 include Mongoid::Document



 embedded_in :video



 # Here background uploading is not used since this entity is created in

 # background already

 mount_uploader :file, ::ImageUploader

end

The next step is to turn back to the Video model again. To do so, we need to define the thumbnails association and the callback method to save our thumbnails:

embeds_many :thumbnails



 # Config option: generate thumbnails for each second of video

 field :needs_thumbnails, type: Boolean, default: false



 # Callback method

 def save_thumbnail_files(files_list)

   files_list.each do |file_path|

     ::File.open(file_path, 'r') do |f|

       thumbnails.create!(file: f)

     end

   end

 end

Next, let's place all thumbnail creation logic in the CarrierWave::Extensions::VideoMultiThumbnailer module. This module will be designed to be compatible with the carrierwave-video gem’s logic.

This file is too large to be fully presented within the article. To see the full file, check out this repository.

Here’s how you can describe such a module:

module CarrierWave

 module Extensions

   module VideoMultiThumbnailer

     def create_thumbnails_for_video(format, opts = {})

       prepare_thumbnailing_parameters_by(format, opts)

       # Create temporary directory where all created thumbnails will be saved

       prepare_tmp_dir



       run_thumbnails_transcoding



       # Run callback for saving thumbnails

       save_thumb_files

       # Remove temporary data

       remove_tmp_dir

     end



     private



     def run_thumbnails_transcoding

       with_trancoding_callbacks do

         if @progress

           @movie.screenshot(*screenshot_options) do |value|

             @progress.call(value)

           end

           # It’s an ugly hack, but this operation returned this in the end

           # 0.8597883597883599 that is not 1.0

           @progress.call(1.0)

         else

           @movie.screenshot(*screenshot_options)

         end

       end

     end



  # Some method definitions are skipped here. Look in the repository for more details.



     # Thumbnails are sorted by their creation date

     # to put them in chronological order.

     def thumb_file_paths_list

       Dir["#{tmp_dir_path}/*.#{@options.format}"].sort_by do |filename|

         File.mtime(filename)

       end

     end



     def save_thumb_files

       model.send(@options.raw[:save_thumbnail_files_method], thumb_file_paths_list)

     end



     def screenshot_options

       [

         "#{tmp_dir_path}/%d.#{@options.format}",

         @options.format_params,

         {

           preserve_aspect_ratio: :width,

           validate: false

         }

       ]

     end

   end

 end

end

```

The module I’ve described above uses the screenshot method directly from the streamio-ffmpeg gem. In our case, it generates a thumbnail for each second of video.

Also, since this operation is local, we have to create, use, and remove temporary directories for our generated thumbnails with the tmp_dir_path. All generated images are sent to the model’s callback, which is specified in the save_thumbnail_files_method parameter.

For some strange reason, its progress value doesn’t always return 1.0 in the end (returning 0.8597, for example), which is why I had to manually set the 1.0 progress value after the operation is finished.

Inside VideoUploader, we include the CarrierWave::Extensions::VideoMultiThumbnailer module and update the encode method using the create_thumbnails_step:

 include ::CarrierWave::Extensions::VideoMultiThumbnailer



 def encode(format, opts = {})

   # ...

   read_video_metadata_step(format, opts)

   create_thumbnails_step('jpg', opts) if model.needs_thumbnails?

 end



 private



 def create_thumbnails_step(format, _opts)

   create_thumbnails_for_video(

     format,

     progress: :processing_progress,

     save_thumbnail_files_method: :save_thumbnail_files,

     resolution: '300x300',

     vframes: model.file_duration, frame_rate: '1', # create thumb for each second of the video

     processing_metadata: { step: 'create_video_thumbnails' }

   )

 end

5. Progress calculation logic

Since each transcoding operation can return progress values, I decided to play with this also. Our video processing consists of many steps and each step has its own progress value, so I wanted to calculate the overall progress of all these steps.

When I was researching this, I noticed that the progress callback is called very often (around one or two times per second). Since I wanted to track this progress in a database, I decided to update it every 10% of each step’s transcoding progress.

In the EncodingConstants, inside PROCESSED_DEFAULTS, we have the definition of the progress callback method (for more information, check out this repository):

PROCESSED_DEFAULTS = {

   # ...

   progress: :processing_progress

 }.freeze

You might have also noticed that we have the following inside each processing step:

opts.merge(processing_metadata: { step: '<step-name-here>' })

This is usually passed inside format_options to the processing_progress callback method in the Video model:

 def processing_progress(format, format_options, new_progress)

   ProgressCalculator.new(self, format, format_options, new_progress).update!

 end

The processing_progress module described above uses the ProgressCalculator instance to handle all these progress percentage calculations, database updates, and WebSocket notifications:

class ProgressCalculator

 attr_reader :video, :format, :format_options, :new_progress



 def initialize(video, format, format_options, new_progress)

   @video = video

   @format = format

   @format_options = format_options

   @new_progress = new_progress

 end



 def update!

   update_progress_data if shoud_update?

 end



 private



 def update_progress_data

   # Had to use atomic set operation here since normal update

   # has been setting file_processing to false while processing still not finished

   step_metadata.set(progress: new_progress.to_f)

   video.set(progress: calculate_overall_progress.to_f)

   notify_about_progress

 end



 def step

   format_options[:processing_metadata][:step]

 end



 def step_metadata

   video.processing_metadatas.find_or_create_by!(step: step) do |pm|

     pm.format = format

   end

 end



 def diff

   new_progress.to_f - step_metadata.progress.to_f

 end



 def shoud_update?

   # Update this value only each 10 percent or when processing is finished

   diff >= 0.1 || (new_progress.to_i == 1)

 end



 def steps_count

   # Normalize step + 1

   # Effects steps + effects.count

   # Watermark step + 1

   # Read video metadata step + 1

   # Generate thumbnails step + 1

   count = ::VideoUploader::OBLIGATORY_STEPS.count + video.effects.count

   count += 1 if video.watermark_image.path.present?

   count += 1 if video.needs_thumbnails?

   count

 end



 def calculate_overall_progress

   video.processing_metadatas.sum(:progress) / steps_count

 end



 def notify_about_progress

   ::ActionCable.server.broadcast(

     'notifications_channel',

     progress_payload

   )

 end



 def progress_payload

   ::CableData::VideoProcessing::ProgressSerializer.new(video).as_json

 end

end

The notify_about_progress method uses the Rails Action Cable for updating progress on the video details page. I’ll get back to this a bit later.

6. Carrierwave video callbacks

The carrierwave-video gem has some nice features: the before_transcode and after_transcode callbacks (find out more here).

I use this gem to clean all previously generated thumbnails and progress metadata if a video file has been updated.

Here’s the callback in the Video model:

 # before_transcode callback

 # Callback method accepts format and raw options but we do not need them here

 def processing_init_callback(_, _)

   # Clean data

 end

This method is called while performing the normalization_step inside the VideoUploader before a new transcoding session:

def normalization_step(format, opts)

   encode_video(

     format,

     opts.merge(

       processing_metadata: { step: 'normalize' },

       callbacks: {

         # Clean previous progress data if encoding happens for existing video record

         # Callback method at model

         before_transcode: :processing_init_callback

       }

     )

   )

 end

7. Custom video saving worker

We’re going to use background workers (or background jobs) for video processing and video storing; workers are the processes that run in the background and don’t have any direct interaction with a user (in production, this worker may upload files to a cloud like Amazon S3, for instance). In other words, these workers are hidden from the user’s eyes.

Only the worker knows when the processing has completed (since it’s responsible for it). That’s why we should send notifications from the worker to let us update the page containing information about the video in the user’s browser.   

We’re also going to use sidekiq workers directly (without Rails’ ActiveJob) because the carrierwave_backgrounder gem’s workers, which are responsible for uploading a video, are not compatible with ActiveJob.

Let’s create a custom worker that lets us send notifications. To do so, we have to redefine the gem’s perform method to run custom callbacks in the run_video_processing_chain_completed_callbacks.

​class VideoSaverWorker < ::CarrierWave::Workers::StoreAsset

 def perform(*args)

   super(*args)

   run_video_processing_chain_completed_callbacks

 end



 private



 def run_video_processing_chain_completed_callbacks

   record.processing_completed_callback if record.respond_to?(:processing_completed_callback)

 end

end

The processing_completed_callback is defined in the Video model. It sends the Action Cable WebSocket notifications informing that processing has finished.

 def processing_completed_callback

   ::ActionCable.server.broadcast(

     'notifications_channel',

     ::CableData::VideoProcessing::CompletedSerializer.new(self).as_json

   )

 end

8. Action Cable and WebSocket notifications

We’ll use Action Cable for instantly processing progress updates via WebSockets. We can use this command to generate the notifications channel: rails generate channel Notifications.

Since we also want to use redis for our Action Cable WebSocket notifications in the development environment, we have to change the config config/cable.yml:

development:

 adapter: redis

 url: redis://localhost:6379/1

 ...

Since we don’t have any users or authorizations, all browsers that open our site will be automatically subscribed to the notifications_channel.

We then should include the following code into app/assets/javascripts/channels/notifications.coffee:

​App.notifications = App.cable.subscriptions.create "NotificationsChannel",

 # ...

 received: (data) ->

   # Called when there's incoming data on the websocket for this channel

   if data['processing_completed'] == false

     $("#video_progress_" + data['id']).replaceWith($(data['html']))

   else

     $("#video_info_" + data['id']).replaceWith($(data['html']))

     if $('#thumbnails').length

       # Init thumbnails carousel when they are present only

       $('.carousel').carousel({})

After each WebSoсket notification is received, it updates the video processing progress information. When the processing is finished, it replaces all page content with the data delivered by the Action Cable.

Normally you should send JSON data over WebSockets and all styling and formatting should happen on the frontend. But let's skip this part. I’ve mentioned it just to describe the general idea.

The channel is defined here on the backend: `app/channels/notifications_channel.rb`:

​class NotificationsChannel < ApplicationCable::Channel

 def subscribed

   stream_from 'notifications_channel'

 end

 ...

end

In our code, we have two places where we send WebSocket notifications from the server to the notifications_channel:

  • ProgressCalculator#notify_about_progress during each progress update

  • Video#processing_completed_callback when processing has been finished and the file is ready.

The Processing Progress Action Cable Notification uses the CableData::VideoProcessing:: ProgressSerializer to generate the notification payload:

module CableData

 module VideoProcessing

   class ProgressSerializer < ActiveModel::Serializer

     attributes :html, :processing_completed, :id



     def html

       # INFO: This is made for simplicity

       #       But in real applications it’s better to send

       #       JSON data via action cable only and process

       #       all styling and markup on frontend

       ApplicationController.renderer.render(

         locals: { video: object },

         partial: 'videos/progress'

       )

     end



     def processing_completed

       false

     end



     def id

       object.id.to_s

     end

   end

 end

end

At the same time, the Processing Completed Action Cable Notification uses  CableData::VideoProcessing::CompletedSerializer to generate the notification payload:

```

module CableData

 module VideoProcessing

   class CompletedSerializer < ActiveModel::Serializer

     attributes :html, :processing_completed, :id



     def html

       # INFO: This is made for simplicity

       #       But in real applications it’s better to send

       #       JSON data via action cable only and process

       #       all styling and markup on frontend

       ApplicationController.renderer.render(

         locals: { video: object },

         partial: 'videos/info'

       )

     end



     def processing_completed

       true

     end

     def id

       object.id.to_s

     end

   end

 end

end

Today we learned:

      1) The types of FFmpeg video and audio filters as well as the types of Frei0r plugin effects

      2) How to process these filters using the carrierwave-video gem

      3) How to read and save video metadata using the streamio-ffmpeg and carrierwave-video gems

      4) How to generate thumbnails for each second of your video with the streamio-ffmpeg gem’s FFMPEG::Movie#screenshot method

      5) How to calculate video processing progress for each processing step with the carrierwave-video gem’s progress callback method

      6) How to perform real-time progress bar updates through WebSockets with the Rails Action Cable

      7) How to use transcoding callbacks from the carrierwave-video gem

      8) How to add the custom VideoSaverWorker as a video saver worker instead of the default carrierwave_backgrounder gem worker to send WebSocket notifications when a video has finished processing and being saved.

As I mentioned, this is merely a sample application with a basic set of features. It could be further improved to be production-ready. The following things may require some further enhancement:

  • Error handling – we can make this more user-friendly. For now, users aren’t informed about any mistakes and all errors are silently processed inside the background worker.

  • For the sake of simplicity and ease of use, all filters have been defined as constants. But if I were to develop a real production app, I’d create a hierarchy of classes for these filters or even using a factory pattern. This would let us set all filter parameters from the frontend and validate that on the backend, which makes for much better customization for users.

  • It would be more effective to have the ability to chain these effects in a similar way to Rails Active Record scopes. This could make it possible for a user to define the order and quantity of video effects (especially if these effects had the ability to accept user-provided filtering parameters).

Next time, in my third article, I’ll examine several of my experiments with FFmpeg video and audio filters and Frei0r plugin effects.

3.7/ 5.0
Article rating
13
Reviews
Remember those Facebook reactions? Well, we aren't Facebook but we love reactions too. They can give us valuable insights on how to improve what we're doing. Would you tell us how you feel about this article?