Experiments with FFmpeg Filters and Frei0r Plugin Effects

This is my third article about video processing with Rails. Here I’m going to show examples of FFmpeg video and audio filters and Frei0r plugin effects. All these features are being added to the app I described in my previous articles:

Read first: Simple Video Processing with FFmpeg and Ruby on Rails. FFmpeg System Usage Limitations

Read second: Advanced Video Processing for Ruby on Rails with FFmpeg Filters and Frei0r Plugin Effects

You can find the project repository on GitHub.

FFmpeg installation

If you’re running macOS, you can use  Homebrew to install FFmpeg:

brew install ffmpeg --with-fdk-aac --with-frei0r --with-libvo-aacenc --with-libvorbis --with-libvpx --with-opencore-amr --with-openjpeg --with-opus --with-schroedinger --with-theora --with-tools

All of the additional parameters I added to the code above might not be needed for your current project. But I’ve included almost all possible parameters to avoid issues when something doesn’t work because some extension isn’t installed.

Fix possible issues with the Frei0r plugin

Frei0r is a cross-platform framework for video effects. With Frei0r, you can avoid issues with recurring re-implementation of standard video effects and easily implement filters and mixers provided by the plugin’s API.

When I installed FFmpeg with Frei0r, the Frei0r effects didn’t work at first. To check if you have such an issue, run FFmpeg and try to add Frei0r effects to a video:

ffmpeg -v debug -i 1.mp4 -vf frei0r=glow:0.5 output.mpg

When you run that command, you might see an error like this:

[Parsed_frei0r_0 @ 0x7fe7834196a0] Looking for frei0r effect in '/Users/user/.frei0r-1/lib/glow.dylib' [Parsed_frei0r_0 @ 0x7fe7834196a0] Looking for frei0r effect in '/usr/local/lib/frei0r-1/glow.dylib' [Parsed_frei0r_0 @ 0x7fe7834196a0] Looking for frei0r effect in '/usr/lib/frei0r-1/glow.dylib'

If you run the ls -l /usr/local/lib/frei0r-1/ command you’ll see that the plugins are installed with the .so extension.
To solve this problem, on my machine (macOS 10.12.5, ffmpeg 3.3.2, frei0r-1.6.1) I copied the .so extensions to .dlib:

for file in /usr/local/lib/frei0r-1/*.so ; do cp $file "${file%.*}.dylib" ; done

You also have to set this environment variable with the path to the folder where the .dylib files are stored:

export FREI0R_PATH=/usr/local/Cellar/frei0r/1.6.1/lib/frei0r-1

This solution looks like some strange hack. Nevertheless, I finally got Frei0r to work well.

FFmpeg and Frei0r filter processing examples

FFmpeg is a video and audio converter that can also process live audio and video. Moreover, you can resize video on the fly with high-quality results. You can find more detailed information about FFmpeg in my previous articles and in the FFmpeg documentation.

For a description of all available FFmpeg filters, check the filters section of the documentation.

Video and audio effects

FFmpeg with the Frei0r plugin is used for video effects. The options -vf and -af specify simple video and audio filters. The -filter_complex option specifies complex filters.

To avoid excessive audio processing, you should copy the audio stream unchanged from your input source file with the option -c:a copy when using all video filters.

For all audio effects, we can copy the video steam unchanged with the -c:v copy option.

You can find more details about stream copy here.

The -c:v copy option isn’t applied for reverse, slow down, and speed up effects because they alter both audio and video streams.

By default, FFmpeg selects the best input stream of each kind (video, audio, subtitles, etc.) Let’s assume that for our audio effects we want to convert all audio input streams, however (maybe we have the same stream in different languages, for example). You can do this with the -map 0 option.

All Frei0r plugin video filters are applied with the FFmpeg simple filter option -vf frei0r=filtername:param1_value|param2_value|param3_value

The official Frei0r page doesn’t contain any documentation on how this option should be used. That’s why I had to conduct my own research. During my research, I found a helpful resource that describes Frei0r filters along with some other plugins. With the help of this resource, I learned about all available filters and their parameters.

Here’s the original video that I used for my video effects processing demonstration:

And here’s the original video I used for my audio effects demonstration:

#1 Sepia effect

The complex colorchannelmixer filter adjusts video input frames by remixing color channels. This filter changes the color temperature to sepia-like.

-filter_complex colorchannelmixer=.393:.769:.189:0:.349:.686:.168:0:.272:.534:.131

#2 Black and white effect

Zero saturation with the hue filter produces a black and white output video.

-vf hue=s=0

#3 Vertigo effect

The vertigo effect performs alpha blending with zoomed and rotated images. For my example video, I set the phase increment parameter to 0.2 and kept the zoomrate parameter with its default value.

-vf frei0r=vertigo:0.2

#4 Vignette effect

The vignette effect makes your video look like it was shot with a vignetting lens. I used the default settings.

-vf frei0r=vignette

#5 Sobel effect

The Sobel effect is used in image processing and computer vision, particularly within edge detection algorithms where it creates an image emphasizing edges. The Sobel effect doesn’t have any parameters you can change.

-vf frei0r=sobel

#6 Pixelizor effect

The pixelizor effect is also applied with default settings and creates a pixelated video.

-vf frei0r=pixeliz0r

#7 Invert0r effect

The invert0r effect inverts all colors of a source image.

-vf frei0r=invert0r

#8 RGBnoise effect

The RGBnoise effect adds RGB noise to your video. It has one parameter that defines the amount of noise added. The result looks like a 90s movie played on an old VHS player.

-vf frei0r=rgbnoise:0.2

#9 Distorter effect

The distorter effect distorts image in a weird way. I set two parameters here: amplitude and frequency of the plasma signal.

-vf frei0r=distort0r:0.05|0.0000001

#10 IIRblur effect

IIRblur provides an Infinite Impulse Response Gaussian blur. I used this filter with its default settings.

-vf frei0r=iirblur

#11 Nervous effect

The nervous effect flushes frames in time in a nervous way; it doesn’t have any additional parameters.

-vf frei0r=nervous

#12 Glow effect

The glow effect creates a glamorous glow. I set the Blur of the glow parameter to its maximum value.

-vf frei0r=glow:1

#13 Reverse effect

The reverse effect is achieved with two FFmpeg filters – video reverse and audio reverse – running at the same time.

-vf reverse -af areverse

#14 Slow down effect

The slow down effect uses the setpts filter to double the Presentation Time Stamp (PTS) of the original video. As a result, you get a video that’s half the speed of the original.

The atempo filter is used to halve the speed of audio with the 0.5 (50% of the original tempo) setting.

-filter:v setpts=2.0*PTS -filter:a atempo=0.5

#15 Speed up effect

The speed up filter works with the same approach as slow down effect, only in the opposite direction. It takes half of the original video Presentation Time Stamp and makes it twice as fast.

-filter:v setpts=0.5*PTS -filter:a atempo=2.0

#16 Echo audio effect

The aecho audio filter reflects your audio stream to create an echo effect just as if you were in the mountains.

-af aecho=0.8:0.9:1000|500:0.7|0.5

The first parameter is the input gain of the reflected signal (0.8). The second parameter is the output gain of the reflected signal (0.9). Next there’s a list of delays from the original signal in milliseconds, separated by the pipe | symbol. After this parameter goes the list of decays for each delay in the previous list. The colon : separates items in lists.

#17 Tremolo audio effect

The tremolo filter performs sinusoidal amplitude modulation.

-af tremolo=f=10.0:d=0.7

The f parameter is for modulation frequency in Hertz. The d parameter shows the depth of modulation as a percentage.

#18 Vibrato Audio effect

The vibrato filter performs sinusoidal phase modulation.

-af vibrato=f=7.0:d=0.5

The f and d parameters perform the same functions as in the tremolo filter.

#19 Chorus audio effect

The chorus filter resembles an echo effect with a short delay. The difference is that the delay in the echo filter is constant, whereas in the chorus filter it varies using sinusoidal or triangular modulation.

-af chorus=0.5:0.9:50|60|40:0.4|0.32|0.3:0.25|0.4|0.3:2|2.3|1.3

The parameters in the chorus filter are ordered as follows: input gain, output gain, delays list in ms, decays list, speeds list, depths list. Parameters are separated by the pipe and items in lists are separated by the colon.

#20 Watermark

The watermark feature is implemented in the carrierwave-video gem. It uses FFmpeg’s overlay filter.

On the video below, you can see the combination of the vertigo, vignette, RGBnoise, distorter, glow, reverse, echo, tremolo, chorus, and watermark effects.

The watermark on this video is a transparent png image with the red text FFmpeg TV:

In this article, we’ve covered implementation of FFmpeg video and audio filters and Frei0r plugin effects. FFmpeg and Frei0r are powerful tools. The filters described here mostly add some additional distortions to video. But FFmpeg can also be used to perform video and audio enhancements such as lens correction, sound level normalization, noise and shake removal, video streaming, and much more.

3.9/ 5.0
Article rating
Remember those Facebook reactions? Well, we aren't Facebook but we love reactions too. They can give us valuable insights on how to improve what we're doing. Whould you tell us how you feel about this article?