Yalantis: iOS, Android And Web App Development Company

Drawing a Cubic Bezier with OpenGL ES for Sound Visualization on Android

As we develop a wide range of apps here at Yalantis, we often face issues with processing audio files on Android. These issues are mostly due to the lack of good native tools for working with audio.

Recently we were researching audio processing and decided to create a useful solution for processing large audio files. As part of this project, we asked our design team to create a challenging design concept for us to work with.

The result was this awesome equalizer concept by our designer Sergii Ganushchak that we called Horizon (see the library on Github):

The best part of this design is that it challenges not only our expertise in audio, but also our expertise in complicated custom animations, which can also be difficult to implement on Android. We were very excited to take on this project. Here’s how it went.

The visual part

First, let’s introduce the visual part of our audio component. The equaliser consists of five waves. Here’s the second wave, which corresponds to bass frequencies, as an example:

If you open the original .svg file in a vector image editor, you’ll see that this wave is made of four cubic Bezier curves! Here’s one such curve selected:

Cool, but what is a Bezier curve?

What Bezier curves are and why they rock

A Bezier curve describes smooth curves mathematically. Bezier curves were popularized, but not actually discovered, by Pierre Bezier. He used them to design automobile bodies. Remember those curvy cars that were popular in '60s?

Nowadays, Bezier curves are widely used in computer graphics, animations, fonts and much more. Every modern vector graphics editor, such as Sketch and Inkscape, supports them.

The best way to understand Bezier curves is to look at some graphs

There are several types of Bezier curves:

Linear Bezier is just a linear interpolation between a start point P0P0 and an end point P1P1. It has no control point.

Quadratic Bezier has two control points. The form of a quadratic Bezier curve is specified by a single control point P1P1.

At every t∈[0;1]t∈[0;1] the position of Q0Q0 needs to be calculated by interpolating between P0P0 and P1P1. Then, in the same way, the position of Q1Q1 is interpolated between P1P1 and P2P2. Finally, the position of the point BB that lies on curve is interpolated between Q0Q0 and Q1Q1.

Cubic Bezier is the most popular kind, and is the one we’ll be using. A cubic Bezier curve needs two control points, P1P1 and P2P2, which allows for greater flexibility:

It’s drawn exactly like the other, only it involves more steps and points:

Higher order Bezier curves exist, but we are not going to look at them for the purposes of this article.

How to draw a Bezier curve on Android

Let’s start with Android Canvas and then move to an OpenGL ES solution.

How to draw a Bezier curve with Canvas

It’s possible to draw Bezier curves with Android Canvas.

First, initialize paint and path:

Then, add the Bezier curve to the path by calling the quadTo or cubitTo method:

  path.reset();
  path.moveTo(p0x, p0y);
  path.quadTo(p1x, p1y, p2x, p2y);
  path.moveTo(p0x, p0y);
  path.close();

And finally, draw the path on the canvas:


   canvas.drawPath(path, paint);

As you can see, drawing Bezier curves with Android Canvas is very easy, but performance generally very poor:

Performance while drawing multiple Bezier curves per second with Canvas. Green line signifies 16ms.

Now let’s see how the OpenGL API can help us create a Bezier curve.

How to draw a cubic Bezier with OpenGL ES

OpenGL ES is very fast at drawing triangles, which means we need to come up with a way to split the shape we want into triangles. Since our wave is convex, we can approximate it by drawing many triangles, all of which have one vertex located at the center of the screen (0, 0).

Here’s the idea:

  1. Split every Bezier curve into an even number of points nn

  2. Generate n−1n−1 triangles with vertices at (N1,N2,O), (N2,N3,O), …, (Nn−1,Nn,O).

  3. Fill these triangles with color.

Splitting the Bezier curve

For each point on the Bezier curve, we are going to generate three attributes for three vertices. This is done with a simple method:

private float[] genTData() {
   //  1---2
   //  | /
   //  3
   float[] tData = new float[Const.POINTS_PER_TRIANGLE * Const.T_DATA_SIZE * mBezierRenderer.numberOfPoints];

   for (int i = 0; i < tData.length; i += Const.POINTS_PER_TRIANGLE) {
       float t = (float) i / (float)tData.length;
       float t1 = (float) (i + 3) / (float)tData.length;

       tData[i] = t;
       tData[i+1] = t1;
       tData[i+2] = -1;
   }

   return tData;
}

Attributes of the first two vertices specify points on the curve. The attribute for the third vertex is always -1, which by our convention means that this vertex is located at (0,0)(0,0).

Next, we need to pass this data to a shader.

Shader pipeline

There are three types of variables in the OpenGL Shading Language:

  • uniform – Data are the same for all vertices. Here we would store Bezier start, end, and control points as well as color and sound level for every wave.

  • attribute – Data differ for each vertex. That’s what we’re  going to use to specify point position on the curve.

  • varying – We’ll use this to pass information from the vertex shader to the fragment shader indicating whether the current fragment (pixel) is near the wave’s edge.

We’ll to use the following variables:

Uniforms (common for the entire wave):

  • vec4 u_Color – Color of the wave

  • float u_Amp – Sound level of the wave

  • vec4 u_BzData – Start and end points of the Bezier curve

  • vec4 u_BzDataCtrl – Two control points of the Bezier curve

Attribute (per individual vertex):

  • float a_Tdata – interpolation coefficient tt (specifies point on the curve)

Now, given start, end, and control points of a curve, as well as tt, we need to find the location of the point on the curve.

Let’s look at the formula for a cubic Bezier:

9N6U521.png

Yikes! Not very intuitive, is it? Still, it’s easy to translate this directly into GLSL:


vec2 b3_translation( in vec2 p0, in vec2 p1, in vec2 p2, in vec2 p3, in float t )
{
   float tt = (1.0 - t) * (1.0 - t);

   return tt * (1.0 - t) * p0 +
       3.0 * t * tt * p1 +
                3.0 * t * t * (1.0 - t) * p2 +
                t * t * t * p3;
}

But we can do better. Let’s look at the geometric explanation of a cubic Bezier curve once more:

With the help of GLSL’s mix function, we interpolate between points and almost program declaratively:

vec2 b3_mix( in vec2 p0, in vec2 p1,
         in vec2 p2, in vec2 p3,
 in float t )
{
   vec2 q0 = mix(p0, p1, t);
   vec2 q1 = mix(p1, p2, t);
   vec2 q2 = mix(p2, p3, t);

   vec2 r0 = mix(q0, q1, t);
   vec2 r1 = mix(q1, q2, t);

   return mix(r0, r1, t);
}

This alternative is much easier to read and, we think, is equivalent in terms of speed.

Color blending

Take another look at how nicely the colors blend together in our original design. For, instance on the left the purple and reddish waves blend into a soft pink, and along the x axis all five waves merge into a whitish color.

This is called additive coloring. It’s similar to the way light mixes in computer monitors:

Additive coloring

By a little trial and error, we found out that this particular color blend mode is called the screen blend mode.

To tell OpenGL that we want screen-like blending, we need to enable GL_BLEND and specify the blend function in our onDrawFrame method before actually drawing the waves:

GLES20.glEnable(GLES20.GL_BLEND);
GLES20.glBlendFuncSeparate(
       GLES20.GL_ONE, GLES20.GL_ONE_MINUS_SRC_COLOR,
       GLES20.GL_ONE, GLES20.GL_ONE_MINUS_SRC_ALPHA
); // Screen blend mode

The end result is:

You can find our source code here (equalizer-related code is in com.yalantis.waves package).

Read also:

 

 

Insights

Music Application Development Tendencies

Tech

Introducing uCrop, Our Own Image Cropping Library for Android

Tech

Introducing Horizon, Our Own Open Source Library for Sound Visualization

Tech

How We Created uCrop, Our Own Image Cropping Library for Android

Design

7 Types of Animations for Your Mobile App

Excited to create something outstanding?

We share the same interests.

Let's team up!