A social platform for sharing event videos with high-quality audio
Our client came to us with a pilot version of a social platform for sharing videos from parties and concerts. He knew that people loved sharing the emotions of these events, but the bad sound in videos wouldn’t let them do that to the fullest extent. That’s why he wanted to enhance his platform with automatic audio improvement and video editing.
In addition, fewer and fewer people were using the app due to the lack of user-friendliness:
- It was difficult for users to navigate videos and get to older posts.
- Chat messages appeared in the middle of the screen, making watching videos uncomfortable.
- The app couldn’t always recognize audio correctly.
For providing video editing capabilities, we developed a custom camera with advanced functionality. And to implement automatic audio enhancement, we created a system that sends users’ videos to the server and returns them with high-quality sound.
Additionally, to make the TagMix app more user-friendly and help our client get more active users, we offered a number of improvements:
- Organized videos as separate posts and implemented a filter system for quick search
- Substituted the chat with comments so as not to interrupt videos
- Simplified the video uploading flow for proper audio recognition and enhancement
We explain how we made all of these improvements below.
- $50,000 to $199,999
- Ferbruary 2017 – Present
- 10 People
- London, UK
- Software development
- Quality Assurance
- UI and UX Design
- Project Management
- Core Data
- AWS Cognito
- Xcode Bots
- Firebase Dynamic Link
- RxJava2 / RxKotlin
- GitLab CI
- + Fabric GitLab CI
- AWS Lambda
- ServerLess Framework
- Amazon DynamoDB
- Elastic Search
- AWS Cognito
- AWS Elastic Beanstalk
- AWS SQS
- AWS S3
- AWS Elemental MediaConvert
- Implement video editing capabilities and automated audio enhancement
- Increase the number of active users
Implement video editing capabilities and automated audio enhancement
Develop a custom camera for recording and editing videos
For iOS, we created the camera itself using the Camera Engine library. Our camera records video, changes the orientation (portrait or landscape) when the phone is turned, adjusts the brightness, and changes from the main camera to the front camera (selfie mode). All of this functionality was created via standard iOS tools without any third-party services or libraries.
For Android, we made the app compatible with a variety of Android devices. The Android native library couldn’t provide us with sophisticated features, so we used more advanced tools: GLSurfaceView, CameraSurfaceRenderer, TextureVideoEncoder. These tools helped us implement focus on touch, hande life cycle changes, start and pause recordings, and record video in MP4 format with a specified size, bitrate, and framerate.
Implement video processing for clean, high-quality audio
At first, we wanted to avoid converting and mixing audio and video files and instead simply play them in parallel. But in that case, users wouldn’t be able to share their videos with others. We also wanted to move all conversions to the backend. We could have used Amazon Transcoder along with feed video and original audio, then receive a video file with a clean audio track. However, we needed to find a cheaper solution.
Finally, we decided to develop the flow this way: The TagMix app sends the original audio in a multipart request using Retrofit and the server starts the analysis and enhancement process. Next, the app downloads the clean sound file. Users aren’t bothered with everything going on in the background, but they’re notified about the progress.
Improve the quality of
video when the internet connection is bad
The app had an issue with playing videos when the internet connection was bad. There was also a problem with videos not starting quickly when we added a new autostart feature in the feed (similar to Instagram). We concluded that we needed to implement HLS technology. You can find more information about HLS on the Apple developer page for HTTP Live Streaming.
On the server side, we encode uploaded video files with different settings, configuring the video resolutions and bitrates for different internet speeds. In iOS, we added settings to upload video files with specified timing and quality for auto-playing in the feed.
Allow downloading video files with clean audio
To avoid large delays (as audio conversion takes some time), the app allows users to simultaneously play downloaded clean audio with the original video. This way users can decide if they like the video and post it. After clicking Post, all the magic begins.
We convert the downloaded clean audio from MP3 to AAC with FFMPEG using specified parameters, as it’s impossible to add an MP3 track to an MP4 container. Next, we create an MP4 container by mixing H.264 video with the converted AAC track with the help of Mp4Parser. After mixing, the video file is uploaded to the server and appears in users’ feeds.
Increase the number
of active users
TagMix had a high uninstall rate when we started our collaboration. Our task was to retain users by making it more user-friendly.
Navigation in the feed.
How it was.
Previously, it was focused on one video at a time and looked just like the camera screen. Users had to swipe through each video and it took a lot of time to get to old posts. There were also floating buttons allowing users to like videos and start chats.
Navigation in the feed.
What we have done.
We chose another approach to organizing content, showing videos as separate posts so users can easily scan the content in the feed. We also added the ability to share videos on other social platforms.
We simplified the flow of uploading video to improve the UX. Now users need to record at least 10 seconds of video so the app can analyze and enhance the audio correctly. After the audio is enhanced, the video is synchronized with the enhanced high-quality audio. Users can see a preview of the recorded video and then add more videos for editing by uploading them from the gallery. Moreover, we implemented a tagging feature to tag friends in videos.
Next, we had to work on the chat. Before, users could start chatting on the video they were watching and messages appeared in the middle of the screen.
To let users communicate without interfering with watching, we replaced this chat feature with the ability to write comments under posts.
Streamlining such a powerful media tool for video and professional sound was quite a challenging task, as there were no similar apps on the market. However, we had an excellent opportunity to add cool features, greatly improving the UI and UX of the app.
Our team managed to get rid of the camera issues on Android, made navigating the app more straightforward, added filters for content, and provided social sharing on Twitter, Facebook, and Instagram. Moreover, users can now communicate in a comments section instead of disturbing video playback and writing messages right on videos.
The TagMix app is available on the App Store and Google Play Store. It’s growing popular and remains a unique product with no alternatives.