EMPOWERING A DATA-DRIVEN CULTURE FOR ONE OF THE LARGEST US FOOD MANUFACTURERS

To expedite project progress, the Yalantis team has become fully integrated into the client’s software development lifecycle (SDLC), closely collaborating with their data experts and functioning as part of their team.

Our goal was to help the client modernize their data processing technology by creating versatile functionality for storing and analyzing data. This involved handling growing data volumes through data lifecycle management (DLM) — the process of managing, organizing, and securing data from creation or collection to archival and deletion.

Together with the client, we began by focusing on sales and finance data to improve sales and resource management, with the intention of extending the efforts to other departments. This involved three key stages:

Migrating data from manual files to a database to create a consistent data storage standard

A key focus during this stage was on providing the sales and finance departments with access to accurate data on business activities, trends, and changes in a comprehensive form for quick retrieval. This involved:

• analyzing source data by defining all available data sources and determining systems of record to avoid loading unnecessary datasets
• standardizing the data structure and populating the database with data from raw sources in the target form
• systematically and efficiently transferring logic from files to the database

Setting up a data warehouse for a unified data management solution

This phase involved consolidating data from diverse sources, ensuring data quality and integrity, and optimizing storage for analytics and reporting. It encompassed the following steps:

• Modeling a data schema. Together with the client’s team, we organized tables, schemas, logic, and data processing on the Amazon SQL server. This included clustering information and staging different types of processing for improved control and scalability.
• Developing an extract, transform, load (ETL) process from the ground up using Azure Data Factory to build data pipelines. The ETL architecture with various data clusters and data schemes ensured the system could support increased data volumes, allowing seamless data processing regardless of the raw data volume.
• Introducing Python for automatic data loading, validation, and testing. Python scripts were scheduled to run automatically, ensuring consistent data processing, reducing manual errors, and efficiently handling large datasets and complex tasks.
• Defining security and access controls to protect sensitive data and control user access, such as by encrypting sensitive data and establishing role-based access control (RBAC).

Optimizing reporting for a unified view of accurate data

In this phase, data engineers and analysts enhanced reporting, which involved migrating all of the client’s reports and creating new ones based on datasets. Activities included:

• shifting calculations to the database. As we moved calculations from the Power BI side to the database side with SQL, Power BI now functions solely as a display tool without performing calculations upon receiving data. This accelerates report processing, enhancing the overall user experience. Analysts and users benefit from swift responses to filters and data updates, all automatically executed through SQL code in the background.
• scheduling data refreshes. Data engineers defined and automated regular data refreshes to maintain the warehouse’s current state.
• archiving data. The team set up tables to archive data, providing coverage for test cases and facilitating comparisons with uploaded raw data. This ensured database integrity and assisted in identifying potential data loss.

In the initial six months, we worked with our client to successfully provide access to valid data in a comprehensive form for the sales and finance departments. Moving forward into 2024, our client has strategic plans to:

• extend data-driven decision-making capabilities to ops and stock management
• establish a comprehensive enterprise data warehouse
• create a robust data core, which will ensure swift and secure access to accurate company data based on an employee’s position and level, facilitating agile product control and proactive analysis of any internal or external impact on the product