From in-depth research to quick results: Delivering high-performance data connectors for Meroxa

Supporting Meroxa in their noble mission to enable a faster and more efficient data flow between developers’ production data stores.

  • Developing connectors to increase project community

  • Approaching each connector individually

  • Setting up a seamless flow of delivering connectors

  • Improving the overall performance of the project platform

Value delivered

  • Successfully developed data connectors within the client’s requirements

  • Ensured timely and predictable delivery of connectors: one connector per developer, resulting in three ready connectors within one and a half months

  • Came up with a quick bug-fixing flow that takes less than 10 percent of the team’s overall capacity

  • Provided comprehensive and detailed technical documentation for each connector

  • Offered ideas for improving the overall workflow of the Conduit platform

Client’s mission

Meroxa is transforming traditional data engineering. The company’s Streaming Application Platform as a Service (SAPaaS) abstracts away the challenging aspects of Kafka such as Topics, Connect, and Partitioning by providing them as a managed service. Meroxa creates a developer-first experience that encourages more agility and participation in data streaming, where engineers can ingest, orchestrate, transform, and stream data to and from anywhere using JavaScript, Python, and Go.

In 2021, the US-based Meroxa closed a Series A funding round. Recently, Meroxa received funding from the United States Department of Defense to work on government projects.

  • CONTRIBUTING TO THE DEVELOPER COMMUNITY

    In January 2022, Meroxa launched an open-source project called Conduit that ensures real-time data movement between databases and services by providing built-in connectors. While most data engineering tools on the market require lots of expertise and manual work on the part of software developers, Conduit allows users to build real-time automatic data pipelines.

    Meroxa was looking for an extension to their in-house team to satisfy the growing need among the developer community for various connectors. They considered several software development companies and eventually opted for Yalantis, as our engineers were able to complete the test tasks in line with the expectations and within the deadlines.

    Meroxa asked Yalantis to:

    • develop connectors in the Go programming language for the requested databases and services
    • contribute to the overall Conduit platform by detecting issues and offering ideas on how to improve them
    • fix issues with connectors developed by other teams

    With our help, Meroxa intended to grow the Conduit community by increasing the speed at which they brought connectors to market.

Solution provided by Yalantis

  • Yalantis team of Go developers enriched the capabilities of the Conduit platform by timely introducing new high-demand connectors, allowing developers to ensure a streamlined data integration between more databases and services.

What our client says

  •  

Entrust Yalantis with your next open-source project

We accept development challenges and stand by our commitments during the complete SDLC.

Contact us

Approach to building connectors

Since the project is open-source, the client’s main requirement is high-quality and well-documented code. Our team is fully responsible for the development of connectors. Yalantis developers apply R&D skills to develop connectors in the best possible way and meet critical project constraints.

Our approach to each connector involves a few steps:

Investigation. This stage begins with studying the documentation of databases or services (e.g. Stripe, Hubspot) for which we have to develop a connector. We also document our research findings, which include our ideas, issues, and needs, to share them with the Meroxa team.

Estimation. At this stage, each developer provides a thorough estimate of their work before beginning development.

Development. The planned duration of the development stage for each connector depends on the estimates. During development, we also make demos to present our work to the client when we hit important milestones.

Testing. To make sure that connectors work as planned, our developers perform unit, integration, and acceptance testing. After that, a QA engineer conducts manual testing of the connectors.

Client approval and editing. Upon completion of the development and
testing stages, our client reviews delivered connectors to provide feedback.
Our developers then make any necessary changes.

Final testing. We finish with the final testing stage to release well-performing connectors to the Conduit community.

Need a custom engineering approach to your project?

Contact us

  • TELLING EACH CONNECTOR’S UNIQUE STORY

     

    Our Go developers approach the development of each Conduit connector individually. The only requirement from Meroxa’s team is that for some databases and services, we have to develop either a source or destination connector. For other databases and services, we’re required to create both a destination and source connector.

     

What’s next

  • Ongoing support of released connectors

  • Support with the development of more than 70 connectors for various projects

  • Development of a testing library for verifying data types transferred by connectors

More projects