Part 1 - Why?
Back in 2019 I was the CTO and Jonathan, current co-founder at Cerebrium, my trusted (but clearly superior) lead engineer at an on-demand grocery delivery company in South Africa, called OneCart. OneCart scaled rapidly during the COVID-19 pandemic due to a strict nationwide lockdown and a sudden spike in demand for online grocery delivery. OneCart had no choice but to transform itself to keep up with the rapid rate of change. After working at OneCart our definition of “scale” changed.
Now when we refer to scale we mean:
- Waking up to 8x the number of orders from the previous day.
- Hiring 250 people in four weeks.
The OneCart team and operations grew so quickly that departments were created overnight and countless initiatives were developed to cope with the sudden increase in demand. However, like many rapidly growing businesses, we were losing money in a lot of areas and needed to optimize. One of the things we identified was to centralize our data across departments so we could have a better sense of what was going on in the business to make decisions. So we dived in! After an initial investigation, we decided we would need to implement the following three things to accomplish this goal:
- Set up a data warehouse
- Sync data across all data sources used in the company which included over 20 SaaS tools and multiple databases
- Integrate a BI tool for end-users to create dashboards and answer data questions.
Nothing crazy right?
We assumed it would be simple and straightforward. We were wrong. It was all painstakingly difficult. It was time-consuming, difficult to develop, and costly. These are some of the problems we encountered:
- ELT tools like Stitch and Fivetran were available but they were very expensive! Furthermore, open source tools like Airbyte were unavailable at the time.
- Data engineering became a repetitive exercise as each data source had its own nuances on how they tracked and represented data.
- The feedback from our team on potential BI tools was that they were clunky, complex, and overwhelming.
Resolving these issues was going to take a significant investment of time and resources and we knew we would have to run many iterations after seeing the team interact with it. We also didn't have the budget to hire in our team since all our money was going to operations. In the end, we hacked something together because OneCart had more pressing issues which could be fixed quickly and would save money immediately. Like anything created in this manner, the system had a limited lifespan. Issues came up frequently and we were continuously “band-aiding” these problems.
So what now?
Like most developers, we couldn't let it go until it was fixed. I started investigating tools, researching data infrastructures other companies were implementing (mostly enterprises), and came across a host of problems, mainly:
- Most data products were targeted towards well-established/enterprise teams and not accessible to small to medium-size businesses (SMB) due to pricing or the time needed to set up the project and maintain it.
- SMBs use a host of Saas and low-code/no-code tools in their business but all these tools have small nuances as to how they define/calculate a metric that can lead to inaccuracies in your data if you don’t read the documentation thoroughly.
- Not everyone in a SMB knows SQL or understands the modeled data structures well enough to know which database tables they should be using so inundate the data/engineering team with requests for data.
If we could solve all these problems within minutes for SMBs, how could the trajectory of their business change? What if we could save them time, money and resources by developing a tool that allowed them to use data for business value?
