Analytics solutions: build in-house or buy off-the-shelf?

By Karen Fang, BI Insights Team at Babylon

The number of tech start-ups are growing exponentially each year. Nowadays, as long as you can think of a problem, there's probably a company working on building a solution for it. In the meantime, there are also lots of open-source solutions for tackling particular problems - if you can define your problems really well, have the right talents and are determined to invest time building it.

With both options having advantages and disadvantages over one another, deciding on what to do sometimes isn't easy and almost always involves risk assessment, and it should.

When it comes to deciding on buy Vs build, oftentimes the answer can be obvious, but it isn’t always the case, especially for a piece of new technology. Although it may sound cliche but it’s true - there isn't a default answer to all the problems you may have, it's all about whether the decision is right for your company at that time. While I can't make the decisions for you,

here are some points that I think may be helpful in deciding what would be more suitable for your needs.

1. Define your purpose, assess capability and toolkits that fit the purpose

This sounds very obvious, but you may be surprised how many projects started without a well-defined and actionable goal. It can be all too tempting when we are under pressure to grow fast and hit targets. “How do I solve this quickly?” As you trawl through google, you see buzzwords floating around in the search results. It’s also too tempting when you find a site that seems to speak to every one of your pain points and promises that you can improve X-fold of [insert KPI here] if you work with them. But wait a second, yes you have the grand goal of growing your KPI by X-fold, but what is the No.1 problem that you are trying to solve right now? How long have you got to get there and what are the milestones that you need to achieve? Are there any preliminary steps that you need to take for a solid data foundation and infrastructure first, e.g. do you need to organise your site tags, streamline data, consolidate your campaign naming convention etc.?

Before considering any solution, really try to define the what, why and when on a project. After you’ve done that, it’s time to assess internal capabilities and toolkits against external ones.

As I give examples later on of the solutions we built / are building in house, I want to be clear that we keep our ears open to solutions that are offered off-the-shelf too. An example being a Programmatic Marketing Data Connectivity Platform that we’ve encouraged our business partners in Marketing to engage with in order to help our Digital Marketing team solve a specific use case in the US. We’re excited about the solution as it uniquely serves the exact purpose that we defined - to open up more opportunities for Digital Marketing in the region. There’s simply no way we can, nor would it be worth us trying to build that in-house.

2. Are you ready for something sophisticated or do you need an MVP first

When I think about a complex problem, I try to think about an MVP first. It helps to break the problem down and enables us to move at speed.

For example, here at Babylon, our team wanted to empower Marketing to champion a “Test and Learn” mindset by enabling and encouraging everyone to conduct A/B tests at speed and learn fast, try out their ideas and have a systemised way to measure success for scalability.

While there are free test calculators online, we didn’t find them intuitive to use and we wanted a tool to be tailored to our Test & Learn framework so our business partners in Marketing can easily adopt it to self-serve their A/B tests setup. In short, we didn’t find one that quite served our purpose. Therefore, we built one ourselves. This in-house calculator is a great little tool in making A/B testing scalable and easy to follow by non-technical users too. It empowers people to make more data-driven decisions without always having to rely on analysts. More importantly, it suits our business partners’ analytical needs in its current state.

The calculator was built from scratch in three months in Python and it’s worth mentioning that people who worked on this project didn't have prior experience in building a web-app nor did they have specific in-depth knowledge of Bayesian statistics at the time. As a team, we researched the techniques required, designed the interface and built it - a tool that does exactly what it was supposed to do. However, we would never attempt building something that completely replaces an end-to-end Product A/B testing tool such as Optimizely which would be for slightly different use cases.

3. Time for integration

I would consider this before the financial budget because time is money:) When it comes to finding a solution, time should be top of the list. If a solution is going to take two years to integrate before you can see any meaningful result, whether it's self-build or off-the-shelf, it's probably not the right one, it’s time to go back to the step above - find an MVP.

When assessing a solution, think about what timeline would be realistic for you to start getting meaningful return on the solution you invested in. Here, I'm not just talking about the time for implementation. If you are buying off-the-shelf, you should also include time for the initial sales conversations (potentially with multiple vendors as you also need to demonstrate that you've done a thorough research of the market), technical scoping conversations, legal and contracting. While each stage can take more than one if not a few months, when it comes to implementation, although it may seem like you've jumped through a few big hurdles already, it can sometimes be painful and time-consuming depending on how well the previous stages such as scoping and planning were carried out. This is because during implementation, you will also need to invest your own analysts’ time to work with the vendors and ensure the solution is configured properly for the right datasets, APIs, custom settings and to produce the right outputs.

I previously indirectly and directly worked on two separate vendor integration projects of third-party marketing attribution solutions at different companies. Both projects were commissioned before I was involved and both required specifically cleaned and structured datasets to be shared in specific ways. On both occasions, the projects took nearly two years before we even managed to get the cleaned datasets to be shared and confirmed with the vendors - and yes, we never even reached the status of the “perfect” datasets before they were decommissioned. The solutions were cumbersome to integrate with and they also became burdens to our internal analytics capacity.

Of course it won’t be the case with all vendors, situations like this sometimes can be avoided through meticulous planning, scoping and contracting beforehand. However, bear in mind that doing a good job in those stages can also be a lot of effort too, so make sure you allow enough time for them.

If we are talking about building in-house, the questions become: do people understand the challenges well enough to be able to work out a suitable solution? Do you have existing analytics capability and capacity to work on building the solution in a reasonable timeframe, and if not, do you need to hire talents and how long would that take?

Sometimes analysts can build a solution quite quickly to suit the immediate needs, but for complex and sophisticated technology, it may be unrealistic to expect two or three analysts to build it in 6 months or a year, in which case you may be better off going with a vendor who specialises in that particular area and that their entire company is dedicated to building it. This leads to the next point...

4. How much control are you willing to give up

One thing you’ve got to accept is, once you’ve decided to buy, at least parts of that solution you are buying will be a black box to you - either the vendor has access to information you don’t have access to, or they have a piece of proprietary technology that they can’t let you see fully, or both - because that’s ultimately how vendors can charge you a premium for. This means giving up some level of control and choosing to trust a vendor that their solution will work and will be configured to work for you in an optimal way.

Of course, you can validate if the vendors’ solutions are working by e.g. running causal experiments, but validation frameworks and criterion sometimes can be hard to determine too. Perhaps more importantly, you don’t really know if the solution is configured in the best possible way for you. So it’s important that you assess how much control you are willing to give up.

When it comes to Analytics solutions, our team is a curious bunch - we like to understand how things work, especially when it comes to new technology and being able to adapt configurations. For example, we wanted to explore an Econometrics model called Marketing Mix Modelling in order to help the business understand the incremental value generated by each Marketing channel. We knew we needed to embrace this methodology given Apple’s privacy updates and the direction the industry is moving towards. The question is do we buy or build. While there are vendors and agencies that provide the solution, it is a method many companies have just started to explore and understand. Meanwhile, we feel that our business model and datasets are somewhat unique, so it’s worth giving it a stab ourselves first to understand if the model would be suitable for the business.

In the end, we decided to do it in house by utlising an open-source model developed by Facebook Marketing Science team. In that way, we can understand and assess how the model works exactly while having full control on adapting it to our needs and adjusting the inputs, outputs and parameters to what suits us. Having said that, depending on how we scale later on, there is a chance that we may want to work with a vendor in the future, and when we do, we should know better on what we will be looking for and the success criteria, having gained knowledge by building the in-house version.

When you assess a vendor, ask for their technical paper, query them ((or find someone who is able to) about the methodology, understand what information they cannot disclose, what features are configurable and how they will work with you to do that. If possible, ask them if and how they or their clients validated the outputs and impact.

5. Industry standard and vendor’s reputation

Some off-the-shelf solutions are well tested, have become the industry standard to implement and are must-haves. For example, when you have a website, you implement Google Analytics or Mixpanel etc. as standard and can use their dashboards and reports.

Vendor's reputation can play a big part too. It goes without saying that a well-regarded vendor with a track record of working with known brands to deliver success are far more likely to offer lots of experiences and insights to the problems you are trying to solve and may help you get to the results quicker.

6. Other factors to consider:

How customisable the solution is

Self-built solutions are tailored for the company and are adaptable. Most mature off-the-shelf solutions offer features that can be customised, so consider what are your must-haves and what are just nice-to-have.

Budget

I left this to the last not because it’s not important but because it’s the most obvious - you can’t buy or invest in what you can’t afford. Consider how much you are willing to invest over the expected return within a defined time window - i.e. the solution is going to cost $$$ (either buy or build), it’ll likely generate a return and benefit the business by $$$ within X months or years.

7. Finally, regardless of which option you go with:

Think about how you would measure success / effectiveness of the solution

This could be actions driven by the output, feedback or even better, revenue / KPI growth.

Assess the risk and think about a back up plan

This is why having a timeline in mind would be helpful. If it doesn't work out in the pre-defined timeline, have you thought about what other options you may have?

When it comes to choosing buy Vs build, it really is down to what’s most suitable for your company at the time and whether using that solution, you can answer business critical questions in a reasonable timeframe. As with most projects, it’s all about identifying the gaps and finding out what works best to resolve them.

Last but not least, a car is only as good as its driver. Remember that no matter what solution you go with, effort will be required to maintain, evaluate and improve it. There is no silver bullet or magic that solves all problems.

— Written by Karen Fang, Marketing Analytics Lead @ BI Insights, Babylon

Got any questions, feedback, interested in joining us or just want to chat about Marketing Analytics in general? Please get in touch: [email protected] or LinkedIn

Try Babylon today

Babylon offers high-quality, 24/7 comprehensive health care. Let us help:

Download the Babylon app

Learn more about our services