The Observability Conundrum!

There are mainly two types of observability platforms out there. 

• Those that you rent. These are proprietary like DataDog, NewRelic, Dynatrace, SumoLogic, Splunk, etc.) or,
• Those that you own. Typically, these are open-sourced solutions like Prometheus, OpenSearch, Mimir, Thanos, Victoria Metrics, etc., 

I am purposefully not mentioning homegrown solutions that the hyper-scalers of the tech world built in-house for their specific needs. I aim to cover that in a subsequent post.

The user of today’s observability platform, in my opinion, is challenged with a situation that is analogous to renting vs. owning a residence. And similar to owning vs. renting of residence mechanics, it may make sense to rent when you have a small footprint. But keeping the future growth (of your business applications and hence the infrastructure you will need) in mind, owning is better. Let us look at this in context of renting vs. owning a residence.

In the residence context, landlords take the maintenance responsibilities (leaks, updates, etc.)., and you pay the rent. They can increase the rent if and when they want, and you choose to stay or leave. When your requirements grow (family etc.) renting a larger residence can be cost prohibitive. Deciding to own instead comes with the headache of dealing with maintenance, for which you independently hire services. What if your requirements grow further? Will your residence be able to support that? Or will you look to move again or add more space? It is also worth mentioning that there are associated challenges each time you decide to update or move. For example, your old furniture may not suit or fit your new residence, or your cable/internet provider doesn't service the new area, etc. Overall, it becomes challenging to deal with these operational issues.

The other alternative is to start with owning a residence. When it is small, the operational issues are minimal. But, your requirements would likely grow. Assuming that you’d ideally like to avoid the “burden” of moving from one residence to another now and then, do you know if it is big enough or is expandable as your requirements grow? What about the maintenance overheads for a larger residence? In short, does it satisfy your needs for the present, or can it serve you equally well in the future? Are you sure that you will be able to manage the operational overheads associated with owning a residence? 

I don't think that every decision-maker who needs to have a residence considers all these, but they should!

The current observability solutions are also mainly divided into two similar categories. Vendor platforms like Datadog, NewRelic, Wavefront, Elastic, Sumologic, etc., all provide great starter packages and experience, with full operational offload from the user’s (renter’s) point of view. But, as the user’s footprint increases, the costs become prohibitively high. Switching to a self-managed platform at this point is not as easy as moving where you live. It not only brings the operational overheads (like managing your service requests, and maintenance) but additionally a huge, overlooked challenge, which is, migrating the data ingestion (agents collecting the data) and the querying ecosystem (be it your dashboards/alerts or other business applications depending on the data). 

On the other hand, if you always start with owning a platform, there are quite a few open-source platforms, but none of them provides a solution for all data streams (metrics, events, logs, traces, etc.). Not just that, the independent platforms for each of these streams are also not necessarily built for enterprise scale (it is altogether a different topic if these disparate platforms are the most efficient in doing what they do). In addition, each of these platforms has to be managed independently and separately (requiring separate skill sets). Overall, you end up having an operational mess!

Similar to the residence decision-maker, I don't think every observability solution decision-maker considers all these, but they should!

But, what if there was a housing developer who would build a residence as per your requirements to begin with, on the land you own/lease, provides the services you need to move to this housing (as needed), and the home warranty service with periodic inspections (for a service charge). Since this residence is on land you own/lease, you can choose to add rooms as and when you wish. Bigger the land and the house, the more you pay for property taxes to the city that gave you the land but not to the housing developer. Also, what if your starter home was free?

Kloudfuse is such a developer in the world of observability platforms. With the Kloudfuse unified observability platform, you

• Own the platform.
• Run it on your cloud account (tenant-local/in your VPC) that you pay for to the cloud provider.
• Keep your application data within your cloud account (better data security and compliance). 
• Scale from minimal configuration to as large as you need.
• Use the ingestion format (agent/collector) that you already use (or use Kloudfuse agent).
• Use it for any or all observability streams.
• Get an open standard for querying, so no vendor lock-in. 
• Get migration services for dashboards/alerts from any vendor platform to open standards. 
• Get an in-built warranty service to monitor the health, provide support, and manage the lifecycle.

Kloudfuse unified observability platform is the industry's first platform designed with following philosophies in mind:

• Data collected should provide direct value to the business outcome.
• It has to be sustainable (technology, cost etc.) for the long run for businesses.
• Cost predictability.