How open source developer portals are changing the tech landscape

How Open Source Developer Portals Are Changing the Tech Landscape

In 2020, Spotify contributed Backstage, an open platform for building internal developer portals, to the CNCF. Since then, the tech landscape for devops and platform leaders has changed. Today, Backstage is one of the top 5 CNCF projects and is perhaps the most important catalyst in making platform engineering a core movement in almost every engineering organization. 

The 2024 State of Internal Developer Portals survey found that of the 85% of respondents reporting that they already have or that are planning to implement an internal developer portal, about 25% use backstage, with 26% of smaller engineering group size companies using backstage and 24% of larger companies doing the same.

Indeed, this movement is so strong that Viktor Farcic of DevOps toolkit said in “The Best DevOps Tools, Platforms, and Services In 2024!” that service catalogs (also known as software catalogs) have been one of the most sought after projects and services for devops.

Why have open source internal developer portals sparked such a movement, even spurring announcements that “devops is dead, long live platform engineering”? 

Indeed, the Platform Engineering movement was born of an open source developer portal, as well as developments such as platform-as-product, and a renewed focus on the developer experience and how to improve it. This is also driving the idea of portal-as-product.

The rise of open source internal developer portals

Why did internal developer portals come to life in the first place? 

There is one main explanation: the complexity of cloud native development.

The rise of microservices, cloud native practices and the increasing complexity of software development is creating a situation that is unsustainable. What this means in practice is that developers and devops are having poor experiences that lead to a drop in productivity. They either need to throw tickets at devops to do their work (and wait…) or deal with complex systems they can’t realistically have enough expertise to operate. In both cases both developers and devops are frustrated and deployment cycles slow considerably.

In many cases, the sheer complexity of microservices and the infrastructure and tools that support them, created untenable situations for developers and devops alike. 

Backstage’s OSS statistics

The results?

• An inability to track basic information about microservices, such as ownership, dependencies or who is on call. Some tried to resolve this with a spreadsheet-based service catalog, leading to endless manual work and many human errors. 
• A disconnection with developer tools leading to knowledge gaps, such as appsec or cost data. Adding data to the service catalog in a spreadsheet isn’t easy, leading to even more manual work. Can you tell which services cost the most? Present the most risk? Should be feature flagged? It isn’t easy with many disparate tools containing data and no single pane of glass.
• No ability to grade service quality or ensure standards are met. When all the data about standards doesn’t exist in the service spreadsheet, how can you add the data about feature flags, cost, or anything else? If you can’t see it, how can it be prioritized?
• Presenting many disparate interfaces to developers for self-service which further confuse them. Developers can either access cloud resources, appsec tools, and anything else in order to work, creating permission issues and adding cognitive load, or use tickets. There is nothing in between.

The importance of internal developer portals 

To understand the importance of internal developer portals, let’s first define what they are:

Internal developer portals offer developers self-sufficiency by offering developer self-service actions with the right guardrails and standards required by the organization. This reduces cognitive load for developers. At the center of portals lie the software catalogs (or service catalogs) which reflect the current state of microservices, tools and infra, abstracted for the personas using the portal, be they developers, SREs, devsecops or managers. Both these allow developers to easily make sense of anything software. 

While there are many definitions, the internal developer platform is quite simple to explain: it is made of the tools and systems that make up the software development life cycle in a given engineering organization, as well as the infrastructure, cloud and other tools and systems that enable it.

Think of the platform as what enables developer self-service: it is made of the tools required to build, deploy and manage anything in the devops world: access to internal APIs, SDKs, microservices, ephemeral environments and more. It also provides integrated tooling like CLI tools, build automation, CI/CD pipelines, and infrastructure provisioning.

How does the developer portal connect to the developer platform? Since the platform is made of all the tools, the internal developer portal connects to the platform tool APIs.

Now that the definitions are straight, let’s explain why internal developer portals are important:

• To solve for a better developer experience by offering developers a self-service interface that is simplified, leads to a golden path, ensures that engineering standards are met
• To offer a single pane of glass - the service catalog - that contains all the data about the SDLC
• To measure engineering standards by using scorecards that use the data in the service catalog, coming from all tools and systems
• To support multiple personas in the engineering organization with the visualizations that they need to track scorecards, standards, deployments, costs and anything else.

In short, open source developer portals have driven the understanding that in order to succeed, developers need a product-like interface that abstracts the knowledge they need to work, allows them to self-serve actions (such as spinning up an environment, scaffolding a service, performing day-2 operations) while providing them a single pane of glass with data abstraction that allows them to do their work well.

Backstage’s code lines changes

Transformative features of open source developer portals

The components of the internal developer portal are what make it transformative:

1. Software catalog: this is a central metadata repository of microservices, tools and resources, in context covering:

• Microservices
• Cloud resources
• CI/CD data
• Kubernetes
• Environments
• Packages
• Pipelines

2. Developer self-service: this presents developers with an interface they can use to self-serve, consisting of a personalized user interface, and the ability to set rules and RBAC. You can define self-service actions that are asynchronous, long running or that require manual approval.

3. Scorecards: using scorecards, platform engineers define standards for anything contained in the service catalog, from production readiness to K8s  scorecards and more. Using automations, you can tie scorecards to alerts (typically when they degrade) or use them to visualize standards for managers and developers. 

4. Role based access control is used to define access to data in the portal as well as the ability to perform certain actions. It also plays a significant role in managing cognitive load and abstracting information for developers.

5. Visualization: a core part of the portal is the ability to create dashboards and reports for the different personas using the portal. 

6. Automation: the ability to send automated messages and alerts based on the changes in the portal as well as provide information to workflows as part of the pipeline.

Backstage’s “Stargezer’s” distribution around the world

Challenges and opportunities in open source

While backstage has many things  going for it, not the least of them being the catalyst of a grand change in how devops will evolve, it presents both opportunities and challenges. 

Opportunities:

• As a result of its being an open source platform, backstage allows anyone to play with it, allowing for no-nonsense experimentation for many devops and platform engineering professionals
• Its plugin vision stresses the importance of including any data possible in the developer portal, truly enabling a single pane of glass for developers and platform people
• Backstage allows you to create anything on top of it, given you have the ability to code.
• Un opinionated - backstage is flexible and allows engineering orgs to create the portal that fits their DNA, within limits - that’s where the challenges come into play.

Challenges:

• While backstage uses a variation of the C4 model, representing almost anything in the SLDC stack, it still can use a little more data model flexibility. An internal developer portal needs to fit what the engineering org does - whether for K8s, the ability to represent running services in production etc. This is where blueprints, used in Port’s internal developer portal, come into play.
• Backstage plugin data doesn’t come into a central metadata repository, meaning that even though plugin data can appear in the portal, it isn’t searchable in the context of other software catalog elements and doesn’t provide enough context, insight or the ability to create a meaningful scorecard. 
• Backstage requires coding in order to get developer self-service - in react. This isn’t simple for many organizations and effectively raises the cost of ownership in a considerable way.
• Backstage doesn’t support automations, at least not without coding
• It also isn’t API first and doesn’t support Kubernetes well. 

Backstage’s pull requests creators companies

In conclusion

Open source internal developer portals formed the basis of the platform engineering movement. They offer the ability to begin to define interfaces that drive better developer experiences and ensure the organization can drive better engineering standards and compliance. At the same time, they may be difficult to use and set up, hurting their eventual adoption. 

You can try Port, a backstage alternative here and get a free account that isn’t limited in time.