How to create a service catalog: step by step guide

Background on Service Catalogs

Service Catalogs are trending in the devops space. Most recently, they featured as a category in Viktor Farcic’s Best DevOps Tools, Platforms and Services recommendations for 2024.

The service catalog is, simply, a catalog of all of an organization’s microservices - to make it easier for developers (and others) to find, track and access microservices they need. 

But users will want to go beyond this basic service catalog, and add additional data so they can get some context - integrating tools from anything in the internal developer platform.

• For example, most cloud resource cost reporting tools report cost data about cloud resources, such as S3, EC2 etc or Kubernetes objects such as deployment, service, namespace, clusters and more. But this type of reporting is almost meaningless for developers, teams and business owners as it doesn’t tie into the microservice or system each of them are responsible for. 
• The same could be said for incident management, feature flags management, and more.

 You may ask why you can’t just use the data within the tools themselves. The reason is because the tools do not have the full picture of the services (as the service catalog does) and therefore can’t provide the context to users (such as “is my service in testing exceeding cost limits?”.) 

In fact, everything in the software development life cycle needs better organization and context - this is why to manage APIs you have an API catalog, and for resources you have a resource catalog. Each microservice has its own data as well as additional metadata that needs to be derived from the different tools such as AppSec, Incident Management, Cost Management and more. 

There’s More to A Catalog Than Services

All of these catalogs, data and metadata need to be housed in one place so that developers can get the answers to the questions they need and understand dependencies. This is called a software catalog. The software catalog (sometimes incorrectly used interchangeably with a service catalog) is a central metadata store for everything application-related in your engineering, from CI/CD metadata through cloud resources, Kubernetes, services and more. 

The software catalog goes beyond a ‘flat’ repository for static metadata such as ownership and logs, and can be continuously updated and enriched with context based on a team’s specific data model. It provides customizable contextual information about any resource or service such as what the service does, where it is deployed, who the owner of the service is and the stability of the service.

Most engineering and developer teams rarely view each service or resource in a silo. For example, an organization may have an API catalog which makes it easy for developers and engineering leaders to find and use APIs. But while it may be organized and make life easier for those using APIs, other parts of software architecture also require organizing, and it makes sense to ensure all of the parts are in the same place. That way you can find the API you’re looking for, but also get more contextual information about the APIs. 

Spotify’s Backstage has become synonymous with a Software Catalog, with two key use cases: helping teams manage and maintain the software they own, and making all of the software in the company discoverable. Meanwhile, Port’s Software Catalog provides the same benefits but also has the ability for organizations to bring their own data model, create dependencies between any data model element, go beyond JSON schema and offer JQ support, mirror properties, markdown and more, and enables API-based data ingestion for REST API, Terraform, GitOps and CI/CD. 

Why a Service Catalog

Microservices architecture has become the norm; whether it’s an established enterprise that has migrated from a monolith architecture or a newly formed business that has started building microservices on day one. The power of microservices architecture is to deliver large, complex applications in a much faster, reliable and efficient way by breaking them down into smaller components. This enables businesses to adapt quickly to changes in the market. Microservices can determine clear ownership over domains and their systems, they can decouple data and can enable independent deployments.

But microservices come with their own challenges - unlike monoliths which have all of the code in one place, data for microservices resides in many places. A unique identifier of a microservice can vary across these intersections, and fetching data can be cumbersome because each intersection is a different technology with a unique API. A large volume of microservices and different owners means increased operational complexity, that requires investment in time and effort from all those involved. Distributed ownership could lead to teams having their own approach to operational procedures and documentation practices, which can hugely impact productivity. On-call rotation, veering away from the Golden Path of Scaffolding for a new service, difficulties in calculating maturity level and configuration are other challenges teams will encounter when managing services.

Add to this, the need for versioning control and communication between teams about changes that could impact services, and it’s clear that engineering organizations need a plan which addresses both the team’s make up and the structure of the application or applications in hand. To overcome these challenges, businesses can use a service catalog. 

What is a Service Catalog

A service catalog is a centralized interface that has all of the information about your microservices including who owns them and what their dependencies are. Having this bird’s eye view that represents microservices in a uniform profile can help you to track all of your services and  scale your efforts. This layer of visibility should include items such as:

• Service Owner (Team or Individual)
• Slack channel used for notifications
• Who is on-call now?
• Service maturity (Security, Compliance, Performance, Tests)
• Service dependencies
• Runtime
• Packages used (In-House or External)
• Deployed version per different environments (Staging, Production, etc.)
• Link to logging system filtered by the microservice
• Link to APM system filtered by the microservice
• Link to Github repository

The additional metadata you can add here is endless, and requires your service catalog to be flexible and have a robust and secure integration strategy.

Spreadsheet as a Service Catalog

In the 2024 State of Internal Developer Portals report, over a third (35%) of engineering leaders who thought they were using an internal developer portal with a software catalog - were actually referring to using spreadsheets with microservice data. This shouldn’t be considered a true internal developer portal - partly because it doesn’t benefit from the same software catalog capabilities. 

Using a spreadsheet as a software catalog to track dependencies, software ownership and on-call personnel is fraught with issues and labor intensive. Each change would require a manual update. This also comes with a lack of standardization that can cause issues in identifying resources, assets or owners. Without up-to-date data and context, it’s impossible to answer critical questions or drive ownership and accountability. This is when a real service catalog can be updated automatically, almost in real time.

Service Catalogs in Dev Tools

Some dev tools come equipped with a service catalog. Is that enough?

For example, an incident management tool with a service catalog would enable users to see relevant information about incidents in the context of the service they are connected to. While these catalogs provide developers with a useful way to keep track of incidents, they still don’t provide the entire context. 

For instance, if a dev team was experiencing API downtime, they may not be able to find all of the information about the API itself through the incident management service catalog. Instead they would have to refer to an API catalog or documentation to find out information about the API’s last response time, who it was last used by, and its stability. 

By using a software catalog, developers would be able to quickly refer to it and isolate the problem and understand its impact on interconnected systems. For example, by looking at the latest health check response time or status, you can see whether the API endpoints are degraded, unhealthy or unavailable. Devs can then use this information to quickly identify the root cause of the issue and take corrective action. The software catalog also provides developers with additional context into the APIs provided and consumed by services. This information is valuable as a proactive measure to prevent incidents.

Service Catalogs and CMDBs

Configuration management databases (CMDBs) were once seen as the solution to the challenge of understanding all things software-related. They enabled users to store configuration information about IT infrastructure and assets, with each tracked item referred to as a CI (Configuration Item). For each CI, the CMDB holds data like the name, type, version, location, status and owner, as well as the relationship between CIs. It enables IT professionals to better manage changes, identify problems, conduct root cause analysis, resolve issues and assess the impact of changes across the IT landscape. The problem is that they’re notoriously difficult to implement - and require a lot of effort to maintain. The service catalog (and now the software catalog) are seen as the modern solution in tackling these same issues, but with the added benefit of democratized knowledge, additional context about the microservice or system such as impact on other services, ownership and health. Many larger enterprises may seek to gain many of the benefits of a software catalog by using an Internal Developer Portal and integrating it with their existing CMDB. This way their developers can use an intuitive interface to find the information they need. 

Components of a Service Catalog

Data Model

To create a service catalog you need to have a data model. The data model is the metadata schema for each of the entities you wish to represent in the software catalog and the relations between each of these entities. (For example, you might have a metadata scheme for packages, K8s clusters, microservices, and VMs – each one defining the metadata associated with these entities.)

Backstage uses a C4 model variation as its basic data model which provides a hierarchical approach to visualize software architectures built around "Context, Containers, Components, and Code." Context reveals the software catalog's broader position in the ecosystem, Containers identify major components, Components delve into internal structures, and Code showcases low-level details. ^‍

Port’s blueprints are used to define a data model, and can offer additional data models, or software catalog taxonomies.‍

Data Model Extensions 

Using integrations, you can extend the data model by adding, for example, metadata about

• Kubernetes (K8s)
• APIsCloud Resources
• CI/CD
• Packages & Libraries
• Vulnerabilities
• Incident Management
• Alerts
• Tests
• Feature Flags
• Misconfigurations
• FinOps

Dependencies

While some questions can be answered by looking at an individual entity within the software catalog (Who is on call? What version? Where’s the monitoring link? The API?), some require additional context around dependencies.

This would answer questions such as which cloud resources in the region are related to which environments and services, and what are the associated resources affected by an outage.

If the software catalog is a living inventory of anything software, cloud or in between, dependency mapping is a living map that lets you visualize dependencies in a way that is understandable to humans, and to filter what you need to see at any given time. In most cases, dependencies are created by determining relationships between types of software catalog entities.

Using a dependency graph, users can visualize the dependencies between entities in the software catalog, providing perfect context when assessing risks, troubleshooting, determining priorities and understanding the environment. 

Entities

Entities are defined through the data model and can be: microservices, cloud resources, CI/CD data, Kubernetes clusters etc, developer environments, pipelines, deployments, and anything cloud tools, libraries, services, and components – anything you define.

Software Catalog Dependency Graph

The software catalog, acting as a central hub, brings together data from services, infrastructure, and cloud assets. If the software catalog is a living inventory of anything software, cloud or in between, dependency mapping is a living map that lets you visualize dependencies in a way that is understandable to humans, and to filter what you need to see at any given time.

Scorecards

A scorecard is a way to measure and track the health and progress of each service and application within your software catalog. Scorecards establish metrics to grade production readiness, code quality, migration quality, operational performance, and more.

For example, a service maturity scorecard could track code coverage, ownership, and documentation for a critical customer-facing service. This provides visibility into whether practices are improving month-over-month.

There are three main reasons organizations use scorecards:

1. Organizational alignment: Scorecards allow you to set clear, established standards and baselines for code quality, documentation, operational performance, and other metrics. 
2. Alerting and prioritization: By monitoring service scores against thresholds, you can be alerted when a service drops below acceptable baseline levels. T
3. Driving quality improvements. Initiatives are groups of scorecards that all fit within a strategic focus area. Engineering leaders rely on initiatives to set organizational priorities – and invest team energy and focus in concerted improvements in a given area.  

Software catalogs usually allow to define scorecards for any of the entities included in them. 

Service Catalog/Software Catalog Creation Steps 

You’d probably start by thinking about the different personas that will be using the software catalog. Developers on the team would probably want to abstract away, say, Kubernetes complexity – for them, the ideal view would probably include production services, vulnerabilities, and a minimal amount of K8s. In contrast, DevOps users would want to understand the infrastructure more deeply, including cost and resource tradeoffs. The point is that there’s no “one size fits all” answer to what data goes inside the catalog and the views that different team members will use. It depends on the user personas and their needs. 

1. This is where different data models come in. You can decide between the different base models and extensions that your organization requires. The structure of the catalog is determined by the blueprints and relations defined in the data model. 
2. It’s important to ensure that your data model supports a stateful representation of the environment. For example: the running service in the classic model reflects the real world, where “live” services are deployed to several environments, such as development, staging or production environments. This is critical for providing context. 
3. The software catalog should sync in real time with data sources to provide a live graph of your services and applications. For example, integrating with CI/CD tools ensures that your software catalog will be kept up to date by making updates directly from your CI/CD.
4. Then you have to consider how you’re able to access and view the software catalog.
5. Rather than having to trawl through large numbers of pages within the software catalog, Port enables admins to curate portal pages into folders for specific use cases or teams. As a result, each persona has their own customized space within the portal with their specific dashboard pages, catalogs, etc. You can arrange folders by teams, use cases (e.g. cost management), roles and more.
   
   Port also supports no-code RBAC that lets you define access permissions based on user roles, and ensures that each persona sees only the information relevant to their responsibilities. 
   
   Finally, personalized dashboards and homepages provide meaningful insights and quick access to necessary information. These dashboards can be configured to show relevant metrics, scorecard statuses, open initiatives, or operational alerts, depending on the role and interests of the persona.
6. To benchmark each element within the software catalog, organizations can use scorecards. Scorecards are used to define and measure quality, production readiness and developer productivity.

Port: A Modern Software Catalog in a Developer Portal

Organizations can overcome difficulties they’re having with their existing service catalog - whether it’s the lack of context, too much manual information, errors, or too many services to keep up-to-date with effectively - by using an Internal Developer Portal. 

Port allows platform engineers to customize the data model to reflect the way the tech stack and use cases are in the specific engineering organization, allowing more flexibility. As it is extensible and open (ie. not limited by specific data sources) it has a more complete software catalog than alternatives and supports a broader range of self-service actions (independent actions developers can make without assistance) and automations.