Announcing the GitHub and Bitbucket Exporters

Announcing the GitHub and Bitbucket Exporters
Example H2
Example H3

As 2023 begins, things are really warming up at Port.

‍

Our GitHub and BitBucket exporters are our newest features. To understand them in context, we wanted to first explain the interplay of GitOps and internal developer portals and then the power of using exporters to populate the software catalog with GitOps data.

‍

Git and an Internal Developer Portal

Git contains a lot of information about the different entities and components you manage. From readmes, microservice names (repo names if you operate in poly repos), IaC files, K8S manifests, ArgoCD files, issues, API Documentation and pull requests. The list goes on.

On our mission to bring everything into one Internal Developer Portal, for context, self-service and even workflow automations, Git providers are a key data source to deliver on this promise.

‍

Ingesting data into the software catalog in the developer portal from your git provider

A simple ingestion of data from GitHub or Bitbucket would essentially sync the relevant YAML about the relevant component (any of service, IaC or k8s related) to the developer portal. It’s almost manual - whatever was written into the YAML  in Git is sent to the software catalog. 

This isn’t necessarily a bad practice, since requiring a service owner to maintain a YAML file is a good driver of “you build it, you own it”, since it forces the service owners to put the relevant data in the YAML file. But it potentially misses out on the valuable data contained in the Git provider.

‍

GitHub and Bitbucket contain very valuable data that would enrich any software catalog. This information (which our GitHub and Bitbucket exporters fetch) is made of two types:

1. Data about user activity: think of issues, pull requests, who ran which workflow. Perhaps hundreds of these data points are created each and every day. This is valuable real time data which isn’t truly accessible and is at no-one's fingertips. 

2. Data about services, IaC, Kubernetes & everything GitOps. The Git provider contains multiple files with important data about repos, environments and how they are structured. This appears in Terraform files for the environment, helm charts with the K8s setup and the values files for K8s, as well as the argo app. Reflecting this data in the developer portal can provide much needed insight about the state of services and infrastructure - in real-time. 

‍

The GitHub and the Bitbucket Exporters - what are they?

Our new GitHub and Bitbucket exporters are integrations that allow you to automatically collect live data to be digested by Port’s software catalog. These integrations export GitHub or Bitbucket objects (like pull requests, issues, workflows, and repositories) to Port. They also let you map your GitHub or Bitbucket resources into Port’s software catalog. 

This means that you can:

  • List and export existing Git Provider data from your organization, such as pull requests or repositories and more
  • Watch for diff changes in specific git files (commit new file / change / delete) in real-time. The changes will be automatically applied to the relevant entities in the software catalog;
  • Configure which objects are relevant, and how to map and transform objects to the relevant entities in the software catalog.

‍

How does it work?

Port collects data across your entire stack, processes it and shows it just where you want it to appear in the software catalog.

‍

Why This Matters

We’ve already explained our approach of streaming exporter data into the right place in the software catalog instead of to the microservice. Similarly, this means that the relevant Bitbucket or GitHub data will appear in the right place in Port’s software catalog. 

And the best news? We’re working on a GitLab exporter too.

Home
Blog
Announcing the GitHub and Bitbucket Exporters
This is some text inside of a div block.
This is some text inside of a div block.
{{cta}}
Developer Portal
DevOps
Developer Experience

Book a demo right now to check out Port's developer portal yourself

Read more
{{jenkins}}
Self-Service
Jenkins
Microservices
DevOps
Developer Experience

It's a Trap - Jenkins as Self service UI

Read more
{{gitops}}
Developer Experience
GitOps
DevOps
Internal Developer Platform

How do GitOps affect developer experience?

Read more
{{ebook}}
Self-Service
Jenkins
Microservices
DevOps
Developer Experience

It's a Trap - Jenkins as Self service UI. Click her to download the eBook

Read more
{{cyberark}}
Developer Portal
Developer Velocity
Developer Experience

Learning from CyberArk - building an internal developer platform in-house

Read more
{{dropdown}}

Example JSON block

{
  "foo": "bar"
}

Order Domain

{
  "properties": {},
  "relations": {},
  "title": "Orders",
  "identifier": "Orders"
}

Cart System

{
  "properties": {},
  "relations": {
    "domain": "Orders"
  },
  "identifier": "Cart",
  "title": "Cart"
}

Products System

{
  "properties": {},
  "relations": {
    "domain": "Orders"
  },
  "identifier": "Products",
  "title": "Products"
}

Cart Resource

{
  "properties": {
    "type": "postgress"
  },
  "relations": {},
  "icon": "GPU",
  "title": "Cart SQL database",
  "identifier": "cart-sql-sb"
}

Cart API

{
 "identifier": "CartAPI",
 "title": "Cart API",
 "blueprint": "API",
 "properties": {
   "type": "Open API"
 },
 "relations": {
   "provider": "CartService"
 },
 "icon": "Link"
}

Core Kafka Library

{
  "properties": {
    "type": "library"
  },
  "relations": {
    "system": "Cart"
  },
  "title": "Core Kafka Library",
  "identifier": "CoreKafkaLibrary"
}

Core Payment Library

{
  "properties": {
    "type": "library"
  },
  "relations": {
    "system": "Cart"
  },
  "title": "Core Payment Library",
  "identifier": "CorePaymentLibrary"
}

Cart Service JSON

{
 "identifier": "CartService",
 "title": "Cart Service",
 "blueprint": "Component",
 "properties": {
   "type": "service"
 },
 "relations": {
   "system": "Cart",
   "resources": [
     "cart-sql-sb"
   ],
   "consumesApi": [],
   "components": [
     "CorePaymentLibrary",
     "CoreKafkaLibrary"
   ]
 },
 "icon": "Cloud"
}

Products Service JSON

{
  "identifier": "ProductsService",
  "title": "Products Service",
  "blueprint": "Component",
  "properties": {
    "type": "service"
  },
  "relations": {
    "system": "Products",
    "consumesApi": [
      "CartAPI"
    ],
    "components": []
  }
}

Component Blueprint

{
 "identifier": "Component",
 "title": "Component",
 "icon": "Cloud",
 "schema": {
   "properties": {
     "type": {
       "enum": [
         "service",
         "library"
       ],
       "icon": "Docs",
       "type": "string",
       "enumColors": {
         "service": "blue",
         "library": "green"
       }
     }
   },
   "required": []
 },
 "mirrorProperties": {},
 "formulaProperties": {},
 "calculationProperties": {},
 "relations": {
   "system": {
     "target": "System",
     "required": false,
     "many": false
   },
   "resources": {
     "target": "Resource",
     "required": false,
     "many": true
   },
   "consumesApi": {
     "target": "API",
     "required": false,
     "many": true
   },
   "components": {
     "target": "Component",
     "required": false,
     "many": true
   },
   "providesApi": {
     "target": "API",
     "required": false,
     "many": false
   }
 }
}

Resource Blueprint

{
 “identifier”: “Resource”,
 “title”: “Resource”,
 “icon”: “DevopsTool”,
 “schema”: {
   “properties”: {
     “type”: {
       “enum”: [
         “postgress”,
         “kafka-topic”,
         “rabbit-queue”,
         “s3-bucket”
       ],
       “icon”: “Docs”,
       “type”: “string”
     }
   },
   “required”: []
 },
 “mirrorProperties”: {},
 “formulaProperties”: {},
 “calculationProperties”: {},
 “relations”: {}
}

API Blueprint

{
 "identifier": "API",
 "title": "API",
 "icon": "Link",
 "schema": {
   "properties": {
     "type": {
       "type": "string",
       "enum": [
         "Open API",
         "grpc"
       ]
     }
   },
   "required": []
 },
 "mirrorProperties": {},
 "formulaProperties": {},
 "calculationProperties": {},
 "relations": {
   "provider": {
     "target": "Component",
     "required": true,
     "many": false
   }
 }
}

Domain Blueprint

{
 "identifier": "Domain",
 "title": "Domain",
 "icon": "Server",
 "schema": {
   "properties": {},
   "required": []
 },
 "mirrorProperties": {},
 "formulaProperties": {},
 "calculationProperties": {},
 "relations": {}
}

System Blueprint

{
 "identifier": "System",
 "title": "System",
 "icon": "DevopsTool",
 "schema": {
   "properties": {},
   "required": []
 },
 "mirrorProperties": {},
 "formulaProperties": {},
 "calculationProperties": {},
 "relations": {
   "domain": {
     "target": "Domain",
     "required": true,
     "many": false
   }
 }
}
{{tabel-1}}

Microservices SDLC

  • Scaffold a new microservice

  • Deploy (canary or blue-green)

  • Feature flagging

  • Revert

  • Lock deployments

  • Add Secret

  • Force merge pull request (skip tests on crises)

  • Add environment variable to service

  • Add IaC to the service

  • Upgrade package version

Development environments

  • Spin up a developer environment for 5 days

  • ETL mock data to environment

  • Invite developer to the environment

  • Extend TTL by 3 days

Cloud resources

  • Provision a cloud resource

  • Modify a cloud resource

  • Get permissions to access cloud resource

SRE actions

  • Update pod count

  • Update auto-scaling group

  • Execute incident response runbook automation

Data Engineering

  • Add / Remove / Update Column to table

  • Run Airflow DAG

  • Duplicate table

Backoffice

  • Change customer configuration

  • Update customer software version

  • Upgrade - Downgrade plan tier

  • Create - Delete customer

Machine learning actions

  • Train model

  • Pre-process dataset

  • Deploy

  • A/B testing traffic route

  • Revert

  • Spin up remote Jupyter notebook

{{tabel-2}}

Engineering tools

  • Observability

  • Tasks management

  • CI/CD

  • On-Call management

  • Troubleshooting tools

  • DevSecOps

  • Runbooks

Infrastructure

  • Cloud Resources

  • K8S

  • Containers & Serverless

  • IaC

  • Databases

  • Environments

  • Regions

Software and more

  • Microservices

  • Docker Images

  • Docs

  • APIs

  • 3rd parties

  • Runbooks

  • Cron jobs

Our blog

Announcing the GCP Exporter
Read more
Building an Internal Developer Portal for a Serverless Architecture 
Read more
Internal Developer Portals: self-service actions using infrastructure as code and GitOps
Read more

Let us walk you through the platform and catalog the assets of your choice.

I’m ready, let’s start