How Twilio and Snyk Improve the Developer Experience

Webinar transcript

Roni Floman:

So, let's begin. We're really pleased to have this webinar, How Twilio and Snyk improve the developer experience. We'll be having two speakers, Asaf and Vlad which I'll introduce in a minute. The webinar is going to be recorded and we'll send you the recording and you can share it. So, if anybody has intense fear of missing anything, there's no such issue. You can use the Q&A button to ask any question, we'll probably answer the questions after the session, but if something comes up as really unclear, we may pop the question in the middle of the presenter speaking. The first person to speak will be Asaf Erlich. He's a staff software engineer at Twilio, and has over a decade of experience in building tools and platforms for other engineers. And he's currently working at Twilio as part of Segment’s developer enablement team, where he focuses on the developer experience, and that's what he’ll share. The team he's in is building Segment’s internal developer portal, that uses backstage, automated engineering, laptop setup, deployment tracking, developer surveys in managing internally deployed infrastructure for retool and Sourcegraph. And the second presenter will be a Vlad Dubrovskis. He's an engineering manager at Snyk, and he's been working with DevEx focused teams for the past five years, and currently working at Snyk’s infrastructure group, focusing on internal developer experience, because this is what this webinar is all about. The team's main focus areas are facilitating deployments across multiple environments, backstage adoption, DORA metrics, gathering insights to pinpoint how they can improve the platform experience, and building custom tooling to ease developers’ lives. Asaf will begin, and after, both of them will have Zohar Einy who is Port’s, co-founder and CEO present them with some questions. So, I am going to pass this over to Asaf. Asaf, the stage is yours.

Asaf Erlich:

Thank you. Let me just get set up here real quick. All right. Okay. Hello. Welcome, everybody. My name is Asaf Erlich, and I'm a staff software engineer at Twilio, on the Segment Dev enablement team. Segment is a subsidiary of Twilio. And today, I'm going to talk about how our team focuses on improving developer experience by focusing on the developer journey. Quick Intro about me, I have 11 years professional software engineering experience. I've worked at various places, including AWS for five years in the deploy team, three years at the Kubernetes platform team, and now I've been at Twilio, one year and seven months. I'm from Pennsylvania on the eastern United States. And in this picture behind me, I'm in the Hobbiton set in New Zealand, I’m a big Lord of the Rings fan, and this was one of the best vacations my wife and I had that I still think back fondly of, there's a lot of great sites to see there, highly recommended. Like I said, today, I'm going to talk about how our team thinks about developer journey and how we use that to decide what we own, and how we prioritize our time. One of the first things on a developer's journey is configuring their laptop. When I joined, this was basically a long wiki page with a lot of manual steps, a lot of copy pasting new installation commands, that were error prone, our support had to spend a lot of time helping people through it. And so that's one of the reasons we decided to make this experience better. Today, it's an automated laptop setup that is built on top of Ansible. It prints, it takes less than 20 minutes now for the majority of that installation, it walks you through and shows you in text, what each step says or what each step does and installs. It has color coded output for whether it succeeds or fails, and it's idempotent, so it can pick up quickly from where you left off if there's a transient issue. I also love that we have office hours where usually the on-call engineer will join and help people out, with every single engineer’s second day onboarding task. Even though we're there to help for our automation, a lot of times it's just fielding questions, people that have to ask, Hey, I just joined this company, here are some things I don't know. And we just try to help and give a human touch to everyone who's joining. On the next step of developer’s journey is often reading documentation, a lot of documentation, and maybe reading a lot of code to get acquainted with their surroundings. Discoverability is very hard, and before we improve this experience, this usually meant getting 5 or 10 links to go through, reading them, and this was an okay experience, but it was still hard to discover more. So, we created a search tool that could comb through the majority of our documentation sources, and help people to, using the keyword, get a little bit closer to what page they need to know to find out more. On top of that, we also own the infrastructure for Sourcegraph, which is a third party tool to improve the code search experience. It allows you to specify some private GitHub orgs, and search through all the repositories with Regex and other filters to help you find what you need. The next part of developer’s journey is getting acquainted with what your team owns, what are the services that they own for each service? What is maybe the dashboard and runbook links? Maybe what are your API docs? Who's on call? And what are other links you can go to? Before we centralize this experience, this usually involves bookmarking a bunch of pages that you have to remember, trying to learn a few CLIs or best scripts that you need to remember to use for certain situations. And without a centralized experience, it's not as nice, and so that's why we chose Backstage to the build developer portal. Backstage, as some of you know, is an open source framework that was built by Spotify, and it is now part of the CNCF. And it basically has a plug-in architecture that lets you pick and choose a lot of open source plugins to tailor for your developer’s needs. So, one example is the pager duty plugin you see here that can show people who's on call, and we've built a few internal ones that are, again, specific to our developers. There's a lot of possibilities for more we want to do with that experience in the future. One of the next steps on a developer's journey is building a service, maybe for the first time and deploying it. Before we tried to improve this experience, it was quite hard, there was a lot of documentation in order to get your first code to production, it involved, maybe you know about a CLI that we used to own to create a repository from a template, it usually did not build successfully the first time, you did not have a container repository, there's a bunch of infrastructure you didn't know about, you had to create first, and then configuration before you can actually deploy to Kubernetes and actually interact with your service. We recently launched this, it's one of the most popular features on our site that actually drives people to Backstage to actually build a service using templates, which is a concept that Backstage has, and actions that you can drive. So now you can go there and create a service from a template, it asks the user for some input, and then lays down on top of that a container repository and a CI pipeline that every time it merges semester, we'll publish that container image, as well as the configuration to be able to deploy later. And one of our principals recently made a series of videos called Zero Kubernetes and walks you through starting that journey at Backstage, and then of course, there's lots of other teams and infrastructure along the way. After you've started Backstage, he then walks people through the subsequent steps until you can actually deploy it, and then interact with it, and watch even metrics in stage and production. And we want to try to turn this into a training that all new hires can experience in the future. One of the later steps in the developer’s journey is actually working with other teams, interacting with other services, that means you might need to know who owns them, who owns other services. Maybe because you want to learn about other services that are similar to yours, or maybe there are other use cases that come up like you’re a security engineer, and you need to know all the languages that are out of date because of a vulnerability, or all databases that need to be upgraded for some reason. And in the past, this involves kind of reaching out on slack, or getting contacts with people you know who are owners in this area, and trying to find who's the owner and tracking down, and it was a painful process. And we called this problem for a long time taxonomy, that we wanted to improve upon. Taxonomy is a big word that basically means, who owns all of the things? And what are details about those things so that I can search by certain keywords? So, one of the reasons we went with Backstage is because it has this concept of a software catalog that you can onboard services to, we've so far unbridled 200 plus of those services, and any service that's created with Backstage, automatically, it's on boarded as well, which is nice, you can actually create relationships in those services. So, one example here on the right, a team created a service and said, it depends on five other components. The nice thing though, what's really beautiful, you can actually extend this concept, this concept of relationships and graphs that Backstage has the power to add to its catalog, we've recently started ingesting workday data. So, workday is just another third party tool that can have employee information in it. So, here we pull in employee information such as the user's email and level, you can look up, and we've just started adding team data into the catalog. So, with teams, we started adding those relationships between, what is a team part of or owning? So here we have an example on the bottom where the deploy team, which is a Cisco team to DevEx, it owns some services, and it's part of developer platform, which is part of infrastructure org. And there's just so many exciting possibilities in the future with the catalog, like adding infrastructure data and cost data.

Roni Floman:

Asaf, a very interesting question came in, so, I'll barge in with it. Ian is asking, what difficulties did you encounter when mapping the Twilio mental model to the Backstage entity model when you created the software catalog? So, if you could address that here, that would be great.

Asaf Erlich:

Sure, that's a great question. It's challenging. I think that it's a case by case basis. Some entities mapped reasonably well, and other ones, it involved a bit of a design process. There was a design doc, a discussion, we actually, this is hard because, I mentioned, we're a subsidiary of Twilio Segment, we own a Backstage instance, there's actually multiple Backstage instances at Twilio, that different platform teams for different large groups own, which is complicated. And basically, we had to create an ADR, an architect design reference that was agreed upon by all of the Twilio groups for how to generalize entities, and what data each entity will have to map to something like GitHub teams, or map to something like internal teams, or mapping to certain infrastructure that is coming up. So, each one of these requires a bit of thinking, and what data do we want there? And what data is general? And what's specific? But each one is a case by case basis. But great question. One of the final ways we try to help on developer journey is of course, providing support, and then asking people, what works and what doesn't? We've always had a support Slack channel, and we try to create a standard that we hold ourselves to of answering any question within an hour during work hours. This is often about our tools, but because we're the pane of glass both with Backstage and with our support for the rest of the infrastructure org, a lot of times people come to us with just general questions, and they don't really know the root cause. And we try to work with them to figure out, what's the best team that can help them and route them to the right location? On top of that, before we had a centralized experience, we didn't really have good data on what parts of our tools people liked, what features they cared about, more or less. But now that we have a centralized developer portal, we've, in this example, installed the Google Analytics plugin, which lets you pipe data on lots of things about what features are used more or less. An example here is looking at page views what pages are people visiting? And that kind of data can really show us and help us. And finally, we recently started doing a quarterly survey. We're now doing it for the second time, technically the third time for this coming year, and this really gives us some great new insights. DX is a third party company that we partner with to deliver this and then that helps us both see on a team level and aggregate level and also see long term, did this initiative that we went about trying to improve this experience, did it actually work or not? And this is a small glimpse to how our team approaches and improves developer experience at Twilio. I hope you enjoyed and I'm happy to continue answering more questions.

Roni Floman:

Which we indeed will answer more questions. It seems to me that Vlad has a ton of questions, because I keep seeing his chat comments. So, Vlad, it's your turn. So, let's share your presentation and go ahead.

Vlad Dubrovskis:

Yeah, I think to set the scene maybe a little bit. The developer experience team at Snyk is, I say, relatively new, we've been together for about eight months. And our engineer joined us about four months ago. So, I think a lot of things that, as I just present it, I'm going to be talking about are from the earliest stage in our journey. So, let me start. I'm assuming you can see my screen, if not, please do shout. So hello, everybody, my name is Vlad, I'm engineering manager for DevEx team at Snyk. And today, I'm going to cover, where do we sit in our organization and why? Some of the challenges that we encountered in our seven, eight months that we existed as a team, and some of the future plans now we understand things a little bit better, kind of like, what do we plan to do in the 2023? And to begin with, what is Snyk? Hopefully, you've heard of us, if not, Snyk is a security software that helps you stay secure. It does so by automatically finding and fixing vulnerabilities in your code, infrastructures called containers, etc. And because we're a software as a service company, we have lots of engineers using our software across the globe in different companies, mostly focusing on engineers. But also, we have lots of engineers inside the company building Snyk day in and day out. And we, our team is supporting the engineers that are building Snyk. The way we are set up is the way we've been in the organisation, we are part of the platform division infrastructure group. And actually, in the same group, we’re sitting alongside SRE team, and call platforms. You might wonder, why an infrastructure group? I'm going to be referring to something called Polaris. This is a what we call our internal developer platform for managing numerous production environments. It's Snyk’s platform as a product, because Snyk has got multiple live environments, not a single one, because of different variations in different regions, etc. And this is a standardized and opinionated way of using the tools we provide, that enables repeatable Snyks. So, if we need to spin up a new instance of Snyk somewhere in a new region, we can do it in an automated way. And what Polaris supports and provides developers with, is things to help them with their deployment, testing, security, role of maintenance, monitoring, and alerting. So, our focus as a team is mostly on Polaris and developer experience for new and existing services. But two other areas where we’re focusing on is also service catalog, also Backstage, where we have service templates, also focusing on documentation, trying to improve that and few other bits and bobs there. And the third area of focus is, engineer insights, which also goes into two different buckets, because one of them is about getting insights to optimise engineering efficiency flow. So, example is DORA metrics. And actually, we had some successful data in our company, where senior stakeholders use DORA metrics to prioritise some of the technical depth. So, we had some decomposition work to do. And the data we traded had been used to justify that and prioritise it, which was very nice to see. And the other bucket is user interviews and surveys, etc. I call it UX box of tricks, we’re borrowing those things, we're not inventing ourselves, we’re borrowing it from user experience overall. Now just the challenges that we encountered in our short journey so far. I think the first one is around managing expectations. Because the moment people hear we have a developer experience team, people go, great, we have a DevEx team, they're going to fix everything. Well, the team have a limit on how many things we can do at one time. And few things that really helped us along the way is defining a mission and focus area. So, our mission as a team is that, developing at Snyk should be a pleasure with safe and intuitive paths, and the focus area where we're focusing that is our internal platform, Polaris. Again, because we're not the only team in the group, and we don't own all the moving pieces of Polaris, one of the things that really helped us is understanding our domain and ownership. Because we need to understand, what is the SRE team responsible for? What are cloud platforms responsible for? And what DevEx is responsible for, which moving pieces? Once you have that, you learn to say, no, and you learn to say no quite well, because as the developer experience team, the last thing you want to do is not to be helpful, because you are there to serve the engineers and to help them. But you have to say no sometimes. And one of the things that can really help with all of the above is aiming to have a roadmap, because if you have a roadmap and you put things there in a visible way, it can save a lot of time and a lot of conversations and efforts for everybody. The next thing that we encountered as a challenge is, especially when you start like this, there's a lot of foundational pieces that need to be done, that can be built. Because where we started, we didn't really have any feedback loops, real feedback loops, you can hear, we have ideas, people have what needs to be prioritized, but we had to think about, how do we do surveys? How do we use user interviews? We also have a support channel, and we had to analyse data there to understand, what are the most common asks to help prioritize focus for us? Or when a support ticket is completed, we also do a survey automatically to ask the person, Hey, did you get the right answer? Are you happy with it? What could be improved? Again, just an extra thing where we can feedback to our team to help us prioritize things. Data insights like engineering metrics like Dora, that was the first thing that the team has built, and it has been quite good. Further down the line, we have ideas for developer satisfaction, we even have some not very nice ways to find out, what is the latest version of internal software being used? And again, something we plan to improve and make a lot more automated, that at the moment is a bit of a manual step. Something that has a fashion, it was called a taxonomy, but bringing order and structure, we call it metadata structure and validity, because if you go into data insights, and try and collect the data, you want to make sure that you can reason with it, and you can describe it somehow, and you can attribute it to different teams or groups, etc. It's like, it's going to be very interesting challenge. We have some ideas, and we input a few things, but there's going to be again, something we're going to look forward a lot more. Actually, overall, because we are a part of the platform team, thinking about platform capabilities, because sometimes you need to understand the challenges that exist and what platform capabilities might be missing, because you might be building something right now that is not super exciting. However, it might unlock value further down the line to optimise flows and help developers along the way.

Roni Floman:

The fact that you said DORA metrics, just created a flurry of questions. So, I'll go with the first one. How are you grabbing DORA metrics and other developer experience insights? And do you maintain your own data pipeline to ingest this data? I’ll ask another one.

Vlad Dubrovskis:

Okay, so with that one, Yes, so we do have our own data pipeline, we use the data from our CICD, we use circle. So, we go for API's, we collect the data, we structure it, and then we put it into Looker and snowflake, where we reason with that data. For incidents, we didn't have incident data for a while, but now we have a vendor for managing incidents, this is going to be the next thing to add, which was missing so far which was frustrating. But yeah, we do it ourselves, we don't use a third party tool, we understand how things connect, and we try to surface the data.

Roni Floman:

And there's another question here, which is quite long about engaging with the user interviews. So just drop a sentence or two about, how you do user interviews if you do them at all, and then we may return to that later in the Q&A section.

Vlad Dubrovskis:

Yeah, with these interviews, it's something, I'm going to touch on that a bit later on, another challenge but it’s something, nobody on the team is an expert UXer, it's something we're doing, we're trying for the first time. We know it might not be ideal, so we thought about it like, the questions we want to ask. And the first user interview we did, I think we found 20 candidates, and then we created some of the personas, and we tried to map them, for example, an Associate engineer who just started their career, somebody who is a senior engineer, staff engineer, and maybe a front end engineer. We try to talk to a variety of people asking the same questions to see for trends, what is difficult at the moment? And then we plan to like, once we identify those areas, we're going to have a follow up chat with them to dig a bit deeper to understand for opportunities. But I can talk about it for hours, it's a very big topic.

Roni Floman:

And a difficult one I understand. 

Vlad Dubrovskis:

Yeah, absolutely. So, back to the slides. So, one of the things, like team focus, I think it's unreasonable for any team to be honest, to expect that 100% of your time is going to be coding. But I think for DevEx team, this is the breakdown I borrowed from my manager Crystal. She'd done a similar chat about, where does the time go for different teams and platform? And the idea that for DevEx teams, an approximate breakdown will be, 10% will be consultancy work with other teams, 24% for advocacy and training, for example, a new cool feature of the platform, 10% of focusing and implementation, 10% planning and architecting because obviously, we always have to do things like this, about 30% of building and integrating. It doesn't mean always building new tools, sometimes, just having a vendor integrating it with our existing tools, and probably about 20% goes to operating, because the things you build, you can't just leave them alone, you have to sometimes update things, sometimes add features, sometimes things just break, it’s just part of software lifecycle. And the last thing into the challenges I want to touch on a very briefly is, in DevEx teams, what is expertise versus good enough? And what is the expertise needed? Because I believe DevEx is at the intersection of technologies and approaches, this is just from our last 7 months as a team, the technology we use like, Kubernetes, AWS, GCP, different cloud providers, Helm, Terraform, for data, we had Looker snowflake, nobody had experience with that. For languages, we had to do Python, Golang, JavaScript, both node and react, again, thank you for Backstage, because I didn’t know how to do react. And we had to wear lots of different hats, like being a data engineer, user research and analyst product, like tech lead, because people need to focus on that. And it can seem quite daunting, I personally think it's quite exciting. But at the same time, as a manager, I think it presents a challenge of, there's no perfect hire, for certain roles, it's very clear, what the expectations are, here, the context, the situation, roadmap at the time will require the team to adapt. And it's something I think is quite important, to be open minded if you want to be working in a DevEx team. Very briefly, I think I'm taking so much time already, possibly future plans. So, the main thing we really want to focus in the next year is reducing the cognitive load. We want to abstract API servers to a higher level, when we did some of the user interviews, one of the things we learned is that most developers, they just want to focus on delivering the value, like working with feature, they don't really want to navigate infrastructure, or think about how to provision a database, they want it to be easy. It's not something that's currently super easy, and there's opportunities at Snyk where we can make it a lot easier to have a super simple abstraction layer, and we have some plans for that. I think at the moment actually, my team is looking into doing a hackathon, trying to get a few ideas out. As part of that as well, one of the things that we have to do is, we provide different Helm charts and Terraform modules for databases, to spin up the database for servers for example. And because they need to be maintained over time, if a new feature gets added or a breaking feature, one of the tricky situation that happens is that, at the moment, the teams need a bit of handholds, because it's a scary high risk cage. And again, something we want to do is make it easier and not high risk, make it something very normal. And because of the first two main points, we have so many, like if you have a high level of abstraction, and you have lots of systems underneath, when things go wrong, you want to make sure that the error propagated back to the user, to your developer doesn't send them on a quest, like from the Shire to the Mordor, I also like Lord of the Rings. So yeah, that's something to be very mindful about. Next thing about, I think, again, we’ve done DORA metrics, but we have a lot of plans for engineering insights overall. The idea is to help create visibility across all layers of engineering, from Devs to EP, making engineering metrics visible and track it over time. The value for engineers is like, I'm pretty sure probably everybody had experience with production readiness checklist, right? Usually have a document somewhere, where there's 20 things and you have to read and understand each one of them, and it's never a pleasant experience. With things like Backstage, you can automate most of those checks. So, you can go to your servers and immediately see green ticks or red ticks against them and saying what you need to do, it can be automated for developers. For directors and VPs, you can answer things like how is certain migration going, for example, if there's a vulnerability in some package, we want to track it, we will be able to provide that. Again, we have plans for that, nothing implemented yet, talk to me in about six months time, there'll be a really cool story to tell. Just one more slide of this, I promise. Again, tech insights and scorecards, I think has got huge potential for both engineers and also for directors and VPs at all levels of engineering. Service templates, we have currently, two service templates, one for Golang, one for node, but I think they can use for much more than just creating a service. I think again, because it's automations, you can automate any steps. One of the things we're going to be doing as well, because we have some custom things in our open API, we want to create a better rendering for our engineers to surface certain details. And, again, from more user research point of view, we want to understand, what do you want to present you with depending on, what hat are you wearing. Are you wearing a hat of, hey, I'm an engineer responsible for the surveys versus, hey, I'm an engineer who wants to integrate with the surveys and understand how to consume it? And the final one is, again, something we haven't done yet, like a proper survey, but absolutely focusing on having that is like a repeatable way to track impact over time, because that will help us understand and ask some questions like, What is getting in the way of delivering customer value for the teams? And overall, are developers happy to work at Snyk? For example, something like NPS score. Okay, I'm almost out of breath, but that's all for me.

Roni Floman:

So that's like the coolest ending slide ever. And I'm going to share the Q&A slide or maybe Vlad, share your slide, it's actually much nicer, we’ll go into the Q&A section. We're going to start with the Q&A section with Zohar asking, but it seems both of you want to ask each other questions. So, maybe Vlad, you ask Asaf a question, and Asaf, you return a question, and then we'll go into Q&A.

Vlad Dubrovskis:

I mean, I can go first because while Asaf was presenting, I think there was loads of very interesting things really there, which I…I think one of the really interesting things that I found while you were talking about was, we call it metadata and you call it taxonomy, which for me was like, this is such a big topic, again, probably a separate topic. Because it doesn't seem like something super exciting when you start thinking about it first, because you go, oh, taxonomy, metadata, but the amount of value is can unlock, and the amount of insights is amazing.

Asaf Erlich:

Yeah, I mean, this is something that certain people in my group in develop platform, have been very excited about for a long time, and have tried really hard to sell other parts of the company about how important it is. And we're solving it for ourselves first at Segment, and I think that as time has gone on, other parts of the company have become more and more interested, because like you said, it sounds almost boring at first, but so many situations people need to look up information about ownership or information about products or services and who owns, and there's just a lot of possibilities if you keep extending down that road, keep pulling that thread, that you can build on.

Roni Floman:

And Asaf, your question?

Asaf Erlich:

My question is similar, more about DORA metrics and getting developer interviews and figuring out… I think DORA metrics has two use cases, there's the, I want to audit every single deployment, I want to during an incident be able to figure out what has changed. And the other use case is, I want to know how developers are doing. And I think they get mixed together, but they're different use cases. But yeah, you touched on this in the presentation, but I want to know what more you can share. 

Vlad Dubrovskis:

Yeah, those are definitely the core use cases there, but I think it depends, again, I think you have to think about, who is using it? Again, think, UX tricks or product tricks? What is the persona that is consuming DORA metric? I just had during an incident, responder, you can consume it in a way to tell you what changed, very quickly surface that information up. As a team, maybe an engineer, you can go in the data and you go, wait a second, it seems to be like we’re not doing that well, but actually, maybe it's okay. You can have those meaningful conversations, a very good example I’ve heard before was, oh, we deploy so many times a day, but actually, the incident rate is very high as well. So maybe it's not a very good thing, maybe you should consider maybe not deploying as often, but think about the quality and the change to go in a different direction. Yeah, also, I think DORA, obviously, accelerate book became very popular, but like I said, I think it's a small piece of the puzzle, I think scorecards and tech insights, and, again, with taxonomy, I think it answers so much more. One of the conversations I had was only recently, the biggest challenge in engineering teams, and tell me if that sounds familiar or not, you go, Hey, we should do this cool feature, because it's going to help engineers move faster, and you go, Okay, how are you going to measure it? Ah, yeah, I’m not sure, it’s probably not, you know what, it doesn't matter, because trying to measure it is probably going to take longer than building it. And then you prove you can build it. But if you can make that data available right away, and you have a question you can answer right away, just like the product does usually, I think that will enable teams a lot more. I don't think maybe it doesn't seem obvious straightaway, but I think if you surface those metrics, it can really help you empower engineers to prioritize certain things a lot more and a lot better. 

Asaf Erlich:

That's a great point. I've heard a term in the wild more recently called observability driven development, to capture what you're saying, you should think about how you're going to measure this before you release it. And I love that, that's a good way.

Vlad Dubrovskis:  

I'm taking notes, observability driven development.

Roni Floman:  

And now we'll move to some more questions and answers with Zohar. So, Zohar go ahead.

Zohar Einy:  

Yeah, so I really enjoyed your session here. And one of the questions that we got was, how does developer experience play with the DevOps teams? Essentially, I think that originally, developer experience was part of the DevOps responsibility. So how does it play with high familiarity, with infrastructure and stuff like that?

Asaf Erlich:  

I guess I'll go first. So, in our org, the developer experience is one of three teams, part of developer platform. So other two teams are CI and deploy. And then there are three other platform teams that make up the entire infrastructure org, which are, compute platform, observability platform and the data platform. So, you can see how we're at a top layer, and like I said, sometimes, we triage a lot of support and then send to them, and you can imagine what things compute observability and data own relative to ours

Zohar Einy:  

Another question that we got is… so basically a lot of questions that you already answered. I think one of them would be, are there some guides for internal developer surveys, like some tips are do's and don'ts that you learned from doing surveys?

Vlad Dubrovskis:  

Our team hasn't done surveys yet, we have some ideas, we’ve done some groundwork. I think I'll be interested to hear more from Asaf because I spoke to GetDx.com because I know they do the surveys. I think they take a lot of groundwork off your hands so you don't have to do it, so you just pay for the solution. So, I'm quite curious kind of Asaf’s experience.

Asaf Erlich:  

Yeah, so I'll start off with where we were before we got a hold of the DX and started developer surveys. We went through a round of, I would say six months to a year, but I wasn't part of the leadership team that did this. We did basically, listening tours/interviews with all the product teams. This hoarded a very big company scale, but it was a good way for us to also introduce ourselves, at least one time, or rebrand ourselves, because we used to actually not have the developer platform, developer experience team. There was a team before, and that just was called the tooling team, but that became harder to scale. So, I think quarterly survey is better. My biggest recommendation, so DX kind of uses this to some extent, but there's a framework called the Space framework that Nicole Ferguson, PhD, she wrote this article, I reread it recently, it tries to explain this, what you should ask on something like quarterly survey, in the space framework, each one stands for something like, satisfaction, performance, activity, communication, collaboration, and efficiency and flow. And in each one, and this is available online to read, they have examples, questions or topics on a rating scale that you can ask. So that's really inspirational if you don't want to go to DX, and I'm not here to sell them to you, there are other solutions out there, and you should evaluate. But that's a good starting point to ask those things, and I think it can give you a lot of good starting insight.

Vlad Dubrovskis:  

I think, because one of the chats I had in a meetup a month ago, don’t remember the company name, I forgot, but what they said they're using, a couple of different tools, not focusing specifically on DX, but just focusing on customer satisfaction, and it measures different sentiments. An interesting observation they shared was that because it gives you different scores for different things, and they tried different things, nothing had very big impact until they started focusing on one measure, which is happiness, are people happy? And it sounds really weird, but apparently, when they started focusing on that and how to improve that developer life, it really helped move other metrics ahead as well. It's a sample size of one I've heard from, I'm just sharing. I mean, you know that I absolutely loved listening tour, it’s something I'm definitely going to borrow, because one of the interesting concepts I learned last year from my manager, Crystal, is I’m not sure if you’ve heard about the four types of work. And it’s work as imagined, work as prescribed, work as disclosed, and work as done. And I think surveys versus observing versus listening, they help you see different types of the same work to get a better insight. So, I will say, don't just do survey, if you do those things, do a collection of them to have a better understanding.

Zohar Einy:  

Amazing. Yes, so basically, we're out of time, and I want to personally thank you for making time for this session. Personally, I really enjoyed the points that you brought up, and learned a lot. And I hope that we managed to make developers a little bit happier every day. So, I want to really thank you for that and the participants. And we didn't cover all the Q&A, we had a lot of questions coming in, but I hope we touched the main points and we’ll follow up with a recording of the session, so don't worry about that, you can listen to it also offline. And thank you everyone for joining us for this session. And it's been a pleasure.

Asaf Erlich:  

Thank you so much for hosting, and this was a pleasure for me as well.