Platform Engineering: What Is It and Who Does It?
You may have heard about platform engineering, a new term adding itself to the already crowded set of roles in the development and DevOps world.
In this post, we’ll cover platform engineering, how it differs from DevOps, why you might consider investing into platform engineering, and who needs platform engineering skills.
What Is Platform Engineering?
Platform engineering is the process of designing, building, and maintaining workflows and tools for software engineering organizations to drive consistency and speed up common tasks.
In addition, many platform engineers maintain an integrated software product called an “internal developer portal,” which consolidates data and knowledge fragmented across clouds, tools and teams so everyone in the engineering organization can discover essential information about their applications, services and infrastructure in one self-serve place.
Platform engineering is a new field that has arisen to meet the growing complexity of distributed software development. As software development has started involving more tools, clouds and workflows, it’s become increasingly difficult for software developers who are now asked to manage more different aspects that go into writing robust, secure, performant and capable applications.
Why Platform Engineering?
Platform engineering arose to help software developers manage the complexity of developing their applications. While DevOps aspires to automate deployment and the end-to-end running of applications, in practice, this is only achievable by the largest and most talented organizations.
In reality, when organizations remove operations and implement DevOps, specific DevOps antipatterns appear. We can see some of these antipatterns in reviewing various “State of DevOps” reports. For example, one study shows a clustering effect where the top-performing organizations have succeeded at a “you build it, you run it” approach. However, the other cluster is where this pattern fails, and these organizations suffer from the following antipattern.
Platform engineering is a new field that has arisen to meet the growing complexity of distributed software development.
Senior developers often take on the role of a shadow ops organization, so the best development resources, the ones that cost the most and are better used for increasing speed and quality within the development team are unable to do the job the company hired them to do because they’re now spending their time covering live operations and other operations-related tasks like service creation.
This leads to fragmentation of ops across the organization, often with varying degrees of quality, and it depends on the amount of time the senior developer could devote to setting it up and maintaining it.
What differentiates successful organizations from those that are not? The most successful had an internal team that is responsible for an internal development portal that supports the development teams. These dedicated teams allowed delivery teams to focus on creating functional software features, not on managing dependencies, pipelines and tools.
Challenges of Platform Engineering
One issue when adopting platform engineering is the tendency to build another silo. A good example would be a ticketing system where users can request features or report bugs, the requests go into the platform engineering realm, and are eventually resolved. You can combat this by focusing on enabling users to self-serve their own needs with your portal by providing accurate and relevant documentation, training sessions and pairing with users to solve their problems.
Another issue is prioritizing the right things. There are a lot of users from many different parts of your organization, so having a single feature request pipeline for those things users cannot self-serve with a committee deciding on priority is essential to servicing the needs of your organization effectively.
Keep your platform team adaptable and not stuck in the past ways of doing things. With the rapid pace of change in IT, it’s hard to keep up. Enablement is one way you can ease the burden on your team, but also allowing your team a consistent amount of time to train on new technologies is another.
What Does a Platform Engineer Do?
Platform engineers are responsible for deploying, and maintaining the internal development portal. They also work with a variety of stakeholders to ensure that the portal meets its users’ needs.
Platform engineers typically have a deep understanding of software engineering practices and how software engineers work. In addition, platform engineers understand what the organization is attempting to deliver and the tools and workflows needed to accomplish those goals. They also have experience with a variety of DevOps tools and practices.
What Are the Skills Required for Platform Engineering Roles?
While a successful platform engineer needs to understand software development and DevOps tools and practices, they also need to be able to work effectively with people. Platform engineering is a team sport. They need to be able to communicate with stakeholders from all parts of the organization, from the C-suite to junior developers.
They also need to be comfortable with change. Platforms, DevOps and software engineering are constantly evolving, and the flexibility to adapt to new technologies and approaches is probably the most important skill they’ll need.
How Does Platform Engineering Relate to DevOps?
Platform engineering is closely related to DevOps. Many platform engineers come from a DevOps background. DevOps is a set of practices that help businesses deliver software faster and more efficiently. It emphasizes collaboration between development and operations teams.
Platform engineering borrows many of the same principles from DevOps, including automation, continuous delivery and continuous integration.
Platform engineering differs from DevOps in that platform engineering builds the tools to help engineers and DevOps perform their tasks. Tool creation is generally not a focus for DevOps, or if tools are created, it’s done on an ad hoc basis.
Is Platform Engineering Like Site Reliability Engineering (SRE)?
Site reliability engineering (SRE) is another closely related field. Like platform engineering, SRE focuses on the availability and performance of software systems. SRE practitioners are often responsible for monitoring and responding to incidents. They also work with developers to prevent outages from happening in the first place. However, platform engineers are more focused on development, while SRE practitioners focus more on operations.
Who Needs Platform Engineering Skills?
If your organization is developing software, there’s a good chance you’ll need platform engineering skills to help accelerate your software development teams. Integrated development platforms are becoming more and more common, and they’re being used to help some of the most disruptive and adaptable software development teams in the world expedite their work.
Platform engineering is a team sport.
Platform engineering is something to consider for all organizations thinking about implementing DevOps. As businesses strive to deliver software faster, they need processes and engineers who can enable their software development teams, not hinder them.
The DevOps antipatterns are something to keep in mind while making your decisions. First, ask yourself if you can execute DevOps successfully. If not, then an internal developer portal and platform engineering may be the right choice for you.
The Tools and Applications Used in Platform Engineering
An internal developer portal is a self-service application and data store that lets developers and managers track and organize everything their engineering teams build and operate.
First off, the catalog consolidates data and knowledge fragmented across clouds, tools and teams so everyone in an organization can discover essential information about their applications, services and infrastructure in one place.
This provides a variety of utility to organizations:
- Discover all services and infrastructure components in one place and see key information about them like ownership, documentation, runbooks, cost, performance, reliability, on-call rotations, CI/CD runs and much more.
- Identify which services and machines are affected by vulnerabilities
- Alleviate on-call stress by organizing key information and enabling an engineer to trigger an incident from the portal
- Enable engineers to deploy more confidently by finding information about dependencies and upstream or downstream services
- The ability to measure and manage the organization toward desired standards
Platform engineering teams can build or buy internal developer portals. Building affords maximum flexibility to tailor the solution to the organization’s needs; however, the level of effort associated with building and maintaining it negates the value entirely unless you’re a mega-cap company. There are open source projects, including Backstage and Gimlet. The first is a DIY portal, and therefore organizations will still be required to invest heavily to attain value. Gimlet focuses on Kubernetes-based environments.
Ready-to-use internal developer portals are rapidly increasing in popularity. Configure8 is an increasingly popular option because of its end-to-end awareness of services and infrastructure, not just Kubernetes, and its advanced analytics.
Next, some platform engineering teams choose to add scaffolding services to their offering. The scaffolder consists of templates that make it simple for developers to spin up new services. This makes it easier for developers to properly launch new production services adhering to organizational standards while enjoying flexibility when needed. Scaffolders can then register the new services and related tools in the catalog so it’s always up to date.
If your organization is developing software, there’s a good chance you’ll need platform engineering skills to help accelerate your software development teams.
Scaffolders are sometimes included with universal catalogs, and they can also be created separately alongside catalogs. In this case, blueprints created in a scaffolder are then “merchandised” inside the enterprise’s catalog solution via a plugin integration.
In terms of standalone tools, there are a number of open source scaffolders, such as Cookiecutter and Yeoman, and standalone commercial software solutions like Humanitec.
It’s important to consider how often your enterprise will actually be creating new services versus managing and improving the existing system as you consider the relative priority of your platform engineering investments.
Finally, some platform engineering teams are investing in making it simpler to manage their infrastructure. The more infrastructure you have, across many accounts and clouds, the more value your organization will get from integrated management capabilities. These capabilities may enable you to change the size of an instance, provision different types of storage or resize Kubernetes clusters.
Many teams have invested in Infrastructure as Code (IaC) solutions such as Cloudformation and Terraform and would run changes through their normal update process while others may want a simple way to deep link into the resources running a service from within the catalog of their IDP. Much like with scaffolding, platform engineering teams should consider the frequency of these types of changes, compatibility with their IaC investments, and the number of users in the overall engineering team affected by these scenarios when planning their investments.
There are a variety of other tools and applications available for platform engineering. Here are some of the most popular ones:
Orchestration:
- Puppet — Helps you manage resources like servers automatically.
- Ansible — Enables you to manage your resources using code. This means you can make changes to how your resources are used by writing code that is simple and easy to understand.
- Terraform — Helps you create your portal’s infrastructure using code and then deploy it on various cloud providers. Containers:
- Docker — Docker helps you package your applications and dependencies into self-contained units that can be deployed anywhere.
- Kubernetes — A container orchestration tool that helps you manage and deploy your containers at scale.
These are just a few of the tools and applications available for platform engineers. Platform engineering is an emerging field, so expect more tools to enter the market.
The Future of Platform Engineering
As distributed cloud systems become more commonly used and architectural patterns continue to evolve, the demand for platform engineers and internal developer portals is expected to grow.
