Crossplane: Introduction to External Resource Management with Kubernetes

Introduction

It's quite likely that at some point in your career, you've had to deal with infrastructure provisioning, possibly using infrastructure-as-code tools like Terraform (a topic I'd love to delve into in the future).

Terraform maintains a state file that records the current state of managed infrastructure. Due to this fundamental nature, configuration drifts can occur, referring to situations where the current state of deployed infrastructure doesn't match the state defined in Terraform's configuration. This can happen for various reasons, such as manual changes to infrastructure made outside of Terraform, failures in applying planned changes by Terraform, or conflicts between different infrastructure management sources. This is where control planes come in. In this article, I aim to kick off an introduction to Crossplane, explaining its architecture and how you can benefit from control planes for your infrastructure provisioning.

Obs: Note that the previously mentioned point does not prevent the use of Terraform. We acknowledge its applicability and understand that it can be used in conjunction with external scripts to meet the same expectations of a control plan. The central idea of this article is to offer new possibilities for infrastructure provisioning using control plans.

Crossplane

Crossplane is an open-source tool developed by Upbound and integrated into the Cloud Native Computing Foundation (CNCF), based on Kubernetes. It enables the extension of Kubernetes for provisioning and managing infrastructure resources. Expanding Kubernetes' capabilities, it introduces a resource model defined as XR - Composite Resource (we will delve into this concept in detail in the upcoming chapters), allowing the definition and management of infrastructure resources using the same API and tools as Kubernetes. These composite resources are defined through Kubernetes Custom Resource Definitions (CRDs). If you have experience with Kubernetes, you're probably familiar with these terms. If not, don't worry, we will explore all of these terms in the upcoming chapters.

You can find detailed information about the project by accessing the official Crossplane website or its page on the CNCF.

Custom Resources

It becomes evident that Crossplane's main differentiator is its effective use of the Kubernetes API. By extending Kubernetes, Crossplane adds infrastructure resource management capabilities across various public and private clouds in a declarative manner through Custom Resource Definitions (CRDs). With Crossplane, new resource types representing infrastructure can be defined, enabling seamless control of resources and providing the same management experience as with a Kubernetes cluster.

Crossplane's custom controllers monitor these new custom resources and interact with cloud APIs via providers to create, update, and delete infrastructure resources. These controllers ensure that the desired state, as defined in the custom resources, is realized in the underlying infrastructure.

Managing infrastructure resources with Crossplane is done declaratively, using the Kubernetes Resource Model (KRM) methodology. This allows a single API schema for each resource to serve as a declarative data model, functioning both as a source and a target for automated components, and even as an intermediate representation for resource transformations before instantiation. This approach enables operators to employ DevOps practices like GitOps for consistent and auditable infrastructure management.

Control Plane

The Crossplane extends Kubernetes' Control Plane, which is responsible for managing the desired state of cluster resources, by adding the capability to manage external infrastructure resources such as databases, networks, and storage services in a declarative manner.

In Kubernetes, the Control Plane oversees the lifecycle of resources within the cluster, such as pods, services, and deployments. It achieves this through components like the API Server, Scheduler, Controller Manager, and etcd. Crossplane introduces an additional layer of functionality to this Control Plane by introducing new resource types that represent infrastructure outside the Kubernetes cluster.

As mentioned earlier, Crossplane utilizes Custom Resource Definitions (CRDs) to define new resource types that describe infrastructure. These CRDs are then registered with the Kubernetes API Server, allowing users to create, read, update, and delete these resources just like any other Kubernetes resource. In addition to CRDs, Crossplane implements custom controllers that are responsible for reconciling the desired state defined in infrastructure resources with the current state of external infrastructure. These controllers continuously monitor the custom resources and make API calls to cloud providers to ensure that the desired state is maintained.

1234567891011121314151617181920212223242526

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myresources.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                message:
                  type: string
  scope: Namespaced
  names:
    plural: myresources
    singular: myresource
    kind: MyResource
    shortNames:
      - myres

To draw an analogy, imagine you are building a house from scratch. The control plane would be akin to the architect and engineer responsible for the entire project. They do not physically build the house but are responsible for planning and designing all the necessary details to ensure the construction happens as desired.

Similarly, Crossplane's control plane is responsible for defining and managing all infrastructure configurations, policies, and resources. It does not directly execute operations but orchestrates how and where cloud resources will be provisioned and configured, ensuring everything aligns with defined needs and policies.

Just as an architect ensures a house is designed to client specifications and building standards, Crossplane's control plane ensures your cloud infrastructure is configured consistently, efficiently, and securely, following guidelines defined by you.

Inside the Cluster

Starting now, we will begin installing Crossplane on our cluster. Throughout this process, we will explain each Crossplane concept being applied. As I often mention in other articles, don't feel pressured to finish reading quickly. Absorb the content, and if necessary, try building and tearing down as many times as needed.

If you don't have a cluster on your machine yet, we can install Minikube to set up Crossplane.

In your terminal run:

bash

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64

Start your cluster:

bash

minikube start

After initializing the cluster, we will install Helm.

In this article, we will use the script installation method, but you can choose your preferred installation option. For more details, please visit Helm installation.

First let's download the installation script

helm install

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3

Let's add some permission.

permission

chmod 700 get_helm.sh

And finally, we run the installation script

install

./get_helm.sh

If everything went well, you should have an output similar to this when checking the version.

version

helm versionversion.BuildInfo{Version:"v3.13.1", GitTreeState:"clean", GoVersion:"go1.20.8"}

Add the Crossplane repository with the helm repo add command.

crossplane installation

helm repo add crossplane-stable https://charts.crossplane.io/stable

Update the local Helm chart cache with helm repo update.

helm update

helm repo update

Install the Crossplane Helm chart with helm install

helm update

helm install crossplane --namespace crossplane-system --create-namespace crossplane-stable/crossplane 

View the installed Crossplane pods with kubectl get pods -n crossplane-system.

crossplane system

kubectl get pods -n crossplane-systemNAME                                                       READY   STATUS     RESTARTS      AGEcrossplane-5d9fb9fc48-xrjvb                                0/1     Init:0/1   2             55dcrossplane-rbac-manager-6d57b8598-qptxh                    1/1     Running    1 (52d ago)   55d

For our first step in the crossplane we will install the s3 provider responsible for s3:

123456

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-s3
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v1.1.0

Don't worry, we'll delve deeper into providers in the next chapter.

Providers

If you've worked with infrastructure as code using Terraform, you're likely familiar with the concept of providers. Providers are an abstraction that translates the resources defined in Crossplane into specific resources of the underlying provider. For example, an AWS provider will translate Crossplane's resource definitions into AWS API calls to provision and manage EC2 instances, S3 buckets, or other AWS resources. Similar to Terraform, this abstraction layer in Crossplane makes it a multi-cloud and multi-platform infrastructure management tool, enabling users to manage resources across different environments in a unified way.

123456789

apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  generateName: crossplane-bucket-
spec:
  forProvider:
    region: us-east-2
  providerConfigRef:
    name: default

Let's dive a bit deeper and understand how this abstraction works using the AWS provider as an example. When we define an S3 bucket using an XR Composite Resource, how does Crossplane translate this into a real bucket?

The code below is taken from the official AWS provider repository for Crossplane .

1234567891011121314151617181920212223242526272829

func (e *external) Create(ctx context.Context, mg resource.Managed) (managed.ExternalCreation, error) {
	cr, ok := mg.(*v1beta1.Bucket)
	if !ok {
		return managed.ExternalCreation{}, errors.New(errUnexpectedObject)
	}

	_, err := e.s3client.CreateBucket(ctx, s3.GenerateCreateBucketInput(meta.GetExternalName(cr), cr.Spec.ForProvider))
	if resource.Ignore(s3.IsAlreadyExists, err) != nil {
		return managed.ExternalCreation{}, errorutils.Wrap(err, errCreate)
	}
	current := cr.Spec.ForProvider.DeepCopy()

	errs := make([]error, 0)
	for _, awsClient := range e.subresourceClients {
		err := awsClient.LateInitialize(ctx, cr)
		if err != nil {
			errs = append(errs, err)
		}
	}
	if !cmp.Equal(current, &cr.Spec.ForProvider) {
		if err := e.kube.Update(ctx, cr); err != nil {
			errs = append(errs, errorutils.Wrap(err, errKubeUpdateFailed))
		}
	}
	if len(errs) != 0 {
		return managed.ExternalCreation{}, k8serrors.NewAggregate(errs)
	}
	return managed.ExternalCreation{}, nil
}

The process begins by converting our resource into a *v1beta1.Bucket, which is a comprehensible interface for the provider. Then, we establish a direct connection with AWS and create the resource in the cloud. It's important to note that for each sub-resource client, such as the bucket policy client, we perform lazy initialization of the resource. If we identify discrepancies between the current state and the desired state of the resource, we update the state in Kubernetes. If errors occur during any step of the process, they are aggregated and returned for proper handling.

Overall, the abstractions in providers follow a similar logic to the one mentioned earlier. These abstractions are designed to convert resource definitions into understandable interfaces for the corresponding provider. They then establish connections with relevant cloud services to create, manage, or update these resources. During this process, it's also common to handle lazy initializations of sub-resources and ensure that the resource state in the provider is synchronized with the desired state defined by the user in Kubernetes.

In the next chapters, we'll install Crossplane and understand how to use a provider to create a resource.

Now that we understand what a Provider is, let's continue with the installation of our aws provider.

Run:

providers

kubectl get providers

If everything goes well, you should see something similar to this:

providers

NAME                          INSTALLED   HEALTHY   PACKAGE                                              AGEprovider-aws-s3               True        True      xpkg.upbound.io/upbound/provider-aws-s3:v0.40.0      55dupbound-provider-family-aws   True        True      xpkg.upbound.io/upbound/provider-family-aws:v1.4.0   55d

The provider requires credentials to create and manage AWS resources.

Generate a Kubernetes Secret from your AWS key-pair and then configure the Provider to use it.

Access the AWS documentation to retrieve your keys. After retrieving them, let's save them in a text file with the following structure:

aws secrets

[default]aws_access_key_id = YOUR_AWS_ACCESS_KEY_IDaws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY

After that, we will create a secret in Kubernetes with your AWS credentials.

k8s secrets

kubectl create secret generic aws-secret -n crossplane-system --from-file=creds=./aws-credentials.txt

To finalize the configuration of our provider, we will apply our credentials using the ProviderConfig.

provider config

cat <<EOF | kubectl apply -f -apiVersion: aws.upbound.io/v1beta1kind: ProviderConfigmetadata:  name: defaultspec:  credentials:    source: Secret    secretRef:      namespace: crossplane-system      name: aws-secret      key: credsEOF

The next step now is to create our first managed resource.

Managed Resource

Essentially, a Managed Resource (MR) represents an external service within a Provider. When creating a new Managed Resource, users trigger the creation of an external resource within the Provider's environment. Each external service managed by Crossplane corresponds to a Managed Resource.

Run:

managed resource

cat <<EOF | kubectl create -f -apiVersion: s3.aws.upbound.io/v1beta1kind: Bucketmetadata:  generateName: crossplane-bucket-spec:  forProvider:    region: us-east-2  providerConfigRef:    name: defaultEOF

After executing the command above, Crossplane may take a few seconds to create your resource in AWS. You can check the status of the Managed Resource (MR) with the following command:

managed resource

kubectl get buckets

You should get a response similar to this:

managed resource

NAME                      READY   SYNCED   EXTERNAL-NAME             AGEcrossplane-bucket-6kc2g   True    True     crossplane-bucket-6kc2g   52d

You can check your buckets in AWS and should see the same resource in your cloud. In the next chapter, we will understand how to better extend the creation of Managed Resources (MR) with compositions.

Composing Resources

As previously mentioned, when creating a managed resource, we are establishing an external service in a provider. Compositions, on the other hand, operate from the perspective of creating one or more resource abstractions together. The idea of creating just one resource through a composition arises from the high level of customization it offers. In this article, we will explore the creation of two manageable resources through a composition. Imagine that in your application, you segment separate environments. You want to create a VPC and a subnet for each environment. Doing this manually would be quite labor-intensive; therefore, this is where we can create a composed resource that configures a VPC, a subnet, and automatically assigns the subnet to the VPC.

See the file subnet.composition.yaml:

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: subnet-composition
  labels:
    provider: aws
spec:
  compositeTypeRef:
    apiVersion: eks.example.org/v1alpha1
    kind: CompositeSubnet
  resources:
    - name: vpc
      base:
        apiVersion: ec2.aws.upbound.io/v1beta1
        kind: VPC
        metadata:
          labels:
            testing.upbound.io/example-name: my-vpc
          name: my-vpc
        spec:
          forProvider:
            cidrBlock: "10.0.0.0/16"
            region: us-west-2
          providerConfigRef:
            name: default
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.cidrBlock
          toFieldPath: spec.forProvider.cidrBlock
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.region
          toFieldPath: spec.forProvider.region
    - name: subnet
      base:
        apiVersion: ec2.aws.upbound.io/v1beta1
        kind: Subnet
        metadata:
          labels:
            testing.upbound.io/example-name: my-subnet
          name: my-subnet
        spec:
          forProvider:
            cidrBlock: "10.0.0.0/16"
            vpcIdSelector:
              matchControllerRef: true
            region: us-west-2
          providerConfigRef:
            name: default
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.cidrBlock
          toFieldPath: spec.forProvider.cidrBlock
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.region
          toFieldPath: spec.forProvider.region

Note that the kind refers to a Composition. We define that our composition will be named subnet-composition and will have the label provider: aws, which will help us identify this resource later. Below, we create an array of resources. These resemble a managed resource, with some details, such as the vpcIdSelector in the subnet, which references the VPC created earlier. All the parameters used are available directly on the Upbound website, within the AWS resource family.

If you are familiar with Kubernetes, you probably already know about CRDs (Custom Resource Definitions). In Crossplane, there is a similar abstraction called XRD (Composite Resource Definition) for defining external resources. Just as CRDs extend the Kubernetes API Server to create custom resource types beyond the standard ones (like Pods, Services, etc.), XRDs are specific to resources managed by external infrastructure providers (like AWS, Azure, etc.).

XRDs allow you to define custom resources that represent specific services from these cloud providers. Once registered, these resources are treated by Crossplane as native Kubernetes resources, enabling you to extend Kubernetes functionality in a customized and uniform manner while managing these resources alongside other Kubernetes resources.

If this still seems confusing, think of XRDs as defining the types of infrastructure resources that Crossplane can manage, while compositions describe how these resources should be configured and provisioned.

Look at the file subnet.xrd.yaml:

12345678910111213141516171819202122232425262728

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: compositesubnets.eks.example.org
spec:
  group: eks.example.org
  names:
    kind: CompositeSubnet
    plural: compositesubnets
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                parameters:
                  type: object
                  properties:
                    cidrBlock:
                      type: string
                    region:
                      type: string
                      default: us-west-2

The kind treats this file as a CompositeResourceDefinition. The name of the resource is CompositeSubnet, and it can be accessed and referenced by other parts of the system. The configurations within it allow for defining things like the CIDR block (which is a way to specify IP ranges) and the default region (which here is us-west-2). The openAPIV3Schema refers to the part of the document that describes the validation schema for the resource defined by the CompositeResourceDefinition. Specifically, it defines the structure and properties that an object of this resource must have to be considered valid. This includes details such as the expected data types for the cidrBlock and region properties, as well as providing a default value for region (in this case, us-west-2).

Finally, we have the Claim. A Claim is essentially a declaration of intent by the user regarding what they want to provision, in our case, a VPC with a subnet. In this sense, compositions describe how resources should be provisioned and configured based on user claims. Compositions are used to translate claims into concrete implementations of infrastructure resources, following the specifications defined in the XRDs. Thus, claims are the expression of user needs, XRDs define the available resource types, and compositions are the mechanisms that allow these claims to be transformed into operational infrastructure resources.

Let's look at the file subnet.claim.yaml:

123456789

apiVersion: eks.example.org/v1alpha1
kind: CompositeSubnet
metadata:
  name: my-subnet-claim
spec:
  parameters:
    cidrBlock: "10.0.1.0/24"
    region: us-west-2

Note that we are passing the parameters that will be used in the composition.

Time to create our resource:

First, let's create our XRD:

bash

kubectl apply -f subnet.xrd.yaml

After that, let's create our composition:

bash

kubectl apply -f subnet.composition.yaml

And finally, let's make our claim by creating the claim:

bash

kubectl apply -f subnet.claim.yaml

Run:

bash

kubectl get compositesubnets my-subnet-claim

You should get a response similar to this:

bash

NAME              SYNCED   READY   COMPOSITION          AGEmy-subnet-claim   True     True    subnet-composition   55s

Still don't believe it, do you? Open your AWS console and see your VPC and subnet created with their configurations.

Gitops

Essentially, Crossplane is a Developer Experience (DX) tool and supports GitOps. Don't know what GitOps is? Read my latest article about it

In this article, we will understand how we can create claims on GitHub and have ArgoCD observe this repository and apply these claims to our cluster, thus creating these resources in our external infrastructure.

Run:

Snippet code=kubectl create namespace argocd title="shell" />

After that we can apply your installation manifest:

shell

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Execute:

shell

kubectl get all -n argocd

If everything went well, you should be able to visualize all the resources provided by ArgoCD.

Next, let's change the service of ArgoCD to type NodePort. This way, the service should be available on all nodes of the Kubernetes cluster on the specified port. You can access it using the IP of any node in the cluster and the specified port.

shell

kubectl patch svc argocd-server -n argocd --type='json' -p  '[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"replace","path":"/spec/ports/0/nodePort","value":30000}]'

You can find the IP of any node using the following command:

shell

kubectl get nodes -o wide

Suppose the IP of one of the nodes is 192.168.1.100. You can access your service using http://192.168.1.100:30000.

After accessing the ArgoCD interface, you'll need to retrieve the administrator password. You can do this using the following command:

shell

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Remember, you should use the username admin and the retrieved password.

12345678

apiVersion: eks.example.org/v1alpha1
kind: CompositeSubnet
metadata:
  name: my-subnet-claim
spec:
  parameters:
    cidrBlock: "10.0.1.0/24"
    region: us-west-2

Now let's apply a manifest to our ArgoCD so that it watches our repository:

argocd.yaml

apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata:  name: network  namespace: argocdspec:  project: default  source:    repoURL: https://github.com/un4uthorized/crossplane-gitops.git # Change this    targetRevision: HEAD    path: .  destination:    server: https://kubernetes.default.svc    namespace: default  syncPolicy:    automated:      prune: true      selfHeal: true    syncOptions:      - SyncWaveOrder=true    retry:      limit: 1      backoff:        duration: 5s        factor: 2        maxDuration: 1m

After applying this manifest, ArgoCD will create your application and apply your claim to the cluster, creating the network resources we defined earlier.

Conclusion

This is just a brief article highlighting the great potential of Crossplane. The focus here isn't on comparing it with other tools like Terraform; I believe each tool has its own purposes, values, and areas of improvement. Based on this, you can explore the technology that best fits your scenario and apply it effectively. I hope this article helps you navigate a bit more about this tool.

Time is a precious commodity, and I appreciate you generously sharing a portion of yours with me.

For more thoughts like thisJoin the community