Kubernetes Cluster API Provider IBM Cloud

Kubernetes-native declarative infrastructure for IBM Cloud.

What is the Cluster API Provider IBM Cloud

The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management.

The API itself is shared across multiple cloud providers allowing for true IBM Cloud hybrid deployments of Kubernetes. It is built atop the lessons learned from previous cluster managers such as kops and kubicorn.

CAPIBM Supported Infrastructure-as-a-Service (IaaS)

Currently, the CAPIBM project exclusively facilitates the deployment of Kubernetes (K8s) clusters solely on two IBM infrastructure offerings, namely IBM VPC (Virtual Server Instances) and IBM PowerVS.

Quick Start

Check out the getting started section to create your first Kubernetes cluster on IBM Cloud using Cluster API.

Tilt-based development environment

See developer guide section for details.

Documentation

Please see our Book for in-depth user documentation.

Additional docs can be found in the /docs directory, and the index is here.

Getting involved and contributing

Are you interested in contributing to cluster-api-provider-ibmcloud? We, the maintainers and community, would love your suggestions, contributions, and help! Also, the maintainers can be contacted at any time to learn more about how to get involved.

In the interest of getting more new people involved, we tag issues with good first issue. These are typically issues that have smaller scope but are good ways to start to get acquainted with the codebase.

We also encourage all active community participants to act as if they are maintainers, even if you don’t have “official” write permissions. This is a community effort, we are here to serve the Kubernetes community. If you have an active interest and you want to get involved, you have real power! Don’t assume that the only people who can get things done around here are the “maintainers”.

We also would love to add more “official” maintainers, so show us what you can do!

This repository uses the Kubernetes bots. See a full list of the commands here.

Join us

The community holds bi-weekly meetings on Thursday at 14:00 IST (Convert to your timezone) on Zoom.

Previous meeting notes, Subscribe to the SIG Cluster Lifecycle Google Group for access to documents and calendars.

Other ways to communicate with the contributors

Please check in with us in the #cluster-api-ibmcloud channel on Slack.

Github issues

Bugs

If you think you have found a bug please follow the instructions below.

• Please spend a small amount of time giving due diligence to the issue tracker. Your issue might be a duplicate.
• Get the logs from the cluster controllers. Please paste this into your issue.
• Open a bug report.
• Remember users might be searching for your issue in the future, so please give it a meaningful title to helps others.

Tracking new features

We also use the issue tracker to track features. If you have an idea for a feature, or think you can help Cluster API Provider IBMCloud become even more awesome, then follow the steps below.

• Open a feature request.
• Remember users might be searching for your issue in the future, so please give it a meaningful title to helps others.
• Clearly define the use case, using concrete examples. EG: I type this and cluster-api-provider-ibmcloud does that.
• Some of our larger features will require some design. If you would like to include a technical design for your feature please include it in the issue.
• After the new feature is well understood, and the design agreed upon we can start coding the feature. We would love for you to code it. So please open up a WIP (work in progress) pull request, and happy coding.

Getting Started

For prerequisites, check the respective sections for VPC and PowerVS

Now that we’ve got all the prerequisites in place, let’s create a Kubernetes cluster and transform it into a management cluster using clusterctl.

Provision local bootstrap management cluster:

1. Create simple, local bootstrap cluster with a control-plane and worker node

   Using kind:

   ~ kind create cluster --name my-bootstrap --config bootstrap.yaml
   

   Example bootstrap.yaml:

   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   nodes:
      - role: control-plane
      - role: worker
   

   Make sure the nodes are in Ready state before moving on.

   ~ kubectl get nodes
   NAME                         STATUS   ROLES           AGE   VERSION
   my-bootstrap-control-plane   Ready    control-plane   46h   v1.34.7
   my-bootstrap-worker          Ready    <none>          46h   v1.34.7
   

2. Set workload cluster environment variables

   Make sure these value reflects your API Key for your target VPC environment or PowerVS environment in IBM Cloud.

   export IBMCLOUD_API_KEY=<YOUR_API_KEY>
   

3. To deploy workload cluster with Custom Service Endpoint, Set SERVICE_ENDPOINT environmental variable in semi-colon separated format:

   ${ServiceRegion1}:${ServiceID1}=${URL1},${ServiceID2}=${URL2};${ServiceRegion2}:${ServiceID1}=${URL1...}.

   Supported ServiceIDs include - vpc, powervs, rc, cos, transitgateway

    export SERVICE_ENDPOINT=us-south:vpc=https://us-south-stage01.iaasdev.cloud.ibm.com,powervs=https://dal.power-iaas.test.cloud.ibm.com,rc=https://resource-controller.test.cloud.ibm.com
   

   Note: Refer Regions-Zones Mapping for more information.

4. For enabling debug level logs for the controller, set the LOGLEVEL environment variable(defaults to 0).

   export LOGLEVEL=5
   

5. Initialize local bootstrap cluster as a management cluster

   When executed for the first time, the following command accepts the infrastructure provider as an input to install. clusterctl init automatically adds to the list the cluster-api core provider, and if unspecified, it also adds the kubeadm bootstrap and kubeadm control-plane providers, thereby converting it into a management cluster which will be used to provision a workload cluster in IBM Cloud.

   ~ clusterctl init --infrastructure ibmcloud:<TAG>
   

   Note: If the latest release version of the provider is available, specifying TAG can be avoided. In other cases, you can specify any prerelease version compatible with the supported API contract as the TAG.
   Example: clusterctl init –infrastructure ibmcloud:v0.2.0-alpha.5

   Output:

   Fetching providers
   Installing cert-manager Version="v1.5.3"
   Waiting for cert-manager to be available...
   Installing Provider="cluster-api" Version="v0.4.4" TargetNamespace="capi-system"
   Installing Provider="bootstrap-kubeadm" Version="v0.4.4" TargetNamespace="capi-kubeadm-bootstrap-system"
   Installing Provider="control-plane-kubeadm" Version="v0.4.4" TargetNamespace="capi-kubeadm-control-plane-system"
   Installing Provider="infrastructure-ibmcloud" Version="v0.1.0-alpha.2" TargetNamespace="capi-ibmcloud-system"
   
   Your management cluster has been initialized successfully!
   
   You can now create your first workload cluster by running the following:
   
   clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -
   

6. Once the management cluster is ready with the required providers up and running, proceed to provisioning the workload cluster. Check the respective sections for VPC and PowerVS to deploy the cluster.

IBM Cloud Machine Images for CAPIBM Clusters

CAPIBM requires a “machine image” containing pre-installed, matching versions of kubeadm and kubelet. Machine image is required during the cluster creation in the IBMVPCMachineTemplate and IBMPowerVSMachineTemplate spec.

Pre-built public Images are published by the maintainers regularly for each new Kubernetes version.

Note: These images are only for the test purpose

• IBM Cloud VPC Images
• IBM Cloud PowerVS Images

VPC Images

Note: These images are built using the image-builder tool and more information can be found here

PowerVS Images

CAPIBM requires a machine boot image to be available in your PowerVS workspace. Two image types are available depending on your cluster topology:

Images are built using the image-builder tool. See How to build machine boot images for details.

Standard images

To import a standard image manually into your workspace, see capibmadm powervs image import.

DHCP-enabled images

These images include a built-in DHCP client configuration required by the LoadBalancer topology, where the network is provisioned by CAPIBM with a DHCP server.

Set COS_BUCKET_NAME, COS_BUCKET_REGION, and COS_OBJECT_NAME to the values from this table when using --flavor=powervs-create-infra.

Topics

This section contains information about using IBM Cloud features with Cluster API Provider IBM Cloud.

• IBM Cloud VPC Cluster
• IBM Cloud PowerVS Cluster

VPC Cluster

Contents

• Prerequisites
• Uploading an image
• Creating a cluster

Prerequisites

1. Install kubectl (see here). Because kustomize was included into kubectl and it’s used by cluster-api-provider-ibmcloud in generating yaml files, so version 1.14.0+ of kubectl is required, see integrate kustomize into kubectl for more info.
2. You can use either VM, container or existing Kubernetes cluster act as the bootstrap cluster.
   • If you want to use container, install kind. This is preferred.
   • If you want to use VM, install minikube, version 0.30.0 or greater.
   • If you want to use existing Kubernetes cluster, prepare your kubeconfig.
3. Install a driver if you are using minikube. For Linux, we recommend kvm2. For MacOS, we recommend VirtualBox.
4. An appropriately configured Go development environment
5. Install clusterctl tool (see here)

Build workload cluster image:

1. Build a qcow2 image suitable for use as a Kubernetes cluster machine as detailed in the image builder book.

   Note: Rename the output image to add the .qcow2 extension. This is required by the next step.

   For more information about the images can be found at machine-images section

2. Upload the VPC Gen2 custom image to IBM Cloud following this section or the detailed explainations in the VPC documentation.

Uploading an image to the IBM Cloud

Build the Ubuntu image as described in the previous VPC section. Make sure to build the qcow2 version by following the instructions for ibmcloud vpc image build.

Since the IBM Cloud does not support dots before the qcow2 extension, rename the file as follows:

ubuntu-2004-ibmcloud-kube-v1-23-4.qcow2

Upload VM image:

1. Create an IBM COS instance
2. Create a bucket in the COS instance.
3. Upload the image
   1. Upload via aspera
      • Install the browser extension for Aspera
      • Downloading the Aspera tool
      • Selecting the image via Aspera dialog
      • Upload the image via aspera
   2. Using minio cli
      • Install minio cli
      • Creating a service credential with hmac=true for the bucket
      • Example upload for eu-de:

        mc alias set uploadcos https://s3.eu-de.cloud-object-storage.appdomain.cloud <hmac access id> <hmac secret key>
        

        mc cp <image-name>.qcow2 uploadcos/<my-bucket-name>
        

Add VM image to VPC

1. Make sure you have editor rights for all/most VPC services
2. Add additional read rights for:

src: service VPC Infrastructure Services resourceType equals image
target: serviceInstance string equals <your-Cloud-Object Storage-VM-plain-name>

Add write rights for:

Service VPC Infrastructure Services in Resource_group <your_resource_group_or_account> resourceType equals image
target: service Cloud object storage in resource_group <your_resource_group_or_account>

3. Go to https://cloud.ibm.com/vpc-ext/provision/customImage

• Fill in imagename, resource group or account
• Choice box: Cloud Object Storage
• Set Filter: <your_cos_plain_name> <eu-de_or_other> <your_vm_bucket>
• Choice box: Select your image
• Select base os (ubuntu-20-04-amd64 for example)
• Click Create Image

Now you can provision a VM with your own VM image. Then please continue with creating a cluster.

Make sure you take the ImageID from your VM image. The ImageID can be determined using ibmcloud cli. In addition, the Kubernetes version must be set to match the image. In this example:

v1.23.4

Provision workload Cluster in IBM Cloud VPC

Now that we have a management cluster ready, you can create your workload cluster by following the steps below.

Note:

1. The cluster will be deployed with cloud controller manager
2. To deploy a cluster using ClusterClass, refer here. In addition to the above flags, set CLUSTER_TOPOLOGY environment variable to true.
3. The list of IBM Cloud VPC Regions and Zones can be found here.

1. Using clusterctl, render the yaml through templates and deploy the cluster

   Note:

   • The IBMVPC_IMAGE_NAME value below should reflect the name of the custom qcow2 image.
   • While working with unreleased versions like from main branch, instead of --flavor=vpc-clusterclass use --from ./templates/cluster-template-vpc-clusterclass.yaml.

   IBMCLOUD_API_KEY="XXXXXXXXXXXXXXXXXX" \
   IBMVPC_REGION=us-south \
   IBMVPC_ZONE=us-south-1 \
   IBMVPC_RESOURCEGROUP=4f15679623607b855b1a27a67f20e1c7 \
   IBMVPC_NAME=ibm-vpc-0 \
   IBMVPC_IMAGE_NAME=capibm-vpc-ubuntu-2404-kube-v1-34-2 \
   IBMVPC_PROFILE=bx2-4x16 \
   IBMVPC_SSHKEY_NAME=capi-vpc-key \
   IBMACCOUNT_ID="ibm-accountid" \
   BASE64_API_KEY=$(echo -n $IBMCLOUD_API_KEY | base64) \
   clusterctl generate cluster ibm-vpc-0 --kubernetes-version v1.34.2 \
   --target-namespace default \
   --control-plane-machine-count=1 \
   --worker-machine-count=2 | kubectl apply -f -
   

   Output:

   cluster.cluster.x-k8s.io/ibm-vpc-0 created
   ibmvpccluster.infrastructure.cluster.x-k8s.io/ibm-vpc-0 created
   kubeadmcontrolplane.controlplane.cluster.x-k8s.io/ibm-vpc-0-control-plane created
   ibmvpcmachinetemplate.infrastructure.cluster.x-k8s.io/ibm-vpc-0-control-plane created
   machinedeployment.cluster.x-k8s.io/ibm-vpc-0-md-0 created
   ibmvpcmachinetemplate.infrastructure.cluster.x-k8s.io/ibm-vpc-0-md-0 created
   kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/ibm-vpc-0-md-0 created
   clusterresourceset.addons.cluster.x-k8s.io/crs-cloud-conf created
   configmap/ibm-cfg created
   secret/ibm-credential created
   configmap/cloud-controller-manager-addon created
   

   Note: Refer below for more detailed information on VPC variables.

   • IBMVPC_REGION
   • IBMVPC_ZONE
   • IBMVPC_RESOURCEGROUP
   • IBMVPC_IMAGE_NAME
   • IBMVPC_PROFILE
   • IBMVPC_SSHKEY_NAME
   • IBMACCOUNT_ID

2. Check the state of the provisioned cluster and machine objects within the local management cluster

   Clusters

   ~ kubectl get clusters
   NAME         PHASE
   ibm-vpc-0    Provisioned
   

   Kubeadm Control Plane

   ~ kubectl get kubeadmcontrolplane
   NAME                       INITIALIZED   API SERVER AVAILABLE   VERSION   REPLICAS   READY   UPDATED   UNAVAILABLE
   ibm-vpc-0-control-plane    true          true                   v1.34.2   1          1       1
   

   Machines

   ~ kubectl get machines
   ibm-vpc-0-control-plane-vzz47     ibmvpc://ibm-vpc-0/ibm-vpc-0-control-plane-rg6xv   Running        v1.34.2
   ibm-vpc-0-md-0-5444cfcbcd-6gg5z   ibmvpc://ibm-vpc-0/ibm-vpc-0-md-0-dbxb7            Running        v1.34.2
   ibm-vpc-0-md-0-5444cfcbcd-7kr9x   ibmvpc://ibm-vpc-0/ibm-vpc-0-md-0-k7blr            Running        v1.34.2
   

3. Deploy Container Network Interface (CNI)

   Example: calico

   ~ clusterctl get kubeconfig ibm-vpc-0 > ~/.kube/ibm-vpc-0
   ~ export KUBECONFIG=~/.kube/ibm-vpc-0
   ~ kubectl apply -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml
   

4. Check the state of the newly provisioned cluster within IBM Cloud

   ~ kubectl get nodes
   NAME                             STATUS   ROLES    AGE   VERSION
   ibm-vpc-0-control-plane-rg6xv    Ready    master   41h   v1.34.2
   ibm-vpc-0-md-0-4dc5c             Ready    <none>   41h   v1.34.2
   ibm-vpc-0-md-0-dbxb7             Ready    <none>   20h   v1.34.2
   

Change disk size for the boot volume

There are two following variables for controlling the volume size for the boot disk.

• IBMVPC_CONTROLPLANE_BOOT_VOLUME_SIZEGIB: Size of the boot volume for the control plane nodes, default set to 20GiB
• IBMVPC_WORKER_BOOT_VOLUME_SIZEGIB: Size of the boot volume for the worker nodes, default set to 20GiB

Note: Default value is set to 20GiB because the images published for testing are of size 20GiB(default size in the image-builder scripts as well).

Deploy a VPC cluster using ClusterClass

IBMVPC_CLUSTER_CLASS_NAME=ibmvpc-clusterclass \
IBMCLOUD_API_KEY="XXXXXXXXXXXXXXXXXX" \
IBMVPC_REGION=us-south \
IBMVPC_ZONE=us-south-1 \
IBMVPC_RESOURCEGROUP=4f15679623607b855b1a27a67f20e1c7 \
IBMVPC_NAME=ibm-vpc-0 \
IBMVPC_IMAGE_NAME=capibm-vpc-ubuntu-2404-kube-v1-34-2 \
IBMVPC_PROFILE=bx2-4x16 \
IBMVPC_SSHKEY_NAME=capi-vpc-key \
IBMACCOUNT_ID="ibm-accountid" \
BASE64_API_KEY=$(echo -n $IBMCLOUD_API_KEY | base64) \
clusterctl generate cluster ibm-vpc-clusterclass --kubernetes-version v1.34.2 --target-namespace default --control-plane-machine-count=1 --worker-machine-count=2 --from=./templates/cluster-template-vpc-clusterclass.yaml | kubectl apply -f -

Note: Refer below for more detailed information on VPC variables.

• IBMVPC_CLUSTER_CLASS_NAME : Name of the cluster that user provides.
• IBMVPC_REGION
• IBMVPC_ZONE
• IBMVPC_RESOURCEGROUP : Go to Manage > Account > Account resources > Resource groups in the IBM Cloud console to get the resource group ID.
• IBMVPC_IMAGE_NAME : Name of the custom image. Refer here
• IBMVPC_PROFILE
• IBMVPC_SSHKEY_NAME : Name of the SSH Key. Refer here how to get the keys.
• IBMACCOUNT_ID : Go to the Manage > Account > Account settings page in the IBM Cloud console to view your account ID and type. The account ID is a 32 character, unique account identifier. Refer here

PowerVS Cluster

IBM Power Systems Virtual Server (PowerVS) is IBM’s cloud offering for running workloads on IBM Power hardware. CAPIBM supports provisioning Kubernetes clusters on PowerVS.

Overview

A PowerVS cluster can be deployed with two different control-plane topologies. The topology determines how the Kubernetes API server endpoint is made highly available, which infrastructure you are responsible for, and which clusterctl flavor to use.

Choose your topology before reading further — prerequisites and steps differ between them.

Option 1 — VirtualIP

Control-plane HA via kube-vip running on the PowerVS network.

You pre-create all PowerVS infrastructure (workspace, public network, port/VIP) and import the machine boot image. CAPIBM only provisions the virtual machines; it references your existing resources.

• Template flavor: --flavor=powervs
• ClusterClass variant: --flavor=powervs-clusterclass
• Image type required: standard PowerVS image

Option 2 — LoadBalancer

Control-plane HA via a VPC Load Balancer.

CAPIBM provisions the complete infrastructure stack: PowerVS workspace, VPC, transit gateway, VPC subnets, load balancer, and virtual machines. The machine boot image is imported automatically from a COS bucket you specify.

• Template flavor: --flavor=powervs-create-infra
• Image type required: DHCP-enabled PowerVS image

Note: All provisioned resource names default to <cluster-name>-<resource>. You can override individual names by setting the corresponding environment variables. If a resource with the given name already exists in your account, the controller adopts it instead of creating a new one.

SEE ALSO

• Prerequisites
• Creating a cluster
• Using autoscaler with scaling from 0 machines

Prerequisites

Common prerequisites

These are required for both topologies.

1. Install kubectl v1.14.0 or later — see install guide.
2. A bootstrap cluster (pick one):
   • kind (preferred) — see kind installation
   • minikube v0.30.0+ — see minikube installation. Requires a driver; kvm2 on Linux, VirtualBox on macOS.
   • An existing Kubernetes cluster with a valid kubeconfig.
3. Install clusterctl — see install guide.
4. Install capibmadm — see install guide.
5. An IBM Cloud account (paid, for PowerVS).
6. An IBM Cloud API key.

Option 1 — VirtualIP: additional prerequisites

The VirtualIP topology requires you to create a PowerVS workspace, network, and a reserved port (VIP) before running clusterctl. Follow the steps below.

1. Create a PowerVS workspace

Create a Power Systems Virtual Server workspace in IBM Cloud. Note the Workspace ID — you will need it as IBMPOWERVS_WORKSPACE_ID.

See IBM Cloud docs for how to create and list workspaces.

2. Create a public network

export IBMCLOUD_API_KEY=<API_KEY>
capibmadm powervs network create --name capi-test \
  --service-instance-id <WORKSPACE_ID> \
  --zone <ZONE>

Output:

Creating PowerVS network service-instance-id="3229a94c-af54-4212-bf60-6202b6fd0a07" zone="osa21"
Successfully created a network networkID="3ee5a1ca-19b4-48c7-a89d-44babdd18703"

The network name (capi-test above) becomes your IBMPOWERVS_NETWORK_NAME.

3. Reserve a port (VIP)

capibmadm powervs port create \
  --network capi-test \
  --description capi-test-port \
  --service-instance-id <WORKSPACE_ID> \
  --zone <ZONE>

Then list the port to get the assigned IP addresses:

capibmadm powervs port list \
  --network capi-test \
  --service-instance-id <WORKSPACE_ID> \
  --zone <ZONE>

Output:

DESCRIPTION      EXTERNAL IP   IP ADDRESS      MAC ADDRESS         PORT ID                                STATUS
capi-test-port   163.68.65.6   192.168.167.6   fa:16:3e:89:c8:80   c7e7b6e0-0b0d-4a11-a90b-6ea293deb5ac   DOWN

From this output:

• IP ADDRESS (192.168.167.6) → IBMPOWERVS_VIP
• EXTERNAL IP (163.68.65.6) → IBMPOWERVS_VIP_EXTERNAL

The CIDR prefix length of the network (IBMPOWERVS_VIP_CIDR) can be found from the network details, e.g. 29 for a /29 subnet.

4. Import the machine boot image

Use a standard (non-DHCP) image. See PowerVS Images for the list of available images and COS bucket details.

capibmadm powervs image import \
  --service-instance-id <WORKSPACE_ID> \
  --zone <ZONE> \
  --bucket-region <BUCKET_REGION> \
  --object <OBJECT_NAME> \
  --name <IMAGE_NAME> \
  --bucket power-oss-bucket \
  --public-bucket

Example:

capibmadm powervs image import \
  --service-instance-id 3229a94c-af54-4212-bf60-6202b6fd0a07 \
  --zone osa21 \
  --bucket-region us-south \
  --object capibm-powervs-centos-streams10-1-34-7.ova.gz \
  --name capibm-powervs-centos-streams10-1-34-7 \
  --bucket power-oss-bucket \
  --public-bucket

The image name you choose becomes your IBMPOWERVS_IMAGE_NAME.

5. Upload an SSH key

capibmadm powervs key create \
  --name my-pub-key \
  --key-path ~/.ssh/id_rsa.pub \
  --service-instance-id <WORKSPACE_ID> \
  --zone <ZONE>

The key name becomes your IBMPOWERVS_SSHKEY_NAME.

Option 2 — LoadBalancer: additional prerequisites

The LoadBalancer topology only needs an SSH key and a COS bucket containing the DHCP-enabled machine boot image. CAPIBM creates the workspace and all other infrastructure automatically.

1. Identify your resource group

Go to Manage → Account → Account resources → Resource groups in the IBM Cloud console to get your resource group name. This becomes IBM_RESOURCE_GROUP.

2. Ensure a DHCP-enabled image is available in COS

Use a DHCP-enabled image. See PowerVS Images with DHCP based network for the list of available images and COS bucket details.

The COS details you need:

• COS_BUCKET_NAME — e.g. power-oss-bucket
• COS_BUCKET_REGION — e.g. us-south
• COS_OBJECT_NAME — e.g. capibm-powervs-centos-streams10-1-34-7-dhcp.ova.gz

CAPIBM creates an IBMPowerVSImage resource that imports this image from COS into the newly provisioned workspace automatically during cluster creation. You do not need to import it manually.

3. Upload an SSH key to IBM Cloud

If you do not already have an SSH key registered in IBM Cloud, you can add one via the IBM Cloud console. The key name becomes IBMPOWERVS_SSHKEY_NAME.

Creating a PowerVS cluster

Before running any of the commands below, make sure your management cluster is initialised:

clusterctl init --infrastructure ibmcloud

See Getting Started for full management cluster setup.

Option 1 — VirtualIP topology

Uses kube-vip on the PowerVS network for control-plane HA. Requires a pre-existing workspace, network, and reserved VIP port — see VirtualIP prerequisites.

Working from main branch? Replace --flavor=powervs with --from=./templates/cluster-template-powervs.yaml.

Required variables

Deploy the cluster

IBMPOWERVS_SSHKEY_NAME="my-pub-key" \
IBMPOWERVS_VIP="192.168.167.6" \
IBMPOWERVS_VIP_EXTERNAL="163.68.65.6" \
IBMPOWERVS_VIP_CIDR="29" \
IBMPOWERVS_IMAGE_NAME="capibm-powervs-centos-streams10-1-34-7" \
IBMPOWERVS_WORKSPACE_ID="3229a94c-af54-4212-bf60-6202b6fd0a07" \
IBMPOWERVS_NETWORK_NAME="capi-test" \
IBMACCOUNT_ID="ibm-accountid" \
IBMPOWERVS_REGION="osa" \
IBMPOWERVS_ZONE="osa21" \
BASE64_API_KEY=$(echo -n $IBMCLOUD_API_KEY | base64) \
clusterctl generate cluster ibm-powervs-1 \
  --kubernetes-version v1.34.7 \
  --target-namespace default \
  --control-plane-machine-count=3 \
  --worker-machine-count=1 \
  --flavor=powervs | kubectl apply -f -

Optional variables

Control-plane machine sizing:

IBMPOWERVS_CONTROL_PLANE_MEMORY      (default: 4 GiB)
IBMPOWERVS_CONTROL_PLANE_PROCESSORS  (default: 0.25)
IBMPOWERVS_CONTROL_PLANE_SYSTYPE     (default: s1022)
IBMPOWERVS_CONTROL_PLANE_PROCTYPE    (default: Shared)

Worker machine sizing:

IBMPOWERVS_COMPUTE_MEMORY      (default: 4 GiB)
IBMPOWERVS_COMPUTE_PROCESSORS  (default: 0.25)
IBMPOWERVS_COMPUTE_SYSTYPE     (default: s1022)
IBMPOWERVS_COMPUTE_PROCTYPE    (default: Shared)

API server port:

API_SERVER_PORT  (default: 6443)

ClusterClass variant

To use the ClusterClass approach, set CLUSTER_TOPOLOGY=true and use --flavor=powervs-clusterclass. All the same variables apply, with one addition:

CLUSTER_TOPOLOGY=true \
IBMPOWERVS_CLUSTER_CLASS_NAME="powervs-cc" \
IBMPOWERVS_SSHKEY_NAME="my-pub-key" \
IBMPOWERVS_VIP="192.168.167.6" \
IBMPOWERVS_VIP_EXTERNAL="163.68.65.6" \
IBMPOWERVS_VIP_CIDR="29" \
IBMPOWERVS_IMAGE_NAME="capibm-powervs-centos-streams10-1-34-7" \
IBMPOWERVS_WORKSPACE_ID="3229a94c-af54-4212-bf60-6202b6fd0a07" \
IBMPOWERVS_NETWORK_NAME="capi-test" \
IBMACCOUNT_ID="ibm-accountid" \
IBMPOWERVS_REGION="osa" \
IBMPOWERVS_ZONE="osa21" \
BASE64_API_KEY=$(echo -n $IBMCLOUD_API_KEY | base64) \
clusterctl generate cluster ibm-powervs-1 \
  --kubernetes-version v1.34.7 \
  --target-namespace default \
  --control-plane-machine-count=3 \
  --worker-machine-count=1 \
  --flavor=powervs-clusterclass | kubectl apply -f -

Working from main branch? Replace --flavor=powervs-clusterclass with --from=./templates/cluster-template-powervs-clusterclass.yaml.

Option 2 — LoadBalancer topology

CAPIBM provisions the full infrastructure stack and uses a VPC Load Balancer for control-plane HA. Requires a COS bucket with a DHCP-enabled image — see LoadBalancer prerequisites.

Working from main branch? Replace --flavor=powervs-create-infra with --from=./templates/cluster-template-powervs-create-infra.yaml.

Required variables

Deploy the cluster

IBMCLOUD_API_KEY=<API_KEY> \
IBMPOWERVS_SSHKEY_NAME="my-ssh-key" \
COS_BUCKET_REGION="us-south" \
COS_BUCKET_NAME="power-oss-bucket" \
COS_OBJECT_NAME="capibm-powervs-centos-streams10-1-34-7-dhcp.ova.gz" \
IBMACCOUNT_ID="<account_id>" \
IBMPOWERVS_REGION="wdc" \
IBMPOWERVS_ZONE="wdc06" \
IBMVPC_REGION="us-east" \
IBM_RESOURCE_GROUP="ibm-resource-group" \
BASE64_API_KEY=$(echo -n $IBMCLOUD_API_KEY | base64) \
clusterctl generate cluster capi-powervs \
  --kubernetes-version v1.34.7 \
  --target-namespace default \
  --control-plane-machine-count=3 \
  --worker-machine-count=1 \
  --flavor=powervs-create-infra | kubectl apply -f -

All infrastructure resources are named <cluster-name>-<resource> by default (e.g. capi-powervs-workspace, capi-powervs-vpc). To override any name, set the corresponding variable before running the command.

Verify the cluster

Once applied, check the status on the management cluster:

Clusters

kubectl get clusters
NAME            PHASE
ibm-powervs-1   Provisioned

Control plane

kubectl get kubeadmcontrolplane
NAME                          INITIALIZED   API SERVER AVAILABLE   VERSION   REPLICAS   READY   UPDATED   UNAVAILABLE
ibm-powervs-1-control-plane   true          true                   v1.34.7   3          3       3

Machines

kubectl get machines
NAME                                   PROVIDERID                                                         PHASE     VERSION
ibm-powervs-1-control-plane-vzz47      ibmpowervs://ibm-powervs-1/ibm-powervs-1-control-plane-rg6xv      Running   v1.34.7
ibm-powervs-1-md-0-5444cfcbcd-6gg5z   ibmpowervs://ibm-powervs-1/ibm-powervs-1-md-0-dbxb7               Running   v1.34.7
ibm-powervs-1-md-0-5444cfcbcd-7kr9x   ibmpowervs://ibm-powervs-1/ibm-powervs-1-md-0-k7blr               Running   v1.34.7

Deploy a CNI

Retrieve the workload cluster kubeconfig and apply a CNI. Example using Calico:

clusterctl get kubeconfig ibm-powervs-1 > ~/.kube/ibm-powervs-1
export KUBECONFIG=~/.kube/ibm-powervs-1
kubectl apply -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml

Verify workload cluster nodes

kubectl get nodes
NAME                                STATUS   ROLES           AGE   VERSION
ibm-powervs-1-control-plane-rg6xv   Ready    control-plane   41h   v1.34.7
ibm-powervs-1-md-0-4dc5c            Ready    <none>          41h   v1.34.7
ibm-powervs-1-md-0-dbxb7            Ready    <none>          20h   v1.34.7

Using Autoscaler to scale from 0 machines

The cluster-autoscaler project supports Cluster API. With the scale-from-zero enhancement, worker nodes can be scaled down to 0 and provisioned on demand.

Setting up the workload cluster

Add the following annotations to your MachineDeployment to opt in to autoscaling. These are required by the autoscaler to know the min/max bounds when scaling from 0.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: "${CLUSTER_NAME}-md-0"
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"

Setting up the cluster-autoscaler

1. Clone the autoscaler repository:

   git clone https://github.com/kubernetes/autoscaler.git
   

2. Build the autoscaler binary:

   cd autoscaler/cluster-autoscaler
   go build .
   

3. Start the autoscaler:

   ./cluster-autoscaler \
     --cloud-provider=clusterapi \
     --v=2 \
     --namespace=default \
     --max-nodes-total=30 \
     --scale-down-delay-after-add=10s \
     --scale-down-delay-after-delete=10s \
     --scale-down-delay-after-failure=10s \
     --scale-down-unneeded-time=5m \
     --max-node-provision-time=30m \
     --balance-similar-node-groups \
     --expander=random \
     --kubeconfig=<workload_cluster_kubeconfig> \
     --cloud-config=<management_cluster_kubeconfig>
   

Note: The autoscaler can be run in several ways — see connecting to management and workload clusters for alternatives. A full list of command-line flags is available in the FAQ.

Walkthrough: scale up and scale down

1. Create a workload cluster with 0 worker machines.

2. Apply a sample workload:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: busybox-deployment
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: busybox
     template:
       metadata:
         labels:
           app: busybox
       spec:
         containers:
           - name: busybox
             image: busybox
             imagePullPolicy: IfNotPresent
             command: ["sh", "-c", "echo Running; sleep 3600"]
             resources:
               requests:
                 cpu: "0.2"
                 memory: 3G
   

3. Scale the deployment to trigger pending pods:

   kubectl scale --replicas=2 deployment/busybox-deployment
   

4. Observe that the second pod is pending (no nodes available yet):

   kubectl get pods
   NAME                                  READY   STATUS    RESTARTS   AGE
   busybox-deployment-7c87788568-qhqdb   1/1     Running   0          48s
   busybox-deployment-7c87788568-t26bb   0/1     Pending   0          5s
   

5. On the management cluster, watch the autoscaler provision a new machine:

   kubectl get machines
   NAME                                  CLUSTER         PHASE          VERSION
   ibm-powervs-control-plane-smvf7       ibm-powervs     Running        v1.34.7
   ibm-powervs-md-0-6b4d67ccf4-npdbm    ibm-powervs     Running        v1.34.7
   ibm-powervs-md-0-6b4d67ccf4-v7xv9    ibm-powervs     Provisioning   v1.34.7
   

6. Once the new node joins, both pods should be running:

   kubectl get nodes
   NAME                               STATUS   ROLES           AGE   VERSION
   ibm-powervs-control-plane-pgwmz   Ready    control-plane   92m   v1.34.7
   ibm-powervs-md-0-n8c6d            Ready    <none>          42s   v1.34.7
   ibm-powervs-md-0-qch8f            Ready    <none>          85m   v1.34.7
   
   kubectl get pods
   NAME                                  READY   STATUS    RESTARTS   AGE
   busybox-deployment-7c87788568-qhqdb   1/1     Running   0          19m
   busybox-deployment-7c87788568-t26bb   1/1     Running   0          18m
   

7. Delete the deployment and observe the autoscaler scale the node back down:

   kubectl delete deployment/busybox-deployment
   
   kubectl get nodes
   NAME                               STATUS   ROLES           AGE    VERSION
   ibm-powervs-control-plane-pgwmz   Ready    control-plane   105m   v1.34.7
   ibm-powervs-md-0-qch8f            Ready    <none>          98m    v1.34.7
   

capibmadm CLI

Kubernetes Cluster API Provider IBM Cloud Management Utility

Install capibmadm

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/releases/download/v0.12.2/capibmadm-linux-amd64 -o capibmadm

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/releases/download/v0.12.2/capibmadm-linux-arm64 -o capibmadm

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/releases/download/v0.12.2/capibmadm-linux-ppc64le -o capibmadm

chmod +x ./capibmadm

sudo mv ./capibmadm /usr/local/bin/capibmadm

capibmadm version -o short

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/releases/download/v0.12.2/capibmadm-darwin-amd64 -o capibmadm

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/releases/download/v0.12.2/capibmadm-darwin-arm64 -o capibmadm

chmod +x ./capibmadm

sudo mv ./capibmadm /usr/local/bin/capibmadm

capibmadm version -o short

curl.exe -L https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/releases/download/v0.12.2/capibmadm-windows-amd64.exe -o capibmadm.exe

curl.exe -L https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/releases/download/v0.12.2/capibmadm-windows-amd64.exe -o capibmadm.exe

capibmadm.exe version -o short

1. PowerVS commands

2. VPC commands

capibmadm powervs <commands>

1. PowerVS commands

• key
  • create
  • delete
  • list
• network
  • create
  • delete
  • list
• port
  • create
  • delete
  • list
• image
  • import
  • list

PowerVS Image Commands

1. capibmadm powervs image import

Usage:

Import PowerVS image.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–bucket: Cloud Object Storage bucket name.

–bucket-region: Cloud Object Storage bucket location.

–object: Cloud Object Storage object name.

–accesskey: Cloud Object Storage HMAC access key.

–secretkey: Cloud Object Storage HMAC secret key.

–name: Name to PowerVS imported image.

–public-bucket: Cloud Object Storage public bucket.

–watch-timeout: watch timeout.

–pvs-storagetype: PowerVS Storage type, accepted values are [tier0, tier1, tier3]..

Example:

export IBMCLOUD_API_KEY=<api-key>
# import image using default storage type (service credential will be autogenerated):
capibmadm powervs image import --service-instance-id <service-instance-id> -b <bucketname> --object rhel-83-10032020.ova.gz --name <imagename> -r <region> --zone <zone>

# import image using default storage type with specifying the accesskey and secretkey explicitly:
capibmadm powervs image import --service-instance-id <service-instance-id> -b <bucketname> --object rhel-83-10032020.ova.gz --name <imagename> -r <region> --zone <zone> --accesskey <accesskey> --secretkey <secretkey>

# with user provided storage type:
capibmadm powervs image import --service-instance-id <service-instance-id> -b <bucketname> --pvs-storagetype <storagetype> --object rhel-83-10032020.ova.gz --name <imagename> -r <region> --zone <zone>

#import image from a public IBM Cloud Storage bucket:
capibmadm powervs image import --service-instance-id <service-instance-id> -b <bucketname>  --object rhel-83-10032020.ova.gz --name <imagename> -r <region> --public-bucket --zone <zone> 

2. capibmadm powervs image list

Usage:

List PowerVS images.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–zone: PowerVS workspace zone.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm powervs image list --service-instance-id <service-instance-id> --zone <zone>

PowerVS Network Commands

1. capibmadm powervs network create

Usage:

Create PowerVS network.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–cidr: The network CIDR. Required for private network type.

–name: The name of the network.

–public: Public (pub-vlan) network type (default true)

–private: Private (vlan) network type (default false)

–gateway: The gateway ip address.

–dns-servers: Comma separated list of DNS Servers to use for this network, Defaults to 8.8.8.8, 9.9.9.9.

–ip-ranges: Comma separated IP Address Ranges.

–jumbo: Enable MTU Jumbo Network.

Example:

export IBMCLOUD_API_KEY=<api-key>
# Public network:
capibmadm powervs network create --public --service-instance-id <service-instance-id> --zone <zone>
# Private network:
capibmadm powervs network create --private --cidr <cidr> --service-instance-id <service-instance-id> --zone <zone>
# Private network with ip address ranges:
capibmadm powervs network create --private --cidr <cidr> --ip-ranges <start-ip>-<end-ip>,<start-ip>-<end-ip> --service-instance-id <service-instance-id> --zone <zone>

2. capibmadm powervs network delete

Usage:

Delete PowerVS network.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–zone: PowerVS workspace zone.

–network: Network ID or Name.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm powervs network delete --network <network-name/network-id> --service-instance-id <service-instance-id> --zone <zone>

3. capibmadm powervs network list

Usage:

List PowerVS networks.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–zone: PowerVS workspace zone.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm powervs network list --service-instance-id <service-instance-id> --zone <zone>

PowerVS Port Commands

1. capibmadm powervs port create

Usage:

Create PowerVS network port.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–zone: PowerVS workspace zone.

–network: Network ID/ Network Name.

–description: Description of the port.

–ip-address: The requested IP address of this port

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm powervs port create --network <netword-id/network-name> --description <description> --service-instance-id <service-instance-id> --zone <zone>

2. capibmadm powervs port delete

Usage:

Delete PowerVS network port.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–zone: PowerVS zone.

–port-id: ID of network port.

–network: Network ID or Name.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm powervs port delete --port-id <port-id> --network <network-name/network-id> --service-instance-id <service-instance-id> --zone <zone>

3. capibmadm powervs port list

Usage:

List PowerVS ports.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–zone: PowerVS zone.

–network: Network ID or Name.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm powervs port list --service-instance-id <service-instance-id> --zone <zone> --network <network-name/network-id>

PowerVS SSH key Commands

1. capibmadm powervs key create

Usage:

Create an SSH key in the PowerVS environment.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–zone: PowerVS zone.

–name: The name of the SSH key.

Either of the arguments need to be provided:

–key: SSH RSA key string within a double quotation marks. For example, “ssh-rsa AAA… “.

–key-path: The absolute path to the SSH key file.

Example:

export IBMCLOUD_API_KEY=<api-key>

# Using SSH key
capibmadm powervs key create --name <key-name> --key "<ssh-key>" --service-instance-id <service-instance-id> --zone <zone>

# Using file-path to SSH key
capibmadm powervs key create --name <key-name> --key-path <path/to/ssh/key> --service-instance-id <service-instance-id> --zone <zone>

2. capibmadm powervs key delete

Usage:

Delete an SSH key in the PowerVS environment.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–zone: PowerVS zone.

–name: The name of the SSH key.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm powervs key delete --name <key-name> --service-instance-id <service-instance-id> --zone <zone>

3. capibmadm powervs key list

Usage:

List all SSH Keys in the PowerVS environment.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–service-instance-id: PowerVS workspace ID.

–zone: PowerVS zone.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm powervs key list --service-instance-id <service-instance-id> --zone <zone>

capibmadm vpc <commands>

1. VPC commands

• key

  • list
  • create
  • delete

• image

  • list

VPC image Commands

1. capibmadm vpc image list

Usage:

List images in given VPC region.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–region: VPC region.

–resource-group-name: IBM Cloud resource group name.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm vpc image list --region <region> --resource-group-name <resource-group>

VPC SSH key Commands

1. capibmadm vpc key list

Usage:

List SSH keys in given VPC region.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–region: VPC region.

–resource-group-name: IBM Cloud resource group name.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm vpc key list --region <region> --resource-group-name <resource-group>

2. capibmadm vpc key create

Usage:

Create a key in the VPC environment.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–name: The name of the key.

–resource-group-name: VPC resource group name.

–region: VPC region.

Either of the arguments need to be provided:

–public-key: Public key string within a double quotation marks. For example, “ssh-rsa AAA… “.

–key-path: The absolute path to the SSH key file.

Example:

export IBMCLOUD_API_KEY=<api-key>

capibmadm vpc key create --name <key-name> --region <region> --public-key "<public-key-string>"

capibmadm vpc key create --name <key-name> --region <region> --key-path <path/to/ssh/key>

3. capibmadm vpc key delete

Usage:

Delete a key in the VPC environment.

Environmental Variable:

IBMCLOUD_API_KEY: IBM Cloud API key.

Arguments:

–name: The name of the key.

–region: VPC region.

Example:

export IBMCLOUD_API_KEY=<api-key>
capibmadm vpc key delete --name <key-name> --region <region>

Developer Guide

• Development with Tilt
• Guide for API conversions
• Release Process
• Release Support Guidelines
• How to build the machine boot images
• Modules and tools dependencies
• E2E testing

Development with Tilt

Overview

This guide sets up a local development environment using kind and Tilt that hot-reloads controller changes without having to rebuild and redeploy manually.

The setup involves two repositories:

• cluster-api-provider-ibmcloud — this repository. The make kind-cluster command and controller code live here.
• cluster-api — the upstream CAPI repo. The tilt-settings.yaml file and tilt up command are run from here.

Prerequisites

1. Container runtime — one of:
   • Docker v19.03 or newer
   • Podman v3.0 or newer (see Podman setup below)
2. kind v0.9 or newer
3. kustomize
4. Tilt v0.30.8 or newer
5. envsubst or similar (for clusterctl variable substitution)
6. Both repositories cloned side-by-side:

   git clone https://github.com/kubernetes-sigs/cluster-api.git
   git clone https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud.git
   

Setup steps

1. Create the kind management cluster

Run from the cluster-api-provider-ibmcloud directory:

make kind-cluster

This creates a local kind cluster and sets it as your active KUBECONFIG context. This cluster acts as the CAPI management cluster — it runs the controllers and is used to provision workload clusters on IBM Cloud.

2. Create a tilt-settings.yaml

Create tilt-settings.yaml in your local cluster-api directory. This file tells Tilt where to find the IBM Cloud provider and which credentials to use.

Minimum configuration:

default_registry: "localhost:5001"
provider_repos:
  - ../cluster-api-provider-ibmcloud
enable_providers:
  - ibmcloud
  - kubeadm-bootstrap
  - kubeadm-control-plane
kustomize_substitutions:
  IBMCLOUD_API_KEY: "<YOUR_API_KEY>"

Note: The path ../cluster-api-provider-ibmcloud assumes both repositories are cloned side-by-side in the same parent directory.

3. Start Tilt

Run from the cluster-api directory:

tilt up

Tilt builds and deploys the controllers onto the kind cluster, then watches for source changes and hot-reloads automatically. Verify the controllers are running:

kubectl get pods -A

Optional configurations

All of the following are additions to the base tilt-settings.yaml shown above. Only include the fields relevant to your use case.

Enable verbose API logging

Logs all PowerVS REST API requests and responses:

extra_args:
  ibmcloud:
    - '--v=5'

Enable ClusterClass support

Required when deploying a cluster using --flavor=powervs-clusterclass. See ClusterClass variant.

kustomize_substitutions:
  IBMCLOUD_API_KEY: "<YOUR_API_KEY>"
  CLUSTER_TOPOLOGY: "true"

Use custom service endpoints (staging/test environments)

Set SERVICE_ENDPOINT in semi-colon separated format: ${ServiceRegion}:${ServiceID1}=${URL1},${ServiceID2}=${URL2}:

kustomize_substitutions:
  IBMCLOUD_API_KEY: "<YOUR_API_KEY>"
  SERVICE_ENDPOINT: "us-south:vpc=https://us-south-stage01.iaasdev.cloud.ibm.com,powervs=https://dal.power-iaas.test.cloud.ibm.com,rc=https://resource-controller.test.cloud.ibm.com"
  IBMCLOUD_AUTH_URL: "https://iam.test.cloud.ibm.com"

Enable observability tools

Deploys Prometheus, Grafana, Loki, and Promtail alongside the controllers. See CAPI observability docs for more detail.

deploy_observability:
  - promtail
  - loki
  - grafana
  - prometheus
extra_args:
  core:
    - "--logging-format=json"
    - "--v=5"
  kubeadm-bootstrap:
    - "--v=5"
    - "--logging-format=json"
  kubeadm-control-plane:
    - "--v=5"
    - "--logging-format=json"
  ibmcloud:
    - "--v=5"
    - "--logging-format=json"

For a full list of supported tilt-settings.yaml fields, see the CAPI Tilt settings reference.

Create workload clusters

With Tilt running, provision a workload cluster using clusterctl. See:

• PowerVS cluster
• VPC cluster

Clean up

Delete all workload clusters before tearing down the management cluster:

# Delete workload clusters first
kubectl delete cluster <cluster-name>

# Stop Tilt (Ctrl-C in the tilt terminal, then)
tilt down

# Delete the kind management cluster
kind delete cluster

Podman setup

If you prefer Podman over Docker, complete these steps before running make kind-cluster.

Emulate the Docker CLI (required for kind to work with Podman):

• General instructions
• macOS-specific instructions

1. Initialise and start the Podman machine:

podman machine init
podman machine start

2. Configure the local registry as insecure:

podman machine ssh
sudo vi /etc/containers/registries.conf

Add at the end of the file:

[[registry]]
location = "localhost:5001"
insecure = true

3. Restart the Podman machine to apply the config:

podman machine stop
podman machine start

Guide for API conversions

Introduction

The purpose of this document is to help/assist contributors with future API conversions using conversion-gen tool.

Prerequisites

1. Create a new API version.

kubebuilder create api --group <group> --version <version> --kind <kind>

2. Copy over existing types, and make the required changes.
3. Mark a storage version, add marker +kubebuilder:storageversion to concerned version package.

NOTE: Refer for more detailed information about prerequisites.

Conversion flow

1. In each “spoke” version package, add marker +k8s:conversion-gen directive pointing to the “hub” version package. It must be in doc.go. Refer
2. In “hub” version package, create doc.go file without any marker. Refer
3. In “spoke” version package, add a var localSchemeBuilder = &SchemeBuilder.SchemeBuilder in groupversion_info.go so the auto-generated code would compile. Refer
4. In “hub” version package, create a conversion.go to implement the “hub” methods. Refer
5. Run target make generate-go-conversions, this will generate zz_generated.conversion.go in the spoke version package.
6. In “spoke” version package, update {kind}_conversion.go to implement Convertible for each type. When conversion-gen stops generating methods because of incompatibilities or we need to override the behavior, we stick them in this source file. Our “spoke” versions need to implement the Convertible interface. Namely, they’ll need ConvertTo and ConvertFrom methods to convert to/from the hub version. Refer

References

• What are hubs and spokes?
• Sample Tutorial
• Prior work to add PowerVS v1beta3 API types

Release Process

Alpha/Beta releases

• Create a tag and push

  git clone git@github.com:kubernetes-sigs/cluster-api-provider-ibmcloud.git
  git tag -s -m "v0.2.0-alpha.3" v0.2.0-alpha.3
  git push origin v0.2.0-alpha.3
  

• Wait for the google cloud build to be finished
• Prepare release notes
• Create a draft release with release notes for the tag
• Tick the prerelease checkbox
• Download the artifacts once cloud build is finished

  gsutil -m cp \
   "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/cluster-template-powervs.yaml" \
   "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/cluster-template.yaml" \
   "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/infrastructure-components.yaml" \
   "gs://artifacts.k8s-staging-capi-ibmcloud.appspot.com/components/v0.2.0-alpha.3/metadata.yaml" \
   .
  

• Upload the downloaded artifacts into the release asset
• Publish the drafted release

Note: In the above instructions, v0.2.0-alpha.3 is the version/tag is being released

GA Releases

• Review if all issues linked to the release version are either completed or moved to the “Next” release.
• Create a release branch from main.
• Clone the repository and create a tag (release tag) and push to origin. Ensure that the GPG keys are set.

  git clone git@github.com:kubernetes-sigs/cluster-api-provider-ibmcloud.git
  git tag -s -m "v0.1.0" v0.1.0
  git push origin v0.1.0
  

• Wait for the Google Cloudbuild to finish, which is triggered once the tag is created.
  • The status of the build jobs can be tracked from: https://prow.k8s.io/?job=post-cluster-api-provider-ibmcloud-push-images
  • The built images are available here: https://console.cloud.google.com/artifacts/docker/k8s-staging-capi-ibmcloud/us/gcr.io
• Create a draft release with release notes for the created tag.
  • Use the make release-notes target to generate release notes. (Refer topic - Prepare release notes)
  • Update the controller image version towards the bottom of the release document.
• Perform the image promotion process:
  • Clone and pull down the latest from kubernetes/k8s.io
  • Create a new branch in your fork of kubernetes/k8s.io.
  • The staging repository is here.
  • Once image is present in the above staging repository, find the sha256 tag for the image by following instructions

  $ manifest-tool inspect --raw gcr.io/k8s-staging-capi-ibmcloud/cluster-api-ibmcloud-controller:v0.1.0 | jq '.digest'
  "sha256:6c92a6a337ca5152eda855ac27c9e4ca1f30bba0aa4de5c3a0b937270ead4363"
  

  • In your kubernetes/k8s.io branch edit registry.k8s.io/images/k8s-staging-capi-ibmcloud/images.yaml and add an entry for the version using the sha256 value got from the above command. For example: "sha256:6c92a6a337ca5152eda855ac27c9e4ca1f30bba0aa4de5c3a0b937270ead4363": ["v0.1.0"]
  • You can use this PR as example.
  • Wait for the PR to be approved and merged.
  • This should trigger a build job to build artifacts through cloud-build / run make release on the release branch.
  • Upload the binaries/files that are uploaded to Google Cloud Storage / built locally and publish the drafted release.
  • Create an alpha tag for the release-version+1 for allowing subsequent commits.

Note: In the above instructions, v0.1.0 is the version/tag is being released

Prepare release notes

1. If you don’t have a GitHub token, create one by going to your GitHub settings, in Personal access tokens. Make sure you give the token the repo scope.

2. Fetch the latest changes from upstream and check out the main branch:

   git fetch upstream
   git checkout main
   

3. Generate release notes by running the following commands on the main branch:

   export GITHUB_TOKEN=<your GH token>
   export RELEASE_TAG=v1.2.3 # change this to the tag of the release to be cut
   make release-notes
   

4. Review the release notes file generated at CHANGELOG/<RELEASE_TAG>.md and make any necessary changes:

• Move items out of “Uncategorized” into an appropriate section.
• Change anything attributed to “k8s-cherrypick-robot” to credit the original author.
• Fix any typos or other errors.
• Add the following section with a link to the full diff:

  ## The image for this release is:
  registry.k8s.io/capi-ibmcloud/cluster-api-ibmcloud-controller:<RELEASE_TAG>
  
  <!-- markdown-link-check-disable-next-line -->
  Full Changelog: https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/compare/v0.9.0...v0.10.0
  

  Be sure to replace the versions in the URL with the appropriate tags.

Post release tasks:

Create a tracker issue using the release template to have a check-list that covers through all tasks that are expected to be done after the release.

Release Support Guidelines

Branches

Cluster API Provider IBM Cloud has two types of branches: the main branch and release-X branches.

The main branch is where development happens. All the latest and greatest code, including breaking changes, happens on main.

The release-X branches contain stable, backwards compatible code. On every major or minor release, a new branch is created. It is from these branches that minor and patch releases are tagged. In some cases, it may be necessary to open PRs for bugfixes directly against stable branches, but this should generally not be the case.

Support and Guarantees

Cluster API Provider IBM Cloud maintains the most recent release/releases for all supported API and contract versions. Support here refers to the ability to backport and release patch versions; the standard backport policy is defined upstream.

For the full version support matrix — including which CAPIBM releases are currently supported, their EOL dates, CAPI compatibility, and Kubernetes version ranges — see the Version Support Policy reference page.

Rules

CAPIBM follows the same N / N-1 / N-2 model as upstream CAPI:

• N and N-1 receive standard support: bug fixes, backports, patch releases, and full CI signal.
• N-2 is in maintenance mode: partial CI only, no proactive backports. Emergency patches may be considered case-by-case by maintainers.
• N-3 and older are EOL and receive no support.
• Test coverage is maintained for all non-EOL minor releases (N, N-1, N-2). When N+1 is released, tests for N-2 are removed (it becomes N-3 / EOL).
• The API version is determined from the GroupVersion defined in the top-level api/ package.
• The EOL date of each API version is determined from the last release available once a new API version is published.

Dependency Updates

• The CAPI, Kubernetes, and test packages receive regular updates for supported releases to ensure they remain synchronised with the CAPI release in use. This is ideally scheduled with every new CAPI n-1 and n-2 minor release.
• IBM Cloud SDK packages are monitored for updates alongside CAPI minor release activity, as long as there are no breaking changes that impact project stability.
• Exceptions can be filed with maintainers and considered on a case-by-case basis.

How to build the machine boot images

VPC

• Building CAPI Images for IBMCLOUD (CAPIBM) VPC

Example

To build an image using a specific version of Kubernetes use the “PACKER_FLAGS” environment variable like in the example below:

# Clone the image-builder repository
$ git clone https://github.com/kubernetes-sigs/image-builder.git
$ cd image-builder/images/capi
$ PACKER_FLAGS="--var 'kubernetes_rpm_version=1.26.2-0' --var 'kubernetes_semver=v1.26.2' --var 'kubernetes_series=v1.26' --var 'kubernetes_deb_version=1.26.2-00'" make build-qemu-ubuntu-2004

PowerVS

• Building CAPI Images for IBMCLOUD (CAPIBM) PowerVS

Example

Compose the user-variables.json file containing the information for the PowerVS

{
  "account_id": "",
  "apikey": "",
  "capture_cos_access_key": "",
  "capture_cos_bucket": "",
  "capture_cos_region": "",
  "capture_cos_secret_key": "",
  "key_pair_name": "",
  "kubernetes_deb_version": "",
  "kubernetes_rpm_version": "",
  "kubernetes_semver": "",
  "kubernetes_series": "",
  "region": "",
  "service_instance_id": "",
  "ssh_private_key_file": "",
  "zone": "",
  "dhcp_network": "false"
}

• account_id: IBM Cloud account ID
• apikey: IBM Cloud API Key
• capture_cos_access_key: IBM Cloud Object Storage(COS) access key where the image will be exported
• capture_cos_bucket: IBM Cloud Object Storage(COS) bucket name
• capture_cos_region: IBM Cloud Object Storage(COS) bucket region
• capture_cos_secret_key: IBM Cloud Object Storage(COS) secret key
• key_pair_name: SSH key name present in the PowerVS
• kubernetes_deb_version: Kubernetes deb version, e.g: 1.34.7-0
• kubernetes_rpm_version: Kubernetes RPM package version, e.g: 1.34.7-0
• kubernetes_semver: e.g: v1.34.7
• kubernetes_series: e.g: v1.34
• region: PowerVS region, e.g: osa
• service_instance_id: PowerVS workspace ID
• ssh_private_key_file: Path to the SSH private key file used to connect to the vm while image preparation, e.g: /Users/manjunath/.ssh/id_rsa
• zone: PowerVS zone, e.g: osa21
• dhcp_network: Set to true if the image has to be built with DHCP support

Note:

1. When setting dhcp_network: true, you need to build an OS image with certain network settings using pvsadm tool and replace the fields with the custom image details.
2. Clone the image-builder repo and run make build commands from a system where the DHCP private IP can be reached and SSH able(you can use a transit gateway with connections added for VPC and PowerVS workspace and build the image from a virtual server instance in VPC).

# Clone the image-builder repository
$ git clone https://github.com/kubernetes-sigs/image-builder.git
$ cd image-builder/images/capi
$ ANSIBLE_SSH_ARGS="-o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa" PACKER_VAR_FILES=user-variables.json make build-powervs-centos-8

Modules and tools dependencies

CAPI Packages

• K8s Packages

• Test Packages

Note: The K8s and Test packages are subject to updates with each new CAPI package release.

IBM Packages

Tools used by E2E tests.

Other Tools

E2E Testing

Introduction

• The end-to-end tests for VPC and PowerVS run on an internal prow cluster on IBM Cloud.
• Resource management is handled via boskos which is an efficient way to lease infra and clean up after every run.
• The E2E tests use the Cluster API test framework. For more information on developing E2E tests, refer here.

Jobs

The following periodic jobs are being run on main branch once every day.

1. periodic-capi-provider-ibmcloud-e2e-powervs
2. periodic-capi-provider-ibmcloud-e2e-vpc

We also test the last two releases, once every week.

Running the end-to-end tests locally

For development and debugging the E2E tests, they can be executed locally.

1. Set the flavor you want to test. By default it is set to powervs-md-remeditaion.

export E2E_FLAVOR=<e2e-flavor>

2. Set the infra environment variables accrodingly based on the flavor being tested. Check the required variables for VPC and PowerVS being set in ci-e2e.sh.
3. Run the e2e test

./hack/scripts/ci/ci-e2e.sh

Step-by-Step Guide: Deploy Latest Branch on Netlify

Step 1: Log in to Netlify

Log in to Netlify at: https://app.netlify.com/

Use the GitHub SSO method to sign in. After logging in, navigate to the project dashboard at: https://app.netlify.com/sites/kubernetes-sigs-cluster-api-ibmcloud

If you do not have access to this Netlify project, you will need to request membership. Raise a Netlify project membership request similar to the following example: https://github.com/kubernetes/org/issues/5284

Note:

• As of today, only @Prajyot-Parab has direct access to this specific Netlify project.
• Members of the Kubernetes Docs Team in Netlify have access to all Netlify projects under the Kubernetes organization.

Step 2: Open Your Project

From the Netlify dashboard, select the project (site) you want to deploy.

Step 3: Go to Project Configuration

In your project dashboard, navigate to Project configuration (or Site settings).

Step 4: Open Build & Deploy Settings

From the left-hand menu, click Build & deploy.

Step 5: Configure Deploy Contexts

Scroll to the Deploy contexts section.

Add the new release branch to the list of branches that should trigger deployments.

Step 6: Create a Pull Request to the Release Branch

In your Git repository (GitHub / GitLab / Bitbucket):

• Create a pull request targeting the release branch
• Ensure all checks pass
• Merge the pull request into the release branch

Step 7: Trigger Netlify Branch Deployment

Once the pull request is merged, Netlify will automatically trigger a deploy for the release branch with the latest changes.

Step 8: Verify Deployment in Netlify

In Netlify, navigate to the Deploys section of your project.

You can:

• Review the branch deploy status and logs for the latest release version
• Preview the deployed site
• Click Retry deploy → Deploy with latest commit if needed

Step 9: Disable Branch Deploys for Custom Domains

In Site settings → Domain management, configure automatic deploy subdomains:

1. Go to Automatic deploy subdomains
2. Click Edit custom domains
3. Uncheck “Branch deploys”
4. Click Save

This ensures branch deploys do not automatically attach to your custom domain.

Step 10: Configure Branch Subdomains

Still in Site settings, configure branch subdomains:

1. Go to Domain management
2. Navigate to Branch subdomains
3. Click Add new branch subdomain
4. Select or enter the new release branch
5. Click Create branch subdomain

Step 11: Enable Branch Deploys for Custom Domains

In Site settings → Domain management, configure automatic deploy subdomains:

1. Go to Automatic deploy subdomains
2. Click Edit custom domains
3. Check “Branch deploys”
4. Click Save

Migration Guides

This section contains migration guides for upgrading between different API versions of the Cluster API Provider for IBM Cloud.

Available Migration Guides

PowerVS API Migrations

• v1beta2 to v1beta3 - Migration guide for PowerVS clusters from v1beta2 to v1beta3 API, covering:
  • Topology specification improvements (VirtualIP vs. LoadBalancer)
  • Data type enhancements (pointer removal for safer, predictable API behavior)
  • Workspace configuration changes (Reference vs. Provision)
  • Network configuration enhancements (explicit type declaration)
  • IBMPowerVSMachine workspace and network reference updates
  • IBMPowerVSImage workspace reference updates
  • Conversion webhook details for automatic migration

Note: This guide covers the currently implemented v1beta3 changes. Additional API improvements will be documented as they are completed.

Why Migrate?

Each new API version brings improvements in:

• Type Safety: Better validation, fewer runtime errors, and pointer-free data structures.
• Discoverability: All configuration options are documented and explorable.
• Maintainability: Clearer intent and simpler controller logic.
• GitOps Compatibility: Better support for modern deployment tools.
• Production Readiness: Enhanced features for enterprise deployments.

Migration Support

• Automatic Conversion: Conversion webhooks provide automatic translation between API versions
• Backward Compatibility: Older API versions continue to work during the migration period
• Validation: CEL validation rules catch configuration errors early
• Documentation: Comprehensive guides with examples for common scenarios

Getting Help

If you encounter issues during migration:

1. Review the specific migration guide for your version
2. Check the Troubleshooting section
3. Consult the API References
4. Open an issue on GitHub
5. Ask questions in the #cluster-api-ibmcloud channel on Kubernetes Slack

Best Practices

When migrating between API versions:

1. Test First: Always test migrations in a non-production environment
2. Read the Guide: Review the complete migration guide before starting
3. Backup Configurations: Keep copies of your current configurations
4. Incremental Migration: Migrate one cluster at a time
5. Monitor: Watch cluster status during and after migration
6. Update Automation: Update any scripts or automation that reference the old API

PowerVS v1beta2 to v1beta3 Migration Guide

Overview

This guide helps you migrate from v1beta2 to v1beta3 PowerVS APIs. The v1beta3 API introduces significant improvements based on Kubernetes API best practices, including enhanced type safety, explicit intent declaration, and better GitOps compatibility.

Table of Contents

What’s Changed

The v1beta3 API introduces several major improvements across PowerVS resources. This guide documents the changes for:

• IBMPowerVSCluster - Topology, Zone, Resource Group, Workspace, Network, TransitGateway, VPC, VPC Subnets, LoadBalancers, VPC Security Groups, COS Instance, and Ignition configuration
• IBMPowerVSMachine - Workspace, Network, Image (Reference/Import), SSH Key, System Type, Processor Type, Processors, Memory, and ProviderID
• IBMPowerVSImage - Workspace, Bucket, Object, Region, Storage Type (typed enum), and Delete Policy (typed enum)

Each section below provides detailed before/after examples and migration guidance.

1. Cluster Topology

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
  annotations:
    powervs.cluster.x-k8s.io/create-infra: "true"  # Annotation-based
spec:
  # Configuration implied by annotation

v1beta3 (New)

Option A: VirtualIP Topology (PowerVS)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  topology: VirtualIP  # Explicit topology declaration
  # No VPC/LoadBalancer configuration needed

Option B: LoadBalancer Topology (PowerVS + VPC)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  topology: LoadBalancer  # Explicit topology declaration
  zone: "wdc06"           # Required for LoadBalancer topology
  resourceGroup:
    type: Reference
    reference:
      name: "my-resource-group"  # Required for LoadBalancer topology
  vpc:
    type: Provision
    region: "us-east"
  # LoadBalancer will be automatically created

Key Points:

• The topology field replaces the annotation-based approach.
• VirtualIP topology: PowerVS network with Virtual IP.
• LoadBalancer topology: Integrates PowerVS with VPC and LoadBalancer.
• The topology is explicit, required, and discoverable via kubectl explain.
• VirtualIP constraints (CEL-enforced): When topology: VirtualIP, workspace must be type: Reference, network must be type: Reference, and transitGateway must not be set.
• LoadBalancer constraints (CEL-enforced): When topology: LoadBalancer, both zone and resourceGroup (with id or name) are required.

2. Zone and Resource Group (Data Type Enhancements)

In v1beta3, structural data types have been flattened to comply with standard Kubernetes API guidelines and prevent runtime errors.

v1beta2 (Deprecated)

In v1beta2, Zone and ResourceGroup used Go pointers (*string and *ResourceReference). This occasionally caused nil-pointer panics in the controller and required complex webhook validations.

# v1beta2 — zone as *string, resourceGroup as *IBMPowerVSResourceReference
spec:
  zone: "wdc06"
  resourceGroup:
    id: "my-rg-id"
    name: "my-resource-group"

v1beta3 (New)

In v1beta3, pointers have been removed in favor of strict value types and a structured ResourceGroupSource.

# v1beta3 — zone as plain string, resourceGroup as ResourceGroupSource
spec:
  zone: "wdc06"
  resourceGroup:
    type: Reference           # Only "Reference" is currently supported
    reference:
      id: "my-rg-id"
      # OR use name
      # name: "my-resource-group"

Key Points:

• Pointer-Free Zone: zone is now a standard string, preventing nil-pointer exceptions.
• Zone is immutable: Once set, zone cannot be changed (CEL immutability rule).
• ResourceGroupSource: resourceGroup now uses a structured ResourceGroupSource type with type and reference fields.
• Only type: Reference is supported for resourceGroup (provisioning a resource group via the API is not supported).
• Conditional Validation: If topology: LoadBalancer, CEL rules strictly enforce that both zone and resourceGroup are provided and non-empty.
• Graceful Omission: If topology: VirtualIP, zone and resourceGroup can be safely omitted.

3. Workspace Configuration

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  serviceInstanceID: "3229a94c-af54-4212-bf60-6202b6fd0a07"  # Deprecated flat field
  # OR the newer (but still v1beta2) form:
  serviceInstance:
    id: "3229a94c-af54-4212-bf60-6202b6fd0a07"
    # OR
    name: "my-existing-workspace"

v1beta3 (New)

Option A: Reference an Existing Workspace

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  workspace:
    type: Reference
    reference:
      id: "3229a94c-af54-4212-bf60-6202b6fd0a07"
      # OR use name instead of id
      # name: "my-existing-workspace"

Option B: Provision a New Workspace

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  workspace:
    type: Provision
    provision:
      name: "my-new-workspace"  # Optional: defaults to <cluster-name>-workspace

Key Points:

• Both serviceInstanceID (deprecated flat field) and serviceInstance are replaced by the workspace field.
• The type field explicitly declares your intent (Reference or Provision).
• Workspace type is immutable: Once set, workspace.type cannot be changed.
• Use reference.id or reference.name to identify existing workspaces; exactly one must be specified (CEL-enforced).
• When provisioning, the workspace name is optional and defaults to <cluster-name>-workspace.
• The controller will only delete workspaces it created (when type: Provision).

4. Network Configuration

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  network:
    name: "capi-test"
  dhcpServer:
    name: "DHCPSERVER-capi-test"
    cidr: "192.168.0.0/24"      # *string pointer
    dnsServer: "8.8.8.8"        # *string pointer, default "1.1.1.1"
    snat: true                   # *bool pointer, default true
    id: "existing-dhcp-server-id"  # optional: reference existing DHCP server

v1beta3 (New)

Option A: Reference an Existing Network

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  network:
    type: Reference
    reference:
      name: "capi-test"
      # OR use id instead of name
      # id: "network-id-12345"

Option B: Provision a New Network with DHCP Server

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  network:
    type: Provision
    provision:
      dhcpServer:
        name: "my-dhcp-server"  # Optional: defaults to DHCPSERVER<cluster-name>_Private
        cidr: "192.168.0.0/24"  # Optional: plain string (no longer a pointer)
        dnsServer: "8.8.8.8"    # Optional: plain string (no longer a pointer)
        snat: Enabled           # Enum: Enabled or Disabled (default: Enabled)

Key Points:

• The type field explicitly declares whether to use an existing or create a new network.
• Network type is immutable: Once set, network.type cannot be changed.
• The top-level dhcpServer field is removed; DHCP configuration is nested under network.provision.dhcpServer.
• SNAT is now an enum (Enabled/Disabled) instead of a *bool pointer.
• All DHCP fields (name, cidr, dnsServer) are plain string values instead of pointers.
• The id field on DHCPServer (to reference an existing server) has been removed; use network.type: Reference instead.
• DHCP server configuration is only valid when type: Provision.
• The controller will only delete networks it created.

5. TransitGateway Configuration

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  transitGateway:
    name: "my-transit-gateway"   # OR use id
    id: "tgw-id-123"
    globalRouting: true          # *bool pointer — true = Global, false = Local

v1beta3 (New)

Option A: Reference an Existing Transit Gateway

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  transitGateway:
    type: Reference
    reference:
      id: "tgw-id-123"
      # OR use name instead of id
      # name: "my-transit-gateway"

Option B: Provision a New Transit Gateway

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  transitGateway:
    type: Provision
    provision:
      name: "my-transit-gateway"       # Optional: defaults to <cluster-name>-tgw
      globalRouting: Global            # Enum: Local or Global (auto-detected if omitted)
    # Optionally control how VPC/PowerVS connections are sourced
    vpcConnection:
      type: Provision
      provision:
        name: "my-vpc-connection"      # Optional: name for the VPC connection
    powerVSConnection:
      type: Provision
      provision:
        name: "my-powervs-connection"  # Optional: name for the PowerVS connection

Option C: Reference Existing Connections within a Provisioned Transit Gateway

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  transitGateway:
    type: Provision
    provision:
      name: "my-transit-gateway"
      globalRouting: Local
    vpcConnection:
      type: Reference
      reference:
        id: "existing-vpc-connection-id"
    powerVSConnection:
      type: Reference
      reference:
        id: "existing-pvs-connection-id"

Key Points:

• The type field (Reference / Provision) replaces the flat name/id struct.
• globalRouting is now an enum (Local / Global) instead of a *bool.
• When globalRouting is omitted, the system automatically selects routing based on PowerVS and VPC regions.
• Individual connections (vpcConnection, powerVSConnection) can each independently reference an existing connection or provision a new one.
• TransitGatewayConnectionSource includes both type, reference, and provision.name fields.
• The controller only deletes Transit Gateways and connections it created (type: Provision).

6. VPC Configuration

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  vpc:
    id: "vpc-id-123"      # OR use name
    name: "my-vpc"
    region: "us-east"     # Required only when create-infra annotation is set

v1beta3 (New)

Option A: Reference an Existing VPC

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  vpc:
    type: Reference
    region: "us-east"     # Always required in v1beta3
    reference:
      id: "vpc-id-123"
      # OR use name instead of id
      # name: "my-vpc"

Option B: Provision a New VPC

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  vpc:
    type: Provision
    region: "us-east"     # Always required in v1beta3
    provision:
      name: "my-new-vpc"  # Optional: defaults to <cluster-name>-vpc

Key Points:

• type (Reference / Provision) is now required and replaces the implicit behavior of v1beta2.
• region is always required in v1beta3 (was only required under the create-infra annotation in v1beta2).
• The type field is immutable once set.
• The controller only deletes VPCs it created (type: Provision).

7. VPC Subnet Configuration

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  vpcSubnets:
    - name: "my-subnet"
      id: "subnet-id-123"    # *string pointer
      zone: "us-east-1"      # *string pointer
      cidr: "10.0.0.0/24"    # *string pointer (IPv4 CIDR block)

v1beta3 (New)

Option A: Reference Existing Subnets

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  subnets:                  # Field renamed from vpcSubnets to subnets
    - type: Reference
      zone: "us-east-1"    # Optional: plain string (no longer a pointer)
      reference:
        id: "subnet-id-123"
        # OR use name instead of id
        # name: "my-subnet"

Option B: Provision New Subnets

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  subnets:
    - type: Provision
      zone: "us-east-1"      # Optional: random zone picked if omitted
      provision:
        name: "my-subnet"    # Optional: defaults to <cluster-name>-vpcsubnet-<INDEX>

Key Points:

• The field was renamed from vpcSubnets to subnets.
• Each entry now requires a type field (Reference / Provision).
• The cidr field from the v1beta2 Subnet struct has been removed in v1beta3.
• All fields (id, name, zone) are plain value types, not pointers.
• When type: Provision and zone is omitted, a random zone is selected from those available in the VPC region.
• The controller only deletes subnets it created (type: Provision).

8. LoadBalancer Configuration

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  loadBalancers:
    - name: "my-lb"
      id: "lb-id-123"          # *string pointer
      public: true             # *bool pointer — true = public, false = private
      additionalListeners:
        - port: 443
          protocol: TCP
          defaultPoolName: "my-pool"   # *string pointer
          selector:
            matchLabels:
              role: worker
      backendPools:
        - name: "my-pool"      # *string pointer
          algorithm: round_robin
          protocol: tcp
          healthMonitor:
            delay: 10
            retries: 3
            timeout: 5
            type: tcp
            port: 8080         # *int64 pointer (optional)
            urlPath: "/healthz" # *string pointer (optional)
      securityGroups:
        - id: "sg-id-123"
          name: "my-sg"        # VPCResource struct with *string fields
      subnets:
        - id: "subnet-id-123"
          name: "my-subnet"    # VPCResource struct with *string fields

v1beta3 (New)

Option A: Reference an Existing Load Balancer

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  loadBalancers:
    - type: Reference
      reference:
        id: "lb-id-123"
        # OR use name instead of id
        # name: "my-lb"

Option B: Provision a New Load Balancer

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  loadBalancers:
    - type: Provision
      provision:
        name: "my-lb"         # Optional: defaults to <cluster-name>-loadbalancer
        type: Public          # Enum: Public or Private (default: Public)
        additionalListeners:
          - port: 443
            protocol: tcp
            defaultPoolName: "my-pool"  # plain string (no longer a pointer)
            selector:
              matchLabels:
                role: worker
        backendPools:
          - name: "my-pool"   # plain string (no longer a pointer)
            algorithm: round_robin
            protocol: tcp
            healthMonitor:
              delay: 10
              retries: 3
              timeout: 5
              type: tcp
              port: 8080       # plain int64 (no longer a pointer)
              urlPath: "/healthz" # plain string (no longer a pointer)
        securityGroups:
          - id: "sg-id-123"   # ResourceIdentifier: id or name (plain strings)
        subnets:
          - name: "my-subnet" # ResourceIdentifier: id or name (plain strings)

Key Points:

• type (Reference / Provision) is now required at the top level of each entry.
• The flat id/name fields on a LoadBalancer entry have moved into reference (when type: Reference).
• The public field (*bool) is replaced by provision.type enum (Public / Private), defaulting to Public.
• securityGroups and subnets now use ResourceIdentifier (plain string fields) instead of the v1beta2 VPCResource struct (pointer fields).
• additionalListeners[].defaultPoolName changed from *string (pointer) to a plain string.
• additionalListeners[].protocol changed from *VPCLoadBalancerListenerProtocol (pointer) to LoadBalancerListenerProtocol (value).
• backendPools[].name changed from *string (pointer) to a plain string.
• healthMonitor.port changed from *int64 (pointer) to a plain int64.
• healthMonitor.urlPath changed from *string (pointer) to a plain string.
• The controller only deletes load balancers it created (type: Provision).

9. VPC Security Groups Configuration

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  vpcSecurityGroups:
    - id: "sg-id-123"          # *string pointer
      name: "my-sg"            # *string pointer
      rules:
        - direction: inbound
          destination:
            protocol: tcp
            portRange:
              minimumPort: 443
              maximumPort: 443
            remotes:
              - remoteType: cidr
                cidrSubnetName: "my-subnet"  # *string pointer
          securityGroupID: "sg-id-123"       # *string pointer
      tags:
        - "env:prod"                         # []*string slice of pointers

v1beta3 (New)

Option A: Reference an Existing Security Group

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  vpcSecurityGroups:
    - type: Reference
      reference:
        id: "sg-id-123"
        # OR use name
        # name: "my-sg"

Option B: Provision a New Security Group

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  vpcSecurityGroups:
    - type: Provision
      provision:
        name: "my-sg"
        tags:
          - "env:prod"              # []string (no longer []*string)
        rules:
          - direction: inbound
            destination:
              protocol: tcp
              portRange:
                minimumPort: 443
                maximumPort: 443
              remotes:
                - remoteType: cidr
                  cidrSubnetName: "my-subnet"  # plain string (no longer a pointer)
            securityGroupID: "sg-id-123"       # plain string (no longer a pointer)

Key Points:

• vpcSecurityGroups now uses VPCSecurityGroupSource with type / reference / provision fields.
• VPCSecurityGroup.id and VPCSecurityGroup.name were *string pointers; now ResourceIdentifier uses plain string values.
• VPCSecurityGroupProvision.tags changed from []*string to []string.
• VPCSecurityGroupRule.securityGroupID changed from *string to a plain string.
• VPCSecurityGroupRule.destination / .source changed from *VPCSecurityGroupRulePrototype (pointer) to VPCSecurityGroupRulePrototype (value).
• VPCSecurityGroupRuleRemote string fields (cidrSubnetName, address, securityGroupName) changed from *string pointers to plain string values.

10. COS Instance Configuration

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  cosInstance:
    name: "my-cos-instance"    # Required when create-infra annotation is set and Ignition is used
    bucketName: "my-bucket"    # Required when create-infra annotation is set and Ignition is used
    bucketRegion: "us-south"   # Required when create-infra annotation is set and Ignition is used

v1beta3 (New)

Option A: Reference an Existing COS Instance

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  cosInstance:
    type: Reference
    bucketName: "my-bucket"    # Required in both Reference and Provision
    bucketRegion: "us-south"   # Required in both Reference and Provision
    reference:
      id: "cos-instance-id"
      # OR use name
      # name: "my-cos-instance"

Option B: Provision a New COS Instance

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  cosInstance:
    type: Provision
    bucketName: "my-bucket"
    bucketRegion: "us-south"
    provision:
      name: "my-cos-instance"  # Optional: name for the COS instance to create

Key Points:

• cosInstance now uses COSInstanceSource with a type / reference / provision structure, consistent with other resources.
• bucketName and bucketRegion are shared fields present at the top level of COSInstanceSource (required regardless of type).
• The flat v1beta2 CosInstance struct (with name, bucketName, bucketRegion) is replaced by this structured form.
• The controller only deletes COS instances it created (type: Provision).

11. Ignition Configuration

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  ignition:
    version: "3.4"  # +kubebuilder:default="2.3", enum: "2.3","2.4","3.0","3.1","3.2","3.3","3.4"

v1beta3 (New)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSCluster
metadata:
  name: my-cluster
spec:
  ignition:
    version: "3.4"  # enum: "2.3","2.4","3.0","3.1","3.2","3.3","3.4"

Key Points:

• The Ignition struct has the same shape in both versions.
• The version field retains the same enum values.
• In v1beta3, the type is now a plain value (not a pointer), consistent with the pointer-free API philosophy.
• CEL validation: If ignition is set, cosInstance must also be configured (required by CEL rule).

12. IBMPowerVSMachine Configuration

The v1beta2 IBMPowerVSMachine spec had several structural issues: dual workspace references (serviceInstanceID + serviceInstance), dual image references (image + imageRef), and pointer-based identifiers. v1beta3 unifies all of these.

12.1 Complete Spec Comparison

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSMachine
metadata:
  name: my-machine
spec:
  # Workspace — two redundant fields:
  serviceInstanceID: "workspace-id-123"   # Deprecated flat field
  serviceInstance:                         # Newer but still v1beta2 form
    id: "workspace-id-123"
    # OR: name: "my-workspace"
    # OR: regex: "workspace-.*"            # regex supported in v1beta2

  # Image — two redundant fields:
  image:                                   # Direct PowerVS image reference
    id: "image-id-123"
    # OR: name: "rhcos-4.14"
    # OR: regex: "rhcos-.*"                # regex supported in v1beta2
  imageRef:                                # Indirect reference via IBMPowerVSImage CRD
    name: "my-ibmpowervsimage"

  # Network
  network:
    id: "network-id-123"
    # OR: name: "my-network"
    # OR: regex: "network-.*"              # regex supported in v1beta2

  # SSH Key
  sshKey: "my-ssh-key"

  # System Type
  # +kubebuilder:validation:Enum:="s922";"e980";"s1022";"e1050";"e1080";""
  systemType: "s922"

  # Processor Type
  # +kubebuilder:validation:Enum:="Dedicated";"Shared";"Capped";""
  processorType: "Shared"

  # Processors (int or string for fractional values)
  processors: "0.25"

  # Memory (GiB)
  memoryGiB: 4

  # ProviderID
  providerID: "ibmpowervs://us-south/my-workspace/my-instance-id"  # *string pointer

v1beta3 (New)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSMachine
metadata:
  name: my-machine
spec:
  # Workspace — single unified field using ResourceIdentifier (id or name, exactly one)
  workspace:
    id: "workspace-id-123"
    # OR (not both): name: "my-workspace"
    # If omitted, workspace is inherited from IBMPowerVSCluster

  # Image — single unified field with explicit type
  image:
    type: Reference         # Use an existing PowerVS image
    reference:
      id: "image-id-123"
      # OR (not both): name: "rhcos-4.14"

  # OR: import via IBMPowerVSImage CRD
  # image:
  #   type: Import
  #   import:
  #     name: "my-ibmpowervsimage"  # name of the IBMPowerVSImage CRD resource

  # Network — uses ResourceIdentifier (id or name, exactly one)
  network:
    id: "network-id-123"
    # OR (not both): name: "my-network"
    # If omitted, network is inherited from IBMPowerVSCluster

  # SSH Key — unchanged
  sshKey: "my-ssh-key"

  # System Type — pattern-based validation (no longer a strict enum)
  # Pattern: ^[a-z][0-9]+$  (e.g. s922, e980, s1022, s1122, e1050, e1080)
  systemType: "s922"

  # Processor Type — same enum values
  # +kubebuilder:validation:Enum:="Dedicated";"Shared";"Capped";""
  processorType: "Shared"

  # Processors — unchanged (int or string for fractional values)
  processors: "0.25"

  # Memory — unchanged
  memoryGiB: 4

  # ProviderID — now a plain string (no longer a *string pointer)
  providerID: "ibmpowervs://us-south/my-workspace/my-instance-id"

12.2 Workspace Reference

v1beta2 (Deprecated)

spec:
  serviceInstanceID: "workspace-id-123"  # Deprecated flat string
  # OR
  serviceInstance:
    id: "workspace-id-123"
    # OR: name: "my-workspace"
    # OR: regex: "workspace-.*"   # regex supported in v1beta2

v1beta3 (New)

spec:
  workspace:
    id: "workspace-id-123"
    # OR (not both)
    # name: "my-workspace"

Key Points:

• Both serviceInstanceID and serviceInstance are replaced by the single workspace field.
• Uses ResourceIdentifier type with id or name (exactly one must be set; CEL-enforced).
• regex is not supported in v1beta3 ResourceIdentifier; use id or name only.
• If omitted, workspace is inherited from the associated IBMPowerVSCluster.

12.3 Image Reference

v1beta2 (Deprecated)

spec:
  # Option 1: Reference an existing PowerVS image directly
  image:
    id: "image-id-123"
    name: "rhcos-4.14"

  # Option 2: Reference via IBMPowerVSImage CRD (mutually exclusive with image above)
  imageRef:
    name: "my-ibmpowervsimage"

v1beta3 (New)

spec:
  # Option 1: Reference an existing PowerVS image (id or name, exactly one)
  image:
    type: Reference
    reference:
      id: "image-id-123"
      # OR (not both): name: "rhcos-4.14"

  # Option 2: Import via IBMPowerVSImage CRD
  image:
    type: Import
    import:
      name: "my-ibmpowervsimage"  # name of the IBMPowerVSImage CRD resource

Key Points:

• The dual image / imageRef fields are unified into a single image field with an explicit type.
• type: Reference replaces the v1beta2 image field.
• type: Import replaces the v1beta2 imageRef field.
• CEL validation ensures reference is present when type: Reference and import is present when type: Import.
• image is a required field in v1beta3.

12.4 Network Reference

v1beta2 (Deprecated)

spec:
  network:                     # Required field in v1beta2
    id: "network-id-123"
    # OR: name: "my-network"
    # OR: regex: "network-.*"  # regex supported in v1beta2

v1beta3 (New)

spec:
  network:                     # Optional field in v1beta3
    id: "network-id-123"
    # OR (not both): name: "my-network"

Key Points:

• The network field uses ResourceIdentifier — the same type as other identifiers in v1beta3.
• network is now optional in v1beta3 (was required in v1beta2); if omitted, the network is inherited from the cluster.
• Supports id or name (exactly one must be set; CEL-enforced). regex is not supported in v1beta3.

12.5 ProviderID

v1beta2 (Deprecated)

spec:
  providerID: "ibmpowervs://us-south/my-workspace/my-instance-id"  # *string (pointer)

v1beta3 (New)

spec:
  providerID: "ibmpowervs://us-south/my-workspace/my-instance-id"  # string (value)

Key Points:

• providerID changed from *string (pointer) to a plain string.

12.6 Machine Status Changes

v1beta2 Status

status:
  ready: true                   # bool
  instanceID: "my-instance-id"
  addresses:
    - type: InternalIP
      address: "192.168.0.10"
  health: "OK"
  instanceState: "ACTIVE"
  fault: "some fault message"   # Removed in v1beta3
  failureReason: "..."          # Deprecated, removed in v1beta3
  failureMessage: "..."         # Deprecated, removed in v1beta3
  region: "us-south"            # *string pointer
  zone: "us-south-1"            # *string pointer
  conditions:
    - type: Ready
      status: "True"
  v1beta2:
    conditions:
      - type: Ready
        status: "True"

v1beta3 Status

status:
  conditions:
    - type: Ready               # Promoted to top-level []metav1.Condition
      status: "True"
  initialization:
    provisioned: true           # Replaces the ready bool field
  instanceID: "my-instance-id"
  addresses:
    - type: InternalIP
      address: "192.168.0.10"
  health: "OK"
  instanceState: "ACTIVE"
  region: "us-south"            # plain string (no longer a pointer)
  zone: "us-south-1"            # plain string (no longer a pointer)
  deprecated:
    v1beta2:
      conditions:               # Deprecated v1beta1-style conditions
        - type: Ready
          status: "True"

Key Points:

• ready bool is replaced by initialization.provisioned *bool.
• fault, failureReason, failureMessage have been removed.
• region and zone changed from *string pointers to plain string values.
• Top-level conditions are now []metav1.Condition (metav1 structured conditions).
• Old-style Conditions (clusterv1beta1) are moved to deprecated.v1beta2.conditions.

13. IBMPowerVSImage Configuration

13.1 Complete Spec Comparison

v1beta2 (Deprecated)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: IBMPowerVSImage
metadata:
  name: my-image
  namespace: default
spec:
  clusterName: "my-cluster"

  # Workspace — two redundant fields:
  serviceInstanceID: "workspace-id-123"   # Deprecated flat field
  serviceInstance:                         # Newer but still v1beta2 form
    id: "workspace-id-123"
    # OR: name: "my-workspace"

  # COS Source
  bucket: "my-cos-bucket"        # *string pointer, required
  object: "rhcos-image.ova.gz"   # *string pointer, required
  region: "us-south"             # *string pointer, required

  # Storage Type — plain string with enum validation
  # +kubebuilder:default=tier1
  # +kubebuilder:validation:Enum=tier0;tier1;tier3
  storageType: "tier1"

  # Delete Policy — plain string with enum validation
  # +kubebuilder:default=delete
  # +kubebuilder:validation:Enum=delete;retain
  deletePolicy: "delete"

v1beta3 (New)

apiVersion: infrastructure.cluster.x-k8s.io/v1beta3
kind: IBMPowerVSImage
metadata:
  name: my-image
  namespace: default
spec:
  clusterName: "my-cluster"   # plain string with MinLength/MaxLength validation

  # Workspace — single unified ResourceIdentifier field
  workspace:
    id: "workspace-id-123"
    # OR: name: "my-workspace"
    # If omitted, workspace is derived from the IBMPowerVSCluster status

  # COS Source — now plain strings (no longer *string pointers)
  bucket: "my-cos-bucket"        # plain string, required
  object: "rhcos-image.ova.gz"   # plain string, required
  region: "us-south"             # plain string, required

  # Storage Type — now a typed enum (PowerVSStorageType)
  # Values: tier0, tier1, tier3
  storageType: tier1

  # Delete Policy — now a typed enum (PowerVSImageDeletePolicy)
  # Values: delete, retain
  deletePolicy: delete

13.2 Workspace Reference

v1beta2 (Deprecated)

spec:
  serviceInstanceID: "workspace-id-123"  # Deprecated flat string
  # OR
  serviceInstance:
    id: "workspace-id-123"
    name: "my-workspace"

v1beta3 (New)

spec:
  workspace:
    id: "workspace-id-123"
    # OR
    name: "my-workspace"

Key Points:

• Both serviceInstanceID and serviceInstance are replaced by the single workspace field using ResourceIdentifier.
• If workspace is omitted, the workspace is automatically derived from the associated IBMPowerVSCluster’s status.

13.3 COS Source Fields

v1beta2 (Deprecated)

spec:
  bucket: "my-cos-bucket"      # *string pointer
  object: "rhcos-image.ova.gz" # *string pointer
  region: "us-south"           # *string pointer

v1beta3 (New)

spec:
  bucket: "my-cos-bucket"      # plain string, MinLength=1, MaxLength=63
  object: "rhcos-image.ova.gz" # plain string, MinLength=1, MaxLength=1024
  region: "us-south"           # plain string, MinLength=1, MaxLength=32

Key Points:

• All three COS fields changed from *string pointers to plain string values.
• v1beta3 adds explicit MinLength / MaxLength validation markers on each field.

13.4 StorageType and DeletePolicy

v1beta2 (Deprecated)

spec:
  storageType: "tier1"   # plain string — validated by +kubebuilder:validation:Enum=tier0;tier1;tier3
  deletePolicy: "delete" # plain string — validated by +kubebuilder:validation:Enum=delete;retain

v1beta3 (New)

spec:
  storageType: tier1   # PowerVSStorageType — a named Go type: tier0 | tier1 | tier3
  deletePolicy: delete # PowerVSImageDeletePolicy — a named Go type: delete | retain

Key Points:

• storageType changed from an untyped string to the named type PowerVSStorageType.
• deletePolicy changed from an untyped string to the named type PowerVSImageDeletePolicy.
• The allowed values remain the same (tier0, tier1, tier3 and delete, retain respectively).
• Typed enums provide better discoverability via kubectl explain and improved code safety.

13.5 Image Status Changes

v1beta2 Status

status:
  ready: true                    # bool
  imageID: "image-id-123"
  imageState: "active"
  jobID: "job-id-123"
  conditions:
    - type: Ready
      status: "True"
  v1beta2:
    conditions:
      - type: Ready
        status: "True"

v1beta3 Status

status:
  conditions:
    - type: Ready               # Promoted to top-level []metav1.Condition
      status: "True"
  imageID: "image-id-123"       # with MinLength/MaxLength validation
  imageState: "active"          # with MinLength/MaxLength validation
  jobID: "job-id-123"           # with MinLength/MaxLength validation
  deprecated:
    v1beta2:
      conditions:               # Deprecated v1beta1-style conditions
        - type: Ready
          status: "True"

Key Points:

• ready bool is removed; readiness is conveyed through conditions.
• Top-level conditions are now []metav1.Condition.
• Old-style Conditions moved to deprecated.v1beta2.conditions.
• imageID, imageState, and jobID now have explicit MinLength / MaxLength validation markers.

14. Status Field Changes

IBMPowerVSCluster v1beta2 Status

status:
  ready: false                              # bool
  resourceGroupID:
    id: "rg-id"
    controllerCreated: true                 # *bool pointer
  serviceInstance:
    id: "workspace-id"
    controllerCreated: true
  network:
    id: "network-id"
    controllerCreated: true
  dhcpServer:
    id: "dhcp-id"
    controllerCreated: true
  vpc:
    id: "vpc-id"
    controllerCreated: true
  vpcSubnet:                                # map[string]ResourceReference
    us-east-1:
      id: "subnet-id"
      controllerCreated: true
  vpcSecurityGroups:                        # map[string]VPCSecurityGroupStatus
    my-sg:
      id: "sg-id"
      ruleIDs: ["rule-id-1"]
      controllerCreated: true
  transitGateway:
    id: "tgw-id"
    controllerCreated: true
    vpcConnection:
      id: "conn-id"
      controllerCreated: true
    powerVSConnection:
      id: "conn-id"
      controllerCreated: true
  cosInstance:
    id: "cos-id"
    controllerCreated: true
  loadBalancers:                            # map[string]VPCLoadBalancerStatus
    my-lb:
      id: "lb-id"
      hostname: "my-lb.example.com"        # *string pointer
      controllerCreated: true
  conditions:
    - type: Ready
      status: "True"
  v1beta2:
    conditions:
      - type: Ready
        status: "True"

IBMPowerVSCluster v1beta3 Status

status:
  conditions:                               # Promoted to top-level []metav1.Condition
    - type: Ready
      status: "True"
  initialization:
    provisioned: true                       # Replaces ready bool
  workspace:                               # Renamed from serviceInstance; no controllerCreated
    id: "workspace-id"
    name: "my-workspace"
  network:                                 # No controllerCreated; DHCP nested here
    id: "network-id"
    name: "my-network"
    dhcpServer:
      id: "dhcp-id"
      name: "my-dhcp"
  resourceGroup:
    id: "rg-id"
    name: "my-resource-group"
  vpc:
    id: "vpc-id"
    name: "my-vpc"
    region: "us-east"                      # Added region to VPC status
  vpcSubnets:                              # list (renamed from vpcSubnet map)
    - id: "subnet-id"
      name: "my-subnet"
      zone: "us-east-1"
  vpcSecurityGroups:                       # list (was map[string]VPCSecurityGroupStatus)
    - id: "sg-id"
      name: "my-sg"
      rules:
        - id: "rule-id-1"
  transitGateway:
    id: "tgw-id"
    name: "my-tgw"
    vpcConnection:                         # Connection status includes name and state
      id: "conn-id"
      name: "my-vpc-conn"
      state: "attached"
    powerVSConnection:
      id: "conn-id"
      name: "my-pvs-conn"
      state: "attached"
  cosInstance:
    id: "cos-id"
    name: "my-cos-instance"
    bucketName: "my-bucket"
    bucketRegion: "us-south"
  loadBalancers:                           # list (was map[string]VPCLoadBalancerStatus)
    - name: "my-lb"
      id: "lb-id"
      state: "active"
      hostname: "my-lb.example.com"        # plain string (no longer a pointer)
  deprecated:
    v1beta2:
      conditions:
        - type: Ready
          status: "True"

Key Points:

• controllerCreated removed from all status fields; ownership is determined solely by the type field in Spec.
• ready bool replaced by initialization.provisioned *bool.
• serviceInstance renamed to workspace in status.
• dhcpServer status moved from a top-level field to nested under network.dhcpServer.
• resourceGroupID field key renamed to resourceGroup in status.
• VPC status now includes region.
• vpcSubnet (keyed map) renamed to vpcSubnets (ordered list); each entry includes zone.
• vpcSecurityGroups changed from map[string]VPCSecurityGroupStatus to []VPCSecurityGroupStatus; each entry now exposes name and per-rule id.
• TransitGateway connection status now includes name and state (e.g., attached, pending).
• loadBalancers changed from map[string]VPCLoadBalancerStatus to []LoadBalancerStatus; hostname is now a plain string.
• COSInstance status now exposes name, bucketName, and bucketRegion.
• Top-level conditions are []metav1.Condition; old-style conditions moved to deprecated.v1beta2.conditions.

15. Conversion Webhook

The v1beta3 API includes automatic conversion webhooks that handle migration:

• v1beta2 → v1beta3: Automatically converts old format to new

  • Status.ControllerCreated: true → Spec.Type: Provision
  • Status.ControllerCreated: false → Spec.Type: Reference
  • serviceInstanceID / serviceInstance → workspace
  • dhcpServer (top-level) → network.provision.dhcpServer
  • Boolean SNAT (*bool) → Enum SNAT (true → Enabled, false → Disabled)
  • *bool globalRouting on TransitGateway → Enum routing (true → Global, false → Local)
  • *bool public on LoadBalancer → Enum type (true → Public, false → Private)
  • Annotation-based topology → Explicit topology field
  • *string zone → plain string zone
  • *IBMPowerVSResourceReference resourceGroup → ResourceGroupSource resourceGroup
  • vpcSubnets[] (flat Subnet struct with *string fields) → subnets[] with type/reference/provision
  • loadBalancers keyed map in status → loadBalancers list in status
  • vpcSubnet keyed map in status → vpcSubnets list in status
  • vpcSecurityGroups keyed map in status → vpcSecurityGroups list in status
  • image + imageRef (IBMPowerVSMachine) → unified image with type: Reference or type: Import
  • *string bucket/object/region (IBMPowerVSImage) → plain string fields
  • Untyped storageType/deletePolicy strings → typed PowerVSStorageType/PowerVSImageDeletePolicy
  • COS instance flat struct → COSInstanceSource with type/reference/provision

• v1beta3 → v1beta2: Converts back for compatibility

  • Spec.Type: Provision → Status.ControllerCreated: true
  • Spec.Type: Reference → Status.ControllerCreated: false
  • workspace → serviceInstance
  • network.provision.dhcpServer → top-level dhcpServer
  • Explicit topology field → Annotation-based configuration
  • Enum SNAT → *bool SNAT
  • Enum routing → *bool globalRouting
  • Enum LB type → *bool public
  • Unified image (Reference/Import) → image + imageRef split
  • Typed enums → plain strings for storageType/deletePolicy

Note: While conversion webhooks provide compatibility, it is recommended to migrate to v1beta3 explicitly for better maintainability.

Additional Resources

• PowerVS Prerequisites
• Creating a PowerVS Cluster
• API References

Troubleshooting

1. Tilt stops working as not able to connect to kind cluster

% kind get clusters
 enabling experimental podman provider
 ERROR: failed to list clusters: command "podman ps -a --filter label=io.x-k8s.kind.cluster --format '{{index .Labels "io.x-k8s.kind.cluster"}}'" failed with error: exit status 125
 Command Output: Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
 Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:61514: connect: connection refused

1. Stop and start the Podman either via cli or from Podman Desktop.

   $ podman machine stop
   $ podman machine start
   

2. Run all the stopped containers like capi-test-control-plane, capi-test-worker, kind-registry.

   $ podman container list -a
     CONTAINER ID  IMAGE                                    NAMES
     512cee59230c  docker.io/library/registry:2             kind-registry
     5b99fd84c41e  docker.io/kindest/node@sha256            capi-test-worker
     94130af58929  docker.io/kindest/node@sha256            capi-test-control-plane
   
   $ podman container start 512cee59230c 5b99fd84c41e 94130af58929
   

3. Try re-running tilt up from cluster-api directory.

2. SSH into data/control plane node configured with DHCP network

1. Since the VM backing the node is configured with DHCP network which is private we can’t directly SSH into it.

2. Create a public VM in the same workspace and attach the DHCP network to it.

   1. Create public network in PowerVS workspace if it does not exist using ibmcloud cli

   $ibmcloud pi subnet create publicnet1 --net-type public
   

   2. List the available images to create VM

   $ibmcloud pi image lc
   

   3. Create the VM with public and DHCP subnet.

   $ibmcloud pi instance create publicVM --image testrhel88 --subnets DHCPSERVERcapi-powervs-new_Private,publicnet1
   

   4. Get the public IP of created VM

   $ibmcloud pi ins get publicVM
   

3. SSH into the DHCP VM using public VM as a jump host.

   ssh -J root@<public_ip> root@<dhcp_ip>
   

3. Failed to apply a cluster template with release not found error

While trying to apply a cluster template from unreleased version like from main branch, we will run into error like release not found for version vX.XX.XX. In that case, instead of --flavor we need to use --from=<path_to_cluster_template>.

4. Debugging Machine struck in PROVISIONED phase

• A Machine’s Running phase indicates that it has successfully created, initialised and has become a Kubernetes Node in a Ready state.

• Sometimes a machine will be in Provisioned phase forever indicating infrastructure has been created and configured but yet to become a Kubernetes node.

• Cloud controller manager(CCM) takes care of turning a machine into a node by fetching and initialising with appropriate data from cloud.

• As a part of cluster create template we make use of ClusterResourceSet to apply the CCM resources into the workload cluster.

• Check the machine’s current status

  $ kubectl get machines
  NAME                          CLUSTER   NODENAME   PROVIDERID                                                                                         PHASE          AGE     VERSION
  powervs-control-plane-pqnt4   powervs              ibmpowervs://osa/osa21/10b1000b-da8d-4e18-ad1f-6b2a56a8c130/bc0c9621-12d2-47f1-932e-a18ff041aba2   Provisioned    5m36s   v1.31.0
  

• Verify that the ClusterResourceSet is applied to the workload cluster

  $ kubectl get clusterresourceset
  NAME             AGE
  crs-cloud-conf   10m
  
  $ kubectl describe clusterresourceset crs-cloud-conf
  .
  .
  Status:
    Conditions:
      Last Transition Time:  2025-05-06T08:36:40Z
      Message:
      Observed Generation:   1
      Reason:                Applied
      Status:                True
      Type:                  ResourcesApplied
      Last Transition Time:  2025-05-06T08:31:27Z
      Message:
      Observed Generation:   1
      Reason:                NotPaused
      Status:                False
      Type:                  Paused
  

• Verify that the CCM resources are created in the workload cluster

  • Get the workload cluster kubeconfig

    $ clusterctl get kubeconfig powervs > workload.conf
    

  • Check the CCM daemonset’s status

    $ kubectl get daemonset -n kube-system --kubeconfig=workload.conf
    NAME                                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                            AGE
    ibmpowervs-cloud-controller-manager   2         2         2       2            2           node-role.kubernetes.io/control-plane=   45m
    

  • Check the logs of CCM

    $ kubectl -n kube-system get pods --kubeconfig=workload.conf
    ibmpowervs-cloud-controller-manager-472lq                     1/1     Running   1 (45m ago)   46m
    ibmpowervs-cloud-controller-manager-fw47h                     1/1     Running   1 (38m ago)   38m
    
    $ kubectl -n kube-system logs ibmpowervs-cloud-controller-manager-472lq --kubeconfig=workload.conf
    I0506 09:23:51.420992       1 ibm_metadata_service.go:206] Retrieving information for node=powervs-control-plane-ftd8j from Power VS
    I0506 09:23:51.421003       1 ibm_powervs_client.go:270] Node powervs-control-plane-ftd8j found metadata &{InternalIP:192.168.236.114 ExternalIP:163.68.98.114 WorkerID:001275c5-f454-4944-8419-61c16f16f8b7 InstanceType:s922 FailureDomain:osa21 Region:osa ProviderID:ibmpowervs://osa/osa21/10b1000b-da8d-4e18-ad1f-6b2a56a8c130/001275c5-f454-4944-8419-61c16f16f8b7} from DHCP cache
    I0506 09:23:51.421038       1 node_controller.go:271] Update 3 nodes status took 7.03624ms.
    

  • Check the cloud-conf config map

    $ kubectl -n kube-system get cm ibmpowervs-cloud-config -o yaml --kubeconfig=workload.conf
    apiVersion: v1
    kind: ConfigMap
    metadata:
       creationTimestamp: "2025-05-06T08:36:39Z"
       name: ibmpowervs-cloud-config
       namespace: kube-system
       resourceVersion: "329"
       uid: ae2bd436-0b1e-4534-9c6c-48f717f6f47e
    data:
    ibmpowervs.conf: |
      [global]
      version = 1.1.0
      [kubernetes]
       config-file = ""
       [provider]
       cluster-default-provider = g2
       .
       .
    

  • Check whether the secret is configured with correct IBM Cloud API key.

    $ kubectl -n kube-system get secret ibmpowervs-cloud-credential -o yaml --kubeconfig=workload.conf
    

• Check whether the node is initialised correctly and does not have taint node.cloudprovider.kubernetes.io/uninitialized taint

  $ kubectl get nodes --kubeconfig=workload.conf
  NAME                          STATUS     ROLES           AGE   VERSION
  powervs-control-plane-ftd8j   NotReady   control-plane   53m   v1.31.0
  powervs-control-plane-pqnt4   NotReady   control-plane   61m   v1.31.0
  powervs-md-0-2dnrm-8658c      NotReady   <none>          56m   v1.31.0
  
  
  $ kubectl get node powervs-control-plane-ftd8j -o yaml --kubeconfig=workload.conf
  apiVersion: v1
  kind: Node
  metadata:
    annotations:
      cluster.x-k8s.io/annotations-from-machine: ""
      cluster.x-k8s.io/cluster-name: powervs
      cluster.x-k8s.io/cluster-namespace: default
      cluster.x-k8s.io/labels-from-machine: ""
      cluster.x-k8s.io/machine: powervs-control-plane-ftd8j
      cluster.x-k8s.io/owner-kind: KubeadmControlPlane
      cluster.x-k8s.io/owner-name: powervs-control-plane
      kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
      node.alpha.kubernetes.io/ttl: "0"
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
  

• On the successful CCM initialisation the machine will turn into Running phase and corresponding NODENAME field will be populated.

  NAME                          CLUSTER   NODENAME                      PROVIDERID                                                                                         PHASE          AGE     VERSION
  powervs-control-plane-pqnt4   powervs   powervs-control-plane-pqnt4   ibmpowervs://osa/osa21/10b1000b-da8d-4e18-ad1f-6b2a56a8c130/bc0c9621-12d2-47f1-932e-a18ff041aba2   Running        8m52s   v1.31.0
  

Reference

Further references to the Cluster API Provider IBM Cloud - CAPIBM project.

• Version Support Policy
• API reference
• Regions-Zones Mapping

Version Support Policy

This page documents the support policy for Cluster API Provider IBM Cloud (CAPIBM), including which CAPIBM releases are supported, which versions of Cluster API (CAPI) they are compatible with, and which Kubernetes versions can be used as management and workload clusters.

Note: CAPIBM ships a single provider binary but contains two independent infrastructure providers — IBM Power Virtual Server (PowerVS) and IBM VPC — each with their own API version track.

Supported CAPIBM Releases

CAPIBM follows the same support model as upstream CAPI:

• N (latest minor) — Standard support: bug fixes, patch releases, full CI signal.
• N-1 — Standard support: bug fixes, patch releases, full CI signal.
• N-2 — Maintenance mode: partial CI only, no proactive backports; emergency patches considered case-by-case by maintainers.
• N-3 and older — EOL: no support.

Cluster API (CAPI) Compatibility

The table below maps each CAPIBM release range to the CAPI contract version it implements. Both providers (PowerVS and VPC) ship in the same binary and share the same CAPI contract.

Current stable: CAPIBM main / v0.15.x targets CAPI v1beta2, which corresponds to CAPI releases v1.11.x, v1.12.x, and v1.13.x.

See the official Cluster API version support page for CAPI’s own support lifecycle and its Kubernetes compatibility matrix.

Kubernetes Version Compatibility

CAPIBM follows CAPI’s Kubernetes support policy. The supported Kubernetes versions for both providers are determined by the CAPI version the release depends on.

Management Cluster

Each CAPI minor release supports four Kubernetes minor versions (N to N-3) for the management cluster at initial cut. The table below shows the combined range covered by the listed CAPI minors for each CAPIBM release line.

Workload Cluster

The workload cluster Kubernetes version is independent of the management cluster version. Management and workload clusters can be upgraded in any order. Both must fall within the range that the CAPI version in use has been tested with.

Each CAPI minor release supports (at initial cut):

• Management cluster: Kubernetes N to N-3 (4 minor versions)
• Workload cluster: Kubernetes N to N-5 (6 minor versions)

As new Kubernetes minor releases ship, CAPI extends both ranges in patch releases. Refer to the Cluster API supported versions page for the exact matrix tied to each CAPI release.

Upgrade and Downgrade Policy

Upgrading CAPIBM

CAPIBM follows the same skip-upgrade limit as upstream CAPI: you can skip at most N-3 minor versions in a single upgrade. For example, upgrading from v0.12.x directly to v0.15.x is the maximum allowed skip. Skipping more than three minor versions may leave the management cluster in a non-functional state.

Always upgrade clusterctl first, then use it to upgrade all other components.

Downgrades

CAPIBM does not support version downgrades. Downgrading may leave the management cluster in a non-functional state.

PowerVS API Version Lifecycle

The PowerVS provider (infrastructure.cluster.x-k8s.io/powervs) has its own API version track, currently ahead of the VPC provider.

• The current stable PowerVS API version is v1beta3 (storage version since v0.14).
• v1beta2 is still served by the API server and automatically converted to v1beta3 at admission. It will remain served until v0.17 per the Kubernetes deprecation policy (minimum 3 minor releases after v1beta3 was introduced in v0.14).
• See the PowerVS v1beta2 → v1beta3 migration guide to migrate your manifests before v1beta2 is unserved.

PowerVS v1beta2 Removal Roadmap

v1beta2 follows the Kubernetes API deprecation policy and must remain served for at least 3 minor CAPIBM releases after v1beta3 was introduced. The planned removal schedule is:

Action required: Migrate all PowerVS manifests and tooling from v1beta2 to v1beta3 before upgrading to CAPIBM v0.17 (expected April 2027), when v1beta2 will no longer be served by the API server.

The four-version gap between unserved (v0.17) and removed (v0.21) ensures that managedField cleanup runs even for clusters that skip up to N-3 versions before upgrading.

VPC API Version Lifecycle

The VPC provider (infrastructure.cluster.x-k8s.io/vpc) has an independent API version track, currently at v1beta2.

• The current stable VPC API version is v1beta2.
• v1beta1 resources are automatically converted to v1beta2 (the hub version) at admission.

Support Rules Summary

1. N and N-1 (two most recent minor releases) receive standard support: bug fixes, backports, patch releases, and full CI signal.
2. N-2 is in maintenance mode: partial CI preserved for emergency patch capability; no proactive backports; security scans may be disabled.
3. N-3 and older are EOL: no support, no backports.
4. Test coverage is maintained for N, N-1, and N-2. When a new release makes the former N-2 into N-3, its tests are removed.
5. The CAPI, Kubernetes, and test package dependencies are kept in sync with supported CAPI minor releases. Updates are targeted with every new CAPI N-1 and N-2 minor release.
6. IBM Cloud SDK packages are updated alongside CAPI minor release updates, as long as there are no breaking changes that impact project stability.
7. Exceptions can be filed with maintainers and considered on a case-by-case basis.

Further Reading

• Cluster API Version Support Policy — upstream CAPI version and Kubernetes compatibility matrix.
• Release Support Guidelines — branch strategy, backport policy, and release cadence for CAPIBM.
• Release Process — how CAPIBM releases are cut.
• CAPIBM Releases — full list of published releases on GitHub.

API References

Cluster API Provider IBMCloud currently exposes the following APIs:

• The Cluster API Provider IBMCloud Custom Resource Definitions (CRDs): documentation

• Golang APIs: godoc

Regions-Zones Mapping

References:

1. IBM Cloud Documentation for:
   • PowerVS Locations
   • VPC Regions
   • Transit Gateway
2. Deploy on IBM Cloud:
   • PowerVS
   • VPC
   • Transit Gateway