Buildkit-cli for kubectl
Building images on the cluster
Developing and testing on a kubernetes cluster usually means building a new image, pushing it to your registry, then pulling and redeploying your containers. Especially for local clusters, like on premises or even on your own workstation in a vm, it would be nice to be able to build your images directly on the nodes, making turn around much faster. Buildkit already exists, but now a new github project makes this even easier by giving you a kubectl extension to build directly on your cluster using regular kubectl commands.
The project is in its early stages, but you can already try it. Check it out at https://github.com/vmware-tanzu/buildkit-cli-for-kubectl. Basically it provides a kubectl build command that will spawn a builder on your cluster (if it isn't yet running), then build your image on the node. When you have a multi node cluster, it can propagate the image to all nodes that have a builder.
Buildkit cli on a local bare metal cluster
We are running on a bare metal K3S cluster (Leda) having three nodes (Helena, Castor & Pollux). First we install version 0.1.0 of the buildkit-cli on our development box:
devbox1:~/$ wget https://github.com/vmware-tanzu/buildkit-cli-for-kubectl/releases/download/v0.1.0/linux-refs.tags.v0.1.0.tgz devbox1:~/$ cd /usr/local/bin devbox1:/usr/local/bin$ sudo tar xzvf ~/linux-refs.tags.v0.1.0.tgz devbox1:~$ kubectl build --help
Next we create a simple project, which is just a static html page hosted with nginx.
devbox1:~/Projects/SimpleServer$ tree . ├── Dockerfile └── html └── index.html 1 directory, 2 files Dockerfile html devbox1:~/Projects/SimpleServer$ cat Dockerfile FROM nginx:latest COPY html /usr/share/nginx/html
If we now run the kubectl build command, a single builder will be spawned and the image will be built on one node. We are providing a specific tag, kube, to make sure kubernetes won't try pulling a latest image when we run the container.
devbox1:~/Projects/SimpleServer$ kubectl build -t hellobuildkit:kube . devbox1:~/Projects/SimpleServer$ ansible leda -m shell -a "docker image ls|grep hellobuild" helena | CHANGED | rc=0 >> hellobuildkit kube 7643a3d01bf5 31 minutes ago 133MB pollux | CHANGED | rc=0 >> castor | CHANGED | rc=0 >> devbox1:~/Projects/SimpleServer$ kubectl run --image=hellobuildkit:kube hellobuildkit -n default
So the image is available on one node only. If we want to make sure the image is available on all nodes, we need to scale up the buildkit to three replicas. Sidenote; the Buildkit-cli project is looking into the possibility to use deamon sets to make sure the builder is available on every node.
devbox1:~/Projects/SimpleServer$ kubectl get pods -n default -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES buildkit-7c5b7d6cfb-h4tql 1/1 Running 0 3m29s 10.42.2.19 castor <none> <none> buildkit-7c5b7d6cfb-hfk4t 1/1 Running 0 33m 10.42.0.45 helena <none> <none> buildkit-7c5b7d6cfb-w4xvj 1/1 Running 0 3m29s 10.42.1.14 pollux <none> <none> hellobuildkit 1/1 Running 0 13m 10.42.0.48 helena <none> <none> devbox1:~/Projects/SimpleServer$ kubectl build -t hellobuildkit:kube . [+] Building 20.9s (10/10) FINISHED => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 31B 0.0s => [internal] load metadata for docker.io/library/nginx:latest 1.3s => [internal] load build context 0.0s => => transferring context: 60B 0.0s => [1/2] FROM docker.io/library/nginx:latest@sha256:c3a1592d2b6d275bef4087573355827b200b00ffc2d9849890a4f3aa2128c4ae 0.0s => => resolve docker.io/library/nginx:latest@sha256:c3a1592d2b6d275bef4087573355827b200b00ffc2d9849890a4f3aa2128c4ae 0.0s => CACHED [2/2] COPY html /usr/share/nginx/html 0.0s => exporting to oci image format 17.7s => => exporting layers 0.0s => => exporting manifest sha256:c7a43630ebc3d197c484ed41b2aa68d1f00d10e4c928955015eb899b237b329c 0.0s => => exporting config sha256:7643a3d01bf5dbb6cc27d6b8dd07a33d5d3f68b3c993df25e6c69c432f940a56 0.0s => => sending tarball 17.6s => loading image to docker runtime via pod buildkit-7c5b7d6cfb-hfk4t 0.0s => loading image to docker runtime via pod buildkit-7c5b7d6cfb-w4xvj 1.8s => loading image to docker runtime via pod buildkit-7c5b7d6cfb-h4tql 1.7s devbox1:~/Projects/SimpleServer$ ansible leda -m shell -a "docker image ls|grep hellobuild" helena | CHANGED | rc=0 >> hellobuildkit kube 7643a3d01bf5 31 minutes ago 133MB pollux | CHANGED | rc=0 >> hellobuildkit kube 7643a3d01bf5 31 minutes ago 133MB castor | CHANGED | rc=0 >> hellobuildkit kube 7643a3d01bf5 31 minutes ago 133MB
So with the image now available on all nodes, we can scale up our container however we want on the cluster.
It's definitely useful to be able to quickly build and deploy your images on a local cluster without having to go through an (external) build pipeline and repository. The project is in the early stages, but it's already quite useful. One to watch for sure.