Buildkit-cli for kubectl

Building images on the cluster

Developing and testing on a kubernetes cluster usually means building a new image, pushing it to your registry, then pulling and redeploying your containers. Especially for local clusters, like on premises or even on your own workstation in a vm, it would be nice to be able to build your images directly on the nodes, making turn around much faster. Buildkit already exists, but now a new github project makes this even easier by giving you a kubectl extension to build directly on your cluster using regular kubectl commands. 

The project is in its early stages, but you can already try it. Check it out at https://github.com/vmware-tanzu/buildkit-cli-for-kubectl. Basically it provides a kubectl build command that will spawn a builder on your cluster (if it isn't yet running), then build your image on the node. When you have a multi node cluster, it can propagate the image to all nodes that have a builder. 

Buildkit cli on a local bare metal cluster

We are running on a bare metal K3S cluster (Leda) having three nodes (Helena, Castor & Pollux). First we install version 0.1.0 of the buildkit-cli on our development box:

devbox1:~/$ wget https://github.com/vmware-tanzu/buildkit-cli-for-kubectl/releases/download/v0.1.0/linux-refs.tags.v0.1.0.tgz
devbox1:~/$ cd /usr/local/bin
devbox1:/usr/local/bin$ sudo tar xzvf ~/linux-refs.tags.v0.1.0.tgz
devbox1:~$ kubectl build --help

Next we create a simple project, which is just a static html page hosted with nginx.     

devbox1:~/Projects/SimpleServer$ tree
.
├── Dockerfile
└── html
    └── index.html

1 directory, 2 files
Dockerfile  html
devbox1:~/Projects/SimpleServer$ cat Dockerfile
 FROM nginx:latest
 COPY html /usr/share/nginx/html

If we now run the kubectl build command, a single builder will be spawned and the image will be built on one node. We are providing a specific tag, kube, to make sure kubernetes won't try pulling a latest image when we run the container.  

devbox1:~/Projects/SimpleServer$ kubectl build -t hellobuildkit:kube .
devbox1:~/Projects/SimpleServer$ ansible leda -m shell -a "docker image ls|grep hellobuild"
helena | CHANGED | rc=0 >>
hellobuildkit                                             kube                 7643a3d01bf5        31 minutes ago      133MB

pollux | CHANGED | rc=0 >>

castor | CHANGED | rc=0 >>
devbox1:~/Projects/SimpleServer$ kubectl run --image=hellobuildkit:kube hellobuildkit -n default

So the image is available on one node only. If we want to make sure the image is available on all nodes, we need to scale up the buildkit to three replicas. Sidenote; the Buildkit-cli project is looking into the possibility to use deamon sets to make sure the builder is available on every node. 

devbox1:~/Projects/SimpleServer$ kubectl get pods -n default -o wide
NAME                        READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
buildkit-7c5b7d6cfb-h4tql   1/1     Running   0          3m29s   10.42.2.19   castor   <none>           <none>
buildkit-7c5b7d6cfb-hfk4t   1/1     Running   0          33m     10.42.0.45   helena   <none>           <none>
buildkit-7c5b7d6cfb-w4xvj   1/1     Running   0          3m29s   10.42.1.14   pollux   <none>           <none>
hellobuildkit               1/1     Running   0          13m     10.42.0.48   helena   <none>           <none>
devbox1:~/Projects/SimpleServer$ kubectl build -t hellobuildkit:kube .
[+] Building 20.9s (10/10) FINISHED
 => [internal] load .dockerignore                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                          0.0s
 => [internal] load build definition from Dockerfile                                                                                                     0.1s
 => => transferring dockerfile: 31B                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/nginx:latest                                                                                          1.3s
 => [internal] load build context                                                                                                                        0.0s
 => => transferring context: 60B                                                                                                                         0.0s
 => [1/2] FROM docker.io/library/nginx:latest@sha256:c3a1592d2b6d275bef4087573355827b200b00ffc2d9849890a4f3aa2128c4ae                                    0.0s
 => => resolve docker.io/library/nginx:latest@sha256:c3a1592d2b6d275bef4087573355827b200b00ffc2d9849890a4f3aa2128c4ae                                    0.0s
 => CACHED [2/2] COPY html /usr/share/nginx/html                                                                                                         0.0s
 => exporting to oci image format                                                                                                                       17.7s
 => => exporting layers                                                                                                                                  0.0s
 => => exporting manifest sha256:c7a43630ebc3d197c484ed41b2aa68d1f00d10e4c928955015eb899b237b329c                                                        0.0s
 => => exporting config sha256:7643a3d01bf5dbb6cc27d6b8dd07a33d5d3f68b3c993df25e6c69c432f940a56                                                          0.0s
 => => sending tarball                                                                                                                                  17.6s
 => loading image to docker runtime via pod buildkit-7c5b7d6cfb-hfk4t                                                                                    0.0s
 => loading image to docker runtime via pod buildkit-7c5b7d6cfb-w4xvj                                                                                    1.8s
 => loading image to docker runtime via pod buildkit-7c5b7d6cfb-h4tql                                                                                    1.7s
devbox1:~/Projects/SimpleServer$ ansible leda -m shell -a "docker image ls|grep hellobuild"
helena | CHANGED | rc=0 >>
hellobuildkit          kube                 7643a3d01bf5        31 minutes ago      133MB

pollux | CHANGED | rc=0 >>
hellobuildkit          kube                 7643a3d01bf5        31 minutes ago      133MB

castor | CHANGED | rc=0 >>
hellobuildkit          kube                 7643a3d01bf5        31 minutes ago      133MB

 So with the image now available on all nodes, we can scale up our container however we want on the cluster.

Conclusion

It's definitely useful to be able to quickly build and deploy your images on a local cluster without having to go through an (external) build pipeline and repository. The project is in the early stages, but it's already quite useful. One to watch for sure.