Adding a container image registry
From the previous post, I now had my 5 node k8s cluster, and had just built my own very basic container based on Alpine Linux 3.8 for the Raspberry Pi. The next step is to have this container available to my k8s cluster, which means setting up a container image registry.
Installing the registry
As far as I can tell at this point, there aren’t as many choices available for a container registry as I expected. The most popular method seems to be running docker-registry as a docker container. This isn’t officially available for the Raspberry Pi platform, although there are some efforts at building an Pi compatible container. In the end, I decided on installing docker-distribution package on a CentOS 7 VM, which includes a version 2.0 docker registry.
yum install docker-distribution
systemctl start docker-distribution && systemctl enable docker-distribution
I’ll also need to open port 5000 on the local firewall to make it accessible to remote hosts. Most registry clients will connect to the registry through a secure (TLS) connection, however by default the installation does not generate a certificate for TLS connections. I’ve used openssl to generate a self-signed certificate and key pair for TLS connections (which is still not ideal, but good enough for a home lab). I have updated my /etc/docker-distribution/registry/config.yml to look as follows:
version: 0.1
log:
fields:
service: registry
storage:
cache:
layerinfo: inmemory
filesystem:
rootdirectory: /var/lib/registry
delete:
enabled: true
http:
addr: :5000
tls:
certificate: /etc/docker-latest/certs.d/localhost:5000/registry.crt
key: /etc/docker-latest/certs.d/localhost:5000/registry.key
Restarting the docker-distribution daemon, and I have my private registry up and running. The next step will be to populate it with the container images I’ve built.
I have also enabled the ability to delete images from the registry. I added this flag later as I hadn’t quite figured out the naming scheme for images I wanted to use and ended up with some duplicates and some naming schemes that didn’t make sense. Also, being able to delete old/unused images freed up some space.
Pushing the images
Back on my Raspberry Pi builder system, I have to push the built image up to the registry. The steps to follow are:
- Build the image.
- Tag the image.
- Push the image to the registry
Building the image:
docker build .
It really is as simple as that. I’ve added my ‘pi’ user to the docker group, which grants members access to the docker daemon. The docker command then builds a container image from the Dockerfile in the current directory. Once built, the list of locally available container images can be displayed as follows:
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> 32106d439320 4 days ago 4.01MB
As can be seen, this image doesn’t belong to a repository and it hasn’t been tagged yet. I can pass a tag flag to docker build to name and tag the image as I’d prefer, for example:
docker build . -t centos-docker.ipa.champion:5000/arm/alpine/alpine:latest
To tag the existing image:
docker tag 32106d439320 centos-docker.ipa.champion:5000/arm/alpine/alpine:latest
Great! I have my local image reading for pushing the remote container registry I have created. I need to configure the local docker daemon on the Pi builder system to use my ‘insecure’ registry (it is to be listed as insecure as I am using a self-signed certificate). To do this I alter (or add) the /etc/docker/daemon.json file to contain the following information:
# cat /etc/docker/daemon.json
{
"insecure-registries" : ["centos-docker.ipa.champion:5000"]
}
Once this file is in place, the docker daemon can be restarted, and the image pushed as such:
docker push centos-docker.ipa.champion:5000/arm/alpine/alpine:latest
The image should then be pushed to my remote container registry.
Pulling the images
This is only half the job complete. I need to be able to pull the images from the registry onto my k8s cluster. This will require configuring the docker daemons on each k8s node to use my private (insecure) registry. As the nodes are simply using docker, it’s the same change as above. The /etc/docker/daemon.json file should contain the same information as on my builder system, and the docker daemon restarted on each node.
As before, I did this via a 5-way tmux console, but a better approach would be to use a tool like Ansible – something for the future I think.
Whether this has all worked, can be tested out by deploying a pod to the k8s cluster using the container that was uploaded to the registry. A sample yaml file (alpine-example.yaml) could be:
apiVersion: v1
kind: Pod
metadata:
name: alpine-test
spec:
containers:
- image: centos-docker.ipa.champion:5000/arm/alpine/alpine
command:
- sleep
- "3600"
imagePullPolicy: Always
name: alpine-test
restartPolicy: Always
This is a simple pod that runs the command ‘sleep 3600’ (keeps it running for one hour, with a restart policy of always meaning it will be restarted after the one hour has elapsed). The pod can be deployed as follows:
$ kubectl create -f alpine-example.yaml
pod/alpine-test created
$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE alpine-test 0/1 ContainerCreating 0 5s <none> green.kube.ipa.champion
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE alpine-test 1/1 Running 0 23s 10.244.1.103 green.kube.ipa.champion
The image has been pulled from the remote registry and is now up and running on the k8s cluster. We can exec into this image as follows:
$ kubectl exec -it alpine-test -- /bin/sh
/ # cat /etc/alpine-release
3.8.0 / #
Since deploying docker-distribution as the registry service, I’ve found both portus (developed by SUSE) and harbor (developed by VMware but now part of the CNCF) as freely available container registries (both seem to be provided as container images though, and require more detailed configuration than docker-distribution).
Where to next?
The next steps in this journey will be to save my container Dockerfiles (and other requirements) for redeployment and rebuilding when necessary.