Eclipse Che on microk8s

In my ongoing search for cloud-native development tools I came across Eclipse Che, a developer workspaces solution for Red Hat OpenShift and Kubernetes. It is the upstream, open-source project to the Red Hat OpenShift Dev Spaces product.

Eclipse Che lets you define development environments with YAML, using devfiles (yes, the same devfiles behind ODO). Che can then reproduce the environment on the click of a button, on a Kubernetes cluster. Working with their own forks of the project, developers can then iterate and test their work, within the Che namespace on the cluster. By default, a VSCode-based browser IDE comes with Che; extensions to VSCode and IntelliJ Idea also exist. Che uses chectl as a command-line interface to create, manage and delete workspaces.

I gather that Eclipse Che is primarily developed for OpenShift as a production target and Minikube as a local development target. Because I have been evaluating for a local development use-case, I tried to deploy it on microk8s on Ubuntu (minikube is not available on Ubuntu and microk8s comes closest to it).

💡

Che supports deployments on microk8s.

OpenID Connect

Eclipse Che needs to manage user logins. It mandates the presence of an OpenID Connect (OIDC) service on the target cluster. From issues and user reports, it looks like Che is well-tuned to use the OIDC services on OpenShift and minikube. Che documents officially recommends Dex, and the latter seems to work well on microk8s.

Feeling confident about it, I tried to configure Dex as the OIDC for Eclipse Che and ran into issues.

💡

The chectl CLI formerly provided a --skip-oidc-provider-check option to disable the mandatory OIDC check. Though this allowed Che to get deployed, without an OIDC, users ran into issues later.

Configuring Dex

I was somewhat relieved to find a document that has instructions for installing Dex on a microk8s cluster. A self-signed certificate is created for Dex. We also create a certificate which dex will use to serve HTTPS traffic. Dex gets deployed with ease and we then use the following patch to launch Che:

kind: CheCluster
apiVersion: org.eclipse.che/v2
spec:
  networking:
    auth:
      oAuthClientName: "che-client"
      oAuthSecret: "placeholder-secret-12345"
      identityProviderURL: "https://192.168.1.39:31000/dex"

Before deploying Che, we also need an update to the Kubernetes API server configuration; the latter needs to be told about the OIDC and its certificate. Subsequently, a microk8s restart is also needed.

# Add this to /var/snap/microk8s/current/args/kube-apiserver
--oidc-issuer-url=https://192.168.1.39:31000/dex
--oidc-client-id=che-client
--oidc-username-claim=email
--oidc-groups-claim=groups

With a lot of hopes, I then tried to deploy Che with this command:

chectl server:deploy \
  --platform microk8s \
  --installer operator \
  --domain 192.168.1.39.nip.io \
   --che-operator-cr-patch-yaml ./che-patch.yaml

Unfortunately, though all the Che pods come up, the Che server pod - che-84566474cf-28t64 below - does not start up. The Che startup eventually times out because the Che server pod never comes up.

$ microk8s kubectl get pods -n eclipse-che
NAME                             READY   STATUS      RESTARTS   AGE
che-84566474cf-28t64             0/1     Running     0          76s
che-dashboard-844ff4fff8-xmm84   1/1     Running     0          90s
che-gateway-5499c97fbb-fnzdv     4/4     Running     0          89s
che-operator-7c69bd76d8-tbmxf    1/1     Running     0          2m5s
che-tls-job-cwvc4                0/1     Completed   0          110s

An analysis of the logs revealed a chain of RuntimeExceptions coming out of the Guice package. At the start of the chain, there is this:

Caused by: ValidatorException: PKIX path building failed: SunCertPathBuilderException: unable to find valid certification path to requested target

This simply means Che cannot recognize the self-signed certificate of the Dex OIDC. Now, I failed to find any guidance on this. I am hoping the solution would be simple enough.

💡

I also found an open issue about OIDC on microk8s. Evidently, this is a documentation problem.

I also to hack around the certificate problem by configuring Dex over HTTP. But, on making that change to the kube-apiserver configuration microk8s fails to restart. Clearly, Kubernetes does not like OIDC over HTTP.

Replacing Dex with KeyCloak

Coming across this document that demonstrates deploying Che with KeyCloak, I followed the instructions therein, adapting them for microk8s. Additionally, I also had to use the KeyCloak admin console and create a realm for Che and a client in that realm.

Unfortunately, but probably expected, I ran into the same certificate validation exception with KeyCloak.

Evidently, Che written in Java does not recognize the self-signed certificate of KeyCloak (and Dex). I guess the solution for this could be straightforward. But I cannot find any documentation around it. I should probably raise a issue with the Eclipse Che project.

Concluding thoughts

Eclipse Che is a developer workspaces solution built for the enterprise, primarily for Openshift and minikube. Scaling it down for a local development use-case with microk8s requires significant effort and Kubernetes skill. It looks like an overkill for a local cloud-native development tool for individuals and small teams.