RSS

Detect Malicious Behaviour on Kubernetes API Server through gathering Audit Logs by using FluentBit - Part 2

By Batuhan Apaydın |

Introduction

In the previous blog post, we had talked about the Audit Logs in more detail, this post is a continuation of the previous blog post, so I suggest you take a look at the previous blog post before continuing reading. The only difference in this blog post is that this time we will use Log Backend mode, which is one of the Audit Backends that kube-apiserver supports, then we will use some kind of log collector project which is FluentBit in this case to gather these audit logs and forward them to some HTTP endpoint. You might be thinking why this kind of architecture can be more useful, the answer is that in the previous blog post we used Webhook Backend mode, so we could send the audit logs to one location only. But with using Log Backend mode and FluentBit, we are now able to send audit logs to different locations, so we are now more flexible in this regard.

Here is the high-level overview architecture that shows what we want to achieve at the end of the day:

falco_fluentbit_kre

Prerequisites

We need tools with the following minimum versions to achieve this demo:

  • minikube v1.22.0
  • kubectl v1.21.2
  • helm v3.6.2+gee407bd

Tutorial

Provision local Kubernetes Cluster

There are various ways to provision a local Kubernetes cluster such as KinD, k3s, k0s, Minikube etc. We are going to use Minikube this time in this walkthrough.

There is a detailed guide about configuring Audit Logging in minikube, if you want to learn more about it, please see.

Let's get provision a local Kubernetes cluster:

$ minikube config view
- cpus: 3
- driver: virtualbox
- memory: 8192

$ minikube start
😄  minikube v1.22.0 on Darwin 10.15.7
✨  Using the virtualbox driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing virtualbox VM for "minikube" ...
🐳  Preparing Kubernetes v1.21.2 on Docker 20.10.6 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Enable Kubernetes Audit Logging

Let's connect to the Minikube instance, which runs in VM, and download the Audit Policy file. This file is kind of a filter to decide what should be in the Audit Logs. This Audit Log file can grow very quickly and cause some performance issues if we didn't filter anything.

This is not the recommended Audit Policy file, this is just an example, it would be better to prepare your Audit Policy file that is suitable for your needs.

$ minikube ssh
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$ sudo mkdir -p /etc/kubernetes/audit
$ sudo curl -sLo /etc/kubernetes/audit/audit-policy.yaml \
       https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/audit/audit-policy.yaml

Let's edit the kube-apiserver, which is deployed as static Pod with the necessary flags and volumes to enable the Audit Logging.

$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
# Update Result
spec.containers.kube-apiserver.command
  + two list entries added:
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/kube-apiserver-audit.log

spec.containers.kube-apiserver.volumeMounts
  + two list entries added:
    - name: audit-log
    │ mountPath: /var/log/kubernetes/kube-apiserver-audit.log
    │ readOnly: false
    - name: audit-policy
    │ mountPath: /etc/kubernetes/audit/audit-policy.yaml
    │ readOnly: true

spec.volumes
  + two list entries added:
    - name: audit-log
    │ hostPath:
    │ │ type: FileOrCreate
    │ │ path: /var/log/kubernetes/kube-apiserver-audit.log
    - name: audit-policy
    │ hostPath:
    │ │ type: File
    │ │ path: /etc/kubernetes/audit/audit-policy.yaml

Once you updated the manifest file, the kube-apiserver can't be reachable for a while, so please keep trying until the kube-apiserver becoming healthy.

The audit log file should be generated when the kube-apiserver becomes healthy, so please be sure about this before moving into the next phase.

$ minikube ssh
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$  ls -latr /var/log/kubernetes/
total 4868
drwxr-xr-x 6 root root    4096 Jul 21 17:37 ..
drwxr-xr-x 2 root root    4096 Jul 21 17:37 .
-rw-r--r-- 1 root root 4973184 Jul 21 18:39 kube-apiserver-audit.log

Install Falco and Falcosidekick with Auditing Feature Enabled

We'll use Helm to install the Falco, so there is an option that we can enable or disable the audit log feature called auditLog.enabled, once we set the value of the option as true, the embedded webserver will be started within the Falco to consume audit events from the port 8765 and behind k8s-audit endpoint.

Firstly, we'll create the namespace that will host both Falco and Falcosidekick:

$ kubectl create namespace falco
namespace/falco created

We add the helm repo:

$ helm repo add falcosecurity https://falcosecurity.github.io/charts
"falcosecurity" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository

In a real project, you should get the whole chart with helm pull falcosecurity/falco --untar and then configure the values.yaml. For this tutorial, will try to keep thing as easy as possible and set configs directly by passing arguments to helm install command line:

$ helm upgrade --install falco falcosecurity/falco --namespace falco \
--set falcosidekick.enabled=true \
--set falcosidekick.webui.enabled=true \
--set auditLog.enabled=true

You should get this output:

Release "falco" does not exist. Installing it now.
NAME: falco
LAST DEPLOYED: Tue Apr 13 10:49:49 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.


No further action should be required.

And you can see your new Falco and Falcosidekick pods:

$ kubectl get pods --namespace falco
NAME                                      READY   STATUS    RESTARTS   AGE
falco-falcosidekick-76f5885f7f-956vj      1/1     Running   0          4m27s
falco-falcosidekick-76f5885f7f-tmff6      1/1     Running   0          4m27s
falco-falcosidekick-ui-5b64749bc8-k8v4p   1/1     Running   0          4m27s
falco-h4qvx                               1/1     Running   0          4m27s

Check your services within the namespace falco, and you should see that Falco service has 8765 port, let's verify that.

$ kubectl get services --namespace falco
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
falco                    ClusterIP   10.96.156.167   <none>        8765/TCP   4m40s
falco-falcosidekick      ClusterIP   10.105.37.184   <none>        2801/TCP   4m40s
falco-falcosidekick-ui   ClusterIP   10.111.57.54    <none>        2802/TCP   4m40s

The argument falcosidekick.enabled=true sets the following settings in Falco for you:

--set falco.jsonOutput=true \
--set falco.httpOutput.enabled=true \
--set falco.httpOutput.url=http://falco-falcosidekick:2801

The arguments --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falco-falcosidekick:2801 are there to configure the format of events and the URL where Falco will send them. As Falco and Falcosidekick will be in the same namespace, it can directly use the name of the service (falco-falcosidekick) above Falcosidekick pods.

We check the logs:

$ kubectl logs deployment/falco-falcosidekick --namespace falco
Found 2 pods, using pod/falco-falcosidekick-76f5885f7f-tmff6
2021/05/22 15:46:58 [INFO]  : Enabled Outputs : [WebUI]
2021/05/22 15:46:58 [INFO]  : Falco Sidekick is up and listening on :2801

WebUI is displayed as enabled output, everything is good 👍.

Deploy FluentBit

In order to deploy FluentBit, we're going to use its Helm chart with our own configurations.

If you want to get more detail about the Helm chart of the FluentBit, please see.

$ cat << EOF | tee values-override.yaml
config:
  inputs: |
   [INPUT]
       Name              tail
       Alias             kube_audit
       Tag               kube_audit
       Buffer_Chunk_Size 512k
       Buffer_Max_Size   5m
       Path              /var/log/kubernetes/kube-apiserver-audit.log
       Parser            json
       DB                /var/log/kube_audit_log.db
       Mem_Buf_Limit     128MB
       Skip_Long_Lines   On
       Refresh_Interval  10

  outputs: |
    [OUTPUT]
        Name            http
        Alias           kube_audit_to_falco
        Match           kube_audit
        Host            falco.falco
        Port            8765
        URI             /k8s-audit
        Format          json
EOF

$ kubectl create namespace logging
namespace/logging created

$ helm upgrade --install fluent-bit fluent/fluent-bit \
        -f values-override.yaml \
        -n logging
Release "fluent-bit" has been upgraded. Happy Helming!
NAME: fluent-bit
LAST DEPLOYED: Wed Jul 21 22:59:25 2021
NAMESPACE: logging
STATUS: deployed
REVISION: 1
NOTES:
Get Fluent Bit build information by running these commands:

export POD_NAME=$(kubectl get pods --namespace logging -l "app.kubernetes.io/name=fluent-bit,app.kubernetes.io/instance=fluent-bit" -o jsonpath="{.items[0].metadata.name}")
echo "curl http://127.0.0.1:2020 for Fluent Bit build information"
kubectl --namespace logging port-forward $POD_NAME 2020:2020

Now we are ready to test everything.

Test

There is a rule called Create/Modify Configmap With Private Credentials in Falco for detecting ConfigMaps that contains private credentials from Kubernetes Audit Logs. You can see the details of the rule from this link. To test this, we should create a ConfigMap that contains private credentials.

$ kubectl --namespace=falco port-forward svc/falco-falcosidekick-ui 2802
Forwarding from 127.0.0.1:2802 -> 2802
Forwarding from [::1]:2802 -> 2802

Now, let's create the ConfigMap.

$ kubectl create configmap aws-creds \
  --from-literal aws_access_key_id=AKES20LNOA
configmap/aws-creds created

As soon as you created the ConfigMap, the Audit Log will be generated, then FluentBit will gather the audit log and send it to Falco. In turn, Falco will detect the malicious behavior from the audit logs and send an alert to the Falcosidekick. Finally, Falcosidekick forwards the alert to the Web UI. So, you should see similar screens like the following:

falcosidekick_web_ui

terminal_view

Conclusion

With this really simple example, we only want to show how FluentBit can be useful for us to implement this kind of architecture.

If you would like to find out more about Falco: