kube-bench/docs/README.md

# Test and config files

`kube-bench` runs checks specified in `controls` files that are a YAML 
representation of the CIS Kubernetes Benchmark checks. There is a 
`controls` file per kubernetes version and node type.

kube-bench automatically selects which `controls` to use based on the detected
node type and the version of kubernetes a cluster is running. This behaviour
can be overridden by specifying the `master` or `node` subcommand and the
`--version` flag on the command line.

For example:
run kube-bench against a master with version  auto-detection:

```
kube-bench master
```

or run kube-bench against a node with the node `controls` for kubernetes 
version 1.12:
```
kube-bench node --version 1.12
```

`controls` for the various versions of kubernetes can be found in directories
with same name as the kubernetes versions under `cfg/`, for example `cfg/1.12`.
`controls` are also organized by distribution under the `cfg` directory for
example `cfg/ocp-3.10`.


## Controls

`controls` is a YAML document that contains checks that must be run against a 
specific kubernetes node type, master or node and version.

`controls` is the fundamental input to `kube-bench`. The following is an example 
of a basic `controls`:

```
---
controls:
id: 1
text: "Master Node Security Configuration"
type: "master"
groups:
- id: 1.1
  text: API Server
  checks:
    - id: 1.1.1
      text: "Ensure that the --allow-privileged argument is set (Scored)"
      audit: "ps -ef | grep kube-apiserver | grep -v grep"
      tests:
      bin_op: or
      test_items:
      - flag: "--allow-privileged"
        set: true
      - flag: "--some-other-flag"
        set: false
      remediation: "Edit the /etc/kubernetes/config file on the master node and
        set the KUBE_ALLOW_PRIV parameter to '--allow-privileged=false'"
      scored: true
- id: 1.2
  text: Scheduler
  checks:
    - id: 1.2.1
      text: "Ensure that the --profiling argument is set to false (Scored)"
      audit: "ps -ef | grep kube-scheduler | grep -v grep"
      tests:
        bin_op: or
        test_items:
          - flag: "--profiling"
            set: true
          - flag: "--some-other-flag"
            set: false
      remediation: "Edit the /etc/kubernetes/config file on the master node and
        set the KUBE_ALLOW_PRIV parameter to '--allow-privileged=false'"
      scored: true
```

`controls` is composed of a hierachy of groups, sub-groups and checks. Each of
the `controls` components have an id and a text description which are displayed 
in the `kube-bench` output.

`type` specifies what kubernetes node type a `controls` is for. Possible  values
for `type` are `master` and `node`.

## Groups

`groups` is list of subgroups which test the various kubernetes components
that run on the node type specified in the `controls`. 

For example one subgroup checks parameters passed to the apiserver binary, while 
another subgroup checks parameters passed to the controller-manager binary.

```
groups:
- id: 1.1
  text: API Server
  ...
- id: 1.2
  text: Scheduler
  ...
```

These subgroups have `id`, `text` fields which serve the same purposes described
in the previous paragraphs. The most important part of the subgroup is the
`checks` field which is the collection of actual `check`s that form the subgroup.

This is an example of a subgroup and checks in the subgroup.

```
id: 1.1
text: API Server
checks:
  - id: 1.1.1
    text: "Ensure that the --allow-privileged argument is set (Scored)"
    audit: "ps -ef | grep kube-apiserver | grep -v grep"
    tests:
    ...
  - id: 1.1.2
    text: "Ensure that the --anonymous-auth argument is set to false (Not Scored)"
    audit: "ps -ef | grep kube-apiserver | grep -v grep"
    tests:
    ...
``` 

`kube-bench` supports running a subgroup by specifying the subgroup `id` on the
command line, with the flag `--group` or `-g`.

## Check

The CIS Kubernetes Benchmark recommends configurations to harden kubernetes 
components. These recommendations are usually configuration options, and can be 
specified by flags to kubernetes binaries, or in configuration files.

The Benchmark also provides commands to audit a kubernetes installation, identify
places where the cluster security can be improved, and steps to remediate these
identified problems.

In `kube-bench`, `check` objects embody these recommendations.  This an example
`check` object:

```
id: 1.1.1
text: "Ensure that the --anonymous-auth argument is set to false (Not Scored)"
audit: "ps -ef | grep kube-apiserver | grep -v grep"
tests:
  test_items:
  - flag: "--anonymous-auth"
    compare:
      op: eq
      value: false
    set: true
remediation: |
  Edit the API server pod specification file kube-apiserver
  on the master node and set the below parameter.
  --anonymous-auth=false
scored: false
```

A `check` object has an `id`, a `text`, an `audit` , a `tests`,`remediation`
and `scored` fields.

`kube-bench` supports running individual checks by specifying the check's `id`
as a comma-delimited list on the command line with the `--check` flag.

The `audit` field specifies the command to run for a check. The output of this
command is then evaluated for conformance with the CIS Kubernetes Benchmark
recommendation.

The audit is evaluated against a criteria specified by the `tests`
object. `tests` contain `bin_op` and `test_items`.

`test_items` specify the criteria(s) the `audit` command's output should meet to
pass a check. This criteria is made up of keywords extracted from the output of
the `audit` command and operations that compare the these keywords against
values expected by the CIS Kubernetes Benchmark. 

The are two ways to extract keywords from the output of the `audit` command,
`flag` and `path`.

`flag` is used when the keyword is a command line flag. The associated `audit`
command is usually a `ps` command and a `grep` for the binary whose flag we are
checking:

```
ps -ef | grep somebinary | grep -v grep
``` 

Here is an example usage of the `flag` option:

```
...
audit: "ps -ef | grep kube-apiserver | grep -v grep"
tests:
  test_items:
  - flag: "--anonymous-auth"
  ...
```

`path` is used when the keyword is an option set in a JSON or YAML config file.
The associated `audit` command is usually `cat /path/to/config-yaml-or-json`.
For example:

```
...

text: "Ensure that the --anonymous-auth argument is set to false (Not Scored)"
audit: "cat /path/to/some/config"
tests:
  test_items:
  - path: "{.someoption.value}"
    ...
```

`test_item` compares the output of the audit command and keywords using the
`set` and `compare` fields.

```
  test_items:
  - flag: "--anonymous-auth"
    compare:
      op: eq
      value: false
    set: true
```

`set` checks if a keyword is present in the output of the audit command or in
a config file. The possible values for `set` are true and false.

If `set` is true, the check passes only if the keyword is present in the output
of the audit command, or config file. If `set` is false, the check passes only
if the keyword is not present in the output of the audit command, or config file.

`compare` has two fields `op` and `value` to compare keywords with expected
value. `op` specifies which operation is used for the comparison , and `value`
specifies the value to compare against.

> To use `compare`, `set` must true. The comparison will be ignored if `set` is
> false

The `op` (operations) currently supported in `kube-bench` are:
- `eq`: tests if the keyword is equal to the compared value.
- `noteq`: tests if the keyword is unequal to the compared value.
- `gt`: tests if the keyword is greater than the compared value.
- `gte`: tests if the keyword is greater than or equal to the compared value.
- `lt`: tests if the keyword is less than the compared value.
- `lte`: tests if the keyword is less than or equal to the compared value.
- `has`: tests if the keyword contains the compared value.
- `nothave`: tests if the keyword does not contain the compared value.

## Configuration and Variables

Kubernetes component configuration and binary file locations and names 
vary based on cluster deployment methods and kubernetes distribution used.
For this reason, the locations of these binaries and config files are configurable
by editing the `cfg/config.yaml` file and these binaries and files can be
referenced in a `controls` file via variables.

The `cfg/config.yaml` file is a global configuration file. Configuration files
can be created for specific Kubernetes versions (distributions). Values in the
version specific config overwrite similar values in `cfg/config.yaml`.

For example, the kube-apiserver in Redhat OCP distribution is run as 
`hypershift openshift-kube-apiserver` instead of the default `kube-apiserver`.
This difference can be specified by editing the `master.apiserver.defaultbin`
entry `cfg/ocp-3.10/config.yaml`.

Below is the structure of `cfg/config.yaml`:

```
nodetype
  |-- components
    |-- component1
  |-- component1
    |-- bins
    |-- defaultbin (optional)
    |-- confs
    |-- defaultconf (optional)
    |-- svcs
    |-- defaultsvc (optional)
    |-- kubeconfig
    |-- defaultkubeconfig (optional)
```

Every node type has a subsection that specifies the main configurations items.

- `components`: A list of components for the node type. For example master 
  will have an entry for **apiserver**, **scheduler** and **controllermanager**.
  
  Each component has the following entries:

- `bins`: A list of candidate binaries for a component. `kube-bench` checks this
   list and selects the first binary that is running on the node, if none is
   running, `kube-bench` terminates.

   If `defaultbin` is specified, `kube-bench` ignores the `bins` list (if it is
   specified) and verifies the binary specified with `defaultbin` is running on
   the node. `kube-bench` terminates if this binary is not running.
   
   The selected binary for a component can be referenced in `controls` using a 
   variable in the form `$<component>bin`. In the example below, we reference 
   the selected API server binary with the variable `$apiserverbin` in an `audit`
   command.
   
   ```
   id: 1.1.1
    text: "Ensure that the --anonymous-auth argument is set to false (Scored)"
    audit: "ps -ef | grep $apiserverbin | grep -v grep"
    ...
   ```
   
- `confs`: A list of candidate configuration files for a component. `kube-bench`
  checks this list and selects the first config fille that is found on the node,
  if none of the config files exists `kube-bench` terminates.
  
  If `defaultconf`is specified for a component, `kube-bench` ignores the `confs`
  list (if it is specified) and verifies the config specified by `defaultconf`
  exists on the node. `kube-bench` terminates if this file does not exist.
  
  The selected config for a component can be referenced in `controls` using a
  variable in the form `$<component>conf`. In the example below we reference the 
  selected API server config file with the variable `$apiserverconf` in an `audit`
  command.
  
  ```
  id: 1.4.1
    text: "Ensure that the API server pod specification file permissions are
    set to 644 or more restrictive (Scored)"
    audit: "/bin/sh -c 'if test -e $apiserverconf; then stat -c %a $apiserverconf; fi'"

  ```
  
- `svcs`:  A list of candidates unitfiles for a component. `kube-bench` checks this 
  list and selects the first unitfile that is found on the node, if none of the
  unitfiles exists `kube-bench` terminates.
  
  If `defaultsvc`is specified for a component, `kube-bench` ignores the `svcs`
  list (if it is specified) and verifies the unitfile specified by `defaultsvc`
  exists on the node. `kube-bench` terminates if this file does not exist.
  
  The selected unitfile for a component can be referenced in `controls` via a
  variable in the form `$<component>svc`. In the example below, the selected 
  kubelet unitfile is referenced with `$kubeletsvc` in the `remediation` of the 
  `check`.
  
  ```
  id: 2.1.1
    ...
    remediation: |
      Edit the kubelet service file $kubeletsvc
      on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.
      --allow-privileged=false
      Based on your system, restart the kubelet service. For example:
      systemctl daemon-reload
      systemctl restart kubelet.service
    ...
  ```
  
  - `kubeconfig`: A list of candidate kubeconfig files for a component. `kube-bench`
    checks this list and selects the first file that is found on the node, if none
    of the files exists `kube-bench` terminates.
    
    If `defaultkubeconfig` is specified for a component, `kube-bench` ignores the
    `kubeconfig` list (if it is specified) and verifies the kubeconfig file exists on
    the node. `kube-bench` terminates if this file does not exist.
    
    The selected kubeconfig for a component can be referenced in `controls` with
    a variable in the form `$<component>kubeconfig`. In the example below, the
    selected kubelet kubeconfig is referenced with `$kubeletkubeconfig` in the
    `audit` command.
    
    ```
    id: 2.2.1
      text: "Ensure that the kubelet.conf file permissions are set to 644 or
      more restrictive (Scored)"
      audit: "/bin/sh -c 'if test -e $kubeletkubeconfig; then stat -c %a $kubeletkubeconfig; fi'"
      ...
    ```