1
0
mirror of https://github.com/aquasecurity/kube-bench.git synced 2024-12-04 22:08:22 +00:00
kube-bench/docs/controls.md

381 lines
14 KiB
Markdown
Raw Normal View History

# Test and config files
`kube-bench` runs checks specified in `controls` files that are a YAML
representation of the CIS Kubernetes Benchmark checks (or other distribution-specific hardening guides).
## Controls
`controls` is a YAML document that contains checks that must be run against a
2019-10-24 08:15:29 +00:00
specific Kubernetes node type, master or node and version.
`controls` is the fundamental input to `kube-bench`. The following is an example
of a basic `controls`:
2019-10-24 08:15:29 +00:00
```yml
---
controls:
id: 1
text: "Master Node Security Configuration"
type: "master"
groups:
- id: 1.1
text: API Server
checks:
- id: 1.1.1
text: "Ensure that the --allow-privileged argument is set (Scored)"
audit: "ps -ef | grep kube-apiserver | grep -v grep"
tests:
bin_op: or
test_items:
- flag: "--allow-privileged"
set: true
- flag: "--some-other-flag"
set: false
remediation: "Edit the /etc/kubernetes/config file on the master node and
set the KUBE_ALLOW_PRIV parameter to '--allow-privileged=false'"
scored: true
- id: 1.2
text: Scheduler
checks:
- id: 1.2.1
text: "Ensure that the --profiling argument is set to false (Scored)"
audit: "ps -ef | grep kube-scheduler | grep -v grep"
tests:
2021-06-09 08:17:16 +00:00
bin_op: and
test_items:
- flag: "--profiling"
set: true
- flag: "--some-other-flag"
set: false
remediation: "Edit the /etc/kubernetes/config file on the master node and
set the KUBE_ALLOW_PRIV parameter to '--allow-privileged=false'"
scored: true
```
2019-10-24 08:15:29 +00:00
`controls` is composed of a hierarchy of groups, sub-groups and checks. Each of
the `controls` components have an id and a text description which are displayed
in the `kube-bench` output.
2019-10-25 18:58:41 +00:00
`type` specifies what kubernetes node type a `controls` is for. Possible values
for `type` are `master` and `node`.
## Groups
`groups` is a list of subgroups that test the various Kubernetes components
that run on the node type specified in the `controls`.
For example, one subgroup checks parameters passed to the API server binary, while
another subgroup checks parameters passed to the controller-manager binary.
2019-10-24 08:15:29 +00:00
```yml
groups:
- id: 1.1
text: API Server
2019-10-24 08:15:29 +00:00
# ...
- id: 1.2
text: Scheduler
2019-10-24 08:15:29 +00:00
# ...
```
These subgroups have `id`, `text` fields which serve the same purposes described
in the previous paragraphs. The most important part of the subgroup is the
`checks` field which is the collection of actual `check`s that form the subgroup.
This is an example of a subgroup and checks in the subgroup.
2019-10-24 08:15:29 +00:00
```yml
id: 1.1
text: API Server
checks:
- id: 1.1.1
text: "Ensure that the --allow-privileged argument is set (Scored)"
audit: "ps -ef | grep kube-apiserver | grep -v grep"
tests:
2019-10-24 08:15:29 +00:00
# ...
- id: 1.1.2
text: "Ensure that the --anonymous-auth argument is set to false (Not Scored)"
audit: "ps -ef | grep kube-apiserver | grep -v grep"
tests:
2019-10-24 08:15:29 +00:00
# ...
```
`kube-bench` supports running a subgroup by specifying the subgroup `id` on the
command line, with the flag `--group` or `-g`.
## Check
The CIS Kubernetes Benchmark recommends configurations to harden Kubernetes components. These recommendations are usually configuration options and can be
2019-10-24 08:15:29 +00:00
specified by flags to Kubernetes binaries, or in configuration files.
2019-10-24 08:15:29 +00:00
The Benchmark also provides commands to audit a Kubernetes installation, identify
places where the cluster security can be improved, and steps to remediate these
identified problems.
In `kube-bench`, `check` objects embody these recommendations. This an example
`check` object:
2019-10-24 08:15:29 +00:00
```yml
id: 1.1.1
text: "Ensure that the --anonymous-auth argument is set to false (Not Scored)"
audit: "ps -ef | grep kube-apiserver | grep -v grep"
tests:
test_items:
- flag: "--anonymous-auth"
compare:
op: eq
value: false
set: true
remediation: |
Edit the API server pod specification file kube-apiserver
on the master node and set the below parameter.
--anonymous-auth=false
scored: false
```
2019-10-24 08:15:29 +00:00
A `check` object has an `id`, a `text`, an `audit`, a `tests`, `remediation`
and `scored` fields.
`kube-bench` supports running individual checks by specifying the check's `id`
as a comma-delimited list on the command line with the `--check` flag.
The `audit` field specifies the command to run for a check. The output of this
command is then evaluated for conformance with the CIS Kubernetes Benchmark
recommendation.
The audit is evaluated against criteria specified by the `tests`
object. `tests` contain `bin_op` and `test_items`.
`test_items` specify the criteria(s) the `audit` command's output should meet to
pass a check. This criteria is made up of keywords extracted from the output of
2019-10-25 18:58:41 +00:00
the `audit` command and operations that compare these keywords against
values expected by the CIS Kubernetes Benchmark.
2021-06-09 08:17:16 +00:00
There are three ways to run and extract keywords from the output of the command used,
| Command | Output var |
|---|---|
| `audit` | `flag` |
| `audit_config` | `path` |
| `audit_env` | `env` |
2021-06-09 08:17:16 +00:00
`flag` is used when the keyword is a command-line flag. The associated `audit` command could
be any binaries available on the system like `ps` command and a `grep` for the binary whose flag we are
checking:
2019-10-24 08:15:29 +00:00
```sh
ps -ef | grep somebinary | grep -v grep
```
Here is an example usage of the `flag` option:
2019-10-24 08:15:29 +00:00
```yml
# ...
audit: "ps -ef | grep kube-apiserver | grep -v grep"
tests:
test_items:
- flag: "--anonymous-auth"
2019-10-24 08:15:29 +00:00
# ...
```
`path` is used when the keyword is an option set in a JSON or YAML config file.
2021-06-09 08:17:16 +00:00
The associated `audit_command` command is usually `cat /path/to/config-yaml-or-json`.
For example:
2019-10-24 08:15:29 +00:00
```yml
# ...
text: "Ensure that the --anonymous-auth argument is set to false (Not Scored)"
audit: "cat /path/to/some/config"
tests:
test_items:
- path: "{.someoption.value}"
2019-10-24 08:15:29 +00:00
# ...
```
Refactor group skip (#783) * Add example IAM policy * Pass RotateKubeletServerCertificate related checks if it's not found (#767) * Allow for environment variables to be checked in tests (#755) * Initial commit for checking environment variables for etcd * Revert config changes * Remove redundant struct data * Fix issues with failing tests * Initial changes based on code review * Add option to disable envTesting + Update docs * Initial tests * Finished testing * Fix broken tests * Add a total summary and always show all tests. (#759) Whether the total summary is shown can be specified with an option. Fixes #528 Signed-off-by: Christian Zunker <christian.zunker@codecentric.cloud> * Update Readme.md file with link to Contribution guide (#754) * Update License with the year and the owner name Please add this to make your license agreement strong * Updated Readme.md file with license and proper documentation links I have added a proper license agreement to the documentation. Also shortened the links to the issues so that it does not break in any on the forks. * Update LICENSE * Update README.md * Update README.md * Remove erroneous license info Co-authored-by: Liz Rice <liz@lizrice.com> * Support auto-detect platform when running on EKS or GKE (#683) * Support auto-detect platform when running on EKS or GKE * Change to get platform name from `kubectl version` * fix regexp and add test * Update Server Version match for EKS * try to get version info from api sever at first * Refactor group skip changed group 'skip' from being a bool to be 'type' string as done in check * Change skip: true -> type: skip Co-authored-by: Huang Huang <mozillazg101@gmail.com> Co-authored-by: Wicked <jason_attwood@hotmail.co.uk> Co-authored-by: Christian Zunker <827818+czunker@users.noreply.github.com> Co-authored-by: Kaiwalya Koparkar <kaiwalyakoparkar@gmail.com> Co-authored-by: Yoav Rotem <yoavrotems97@gmail.com>
2020-12-21 11:18:54 +00:00
`env` is used to check if the value is present within a specified environment variable. The presence of `env` is treated as an OR operation, if both `flag` and `env` are supplied it will use either to attempt pass the check.
The command used for checking the environment variables of a process **is generated by default**.
2021-06-09 08:17:16 +00:00
If the command being generated is causing errors, you can override the command used by setting `audit_env` on the check.
Refactor group skip (#783) * Add example IAM policy * Pass RotateKubeletServerCertificate related checks if it's not found (#767) * Allow for environment variables to be checked in tests (#755) * Initial commit for checking environment variables for etcd * Revert config changes * Remove redundant struct data * Fix issues with failing tests * Initial changes based on code review * Add option to disable envTesting + Update docs * Initial tests * Finished testing * Fix broken tests * Add a total summary and always show all tests. (#759) Whether the total summary is shown can be specified with an option. Fixes #528 Signed-off-by: Christian Zunker <christian.zunker@codecentric.cloud> * Update Readme.md file with link to Contribution guide (#754) * Update License with the year and the owner name Please add this to make your license agreement strong * Updated Readme.md file with license and proper documentation links I have added a proper license agreement to the documentation. Also shortened the links to the issues so that it does not break in any on the forks. * Update LICENSE * Update README.md * Update README.md * Remove erroneous license info Co-authored-by: Liz Rice <liz@lizrice.com> * Support auto-detect platform when running on EKS or GKE (#683) * Support auto-detect platform when running on EKS or GKE * Change to get platform name from `kubectl version` * fix regexp and add test * Update Server Version match for EKS * try to get version info from api sever at first * Refactor group skip changed group 'skip' from being a bool to be 'type' string as done in check * Change skip: true -> type: skip Co-authored-by: Huang Huang <mozillazg101@gmail.com> Co-authored-by: Wicked <jason_attwood@hotmail.co.uk> Co-authored-by: Christian Zunker <827818+czunker@users.noreply.github.com> Co-authored-by: Kaiwalya Koparkar <kaiwalyakoparkar@gmail.com> Co-authored-by: Yoav Rotem <yoavrotems97@gmail.com>
2020-12-21 11:18:54 +00:00
Similarly, if you don't want the environment checking command to be generated or run at all, specify `disableEnvTesting` as true on the check.
The example below will check if the flag `--auto-tls` is equal to false *OR* `ETCD_AUTO_TLS` is equal to false
```yml
test_items:
- flag: "--auto-tls"
env: "ETCD_AUTO_TLS"
compare:
op: eq
value: false
```
2021-06-09 08:17:16 +00:00
**Note:** flag, path and env will act as OR if more then one present.
Refactor group skip (#783) * Add example IAM policy * Pass RotateKubeletServerCertificate related checks if it's not found (#767) * Allow for environment variables to be checked in tests (#755) * Initial commit for checking environment variables for etcd * Revert config changes * Remove redundant struct data * Fix issues with failing tests * Initial changes based on code review * Add option to disable envTesting + Update docs * Initial tests * Finished testing * Fix broken tests * Add a total summary and always show all tests. (#759) Whether the total summary is shown can be specified with an option. Fixes #528 Signed-off-by: Christian Zunker <christian.zunker@codecentric.cloud> * Update Readme.md file with link to Contribution guide (#754) * Update License with the year and the owner name Please add this to make your license agreement strong * Updated Readme.md file with license and proper documentation links I have added a proper license agreement to the documentation. Also shortened the links to the issues so that it does not break in any on the forks. * Update LICENSE * Update README.md * Update README.md * Remove erroneous license info Co-authored-by: Liz Rice <liz@lizrice.com> * Support auto-detect platform when running on EKS or GKE (#683) * Support auto-detect platform when running on EKS or GKE * Change to get platform name from `kubectl version` * fix regexp and add test * Update Server Version match for EKS * try to get version info from api sever at first * Refactor group skip changed group 'skip' from being a bool to be 'type' string as done in check * Change skip: true -> type: skip Co-authored-by: Huang Huang <mozillazg101@gmail.com> Co-authored-by: Wicked <jason_attwood@hotmail.co.uk> Co-authored-by: Christian Zunker <827818+czunker@users.noreply.github.com> Co-authored-by: Kaiwalya Koparkar <kaiwalyakoparkar@gmail.com> Co-authored-by: Yoav Rotem <yoavrotems97@gmail.com>
2020-12-21 11:18:54 +00:00
`test_item` compares the output of the audit command and keywords using the
`set` and `compare` fields.
2019-10-24 08:15:29 +00:00
```yml
test_items:
- flag: "--anonymous-auth"
compare:
op: eq
value: false
set: true
```
`set` checks if a keyword is present in the output of the audit command or a config file. The possible values for `set` are true and false.
If `set` is true, the check passes only if the keyword is present in the output
of the audit command, or config file. If `set` is false, the check passes only
if the keyword is not present in the output of the audit command, or config file.
2021-06-09 08:17:16 +00:00
`set` is true by default.
`compare` has two fields `op` and `value` to compare keywords with expected
2019-10-25 18:58:41 +00:00
value. `op` specifies which operation is used for the comparison, and `value`
specifies the value to compare against.
> To use `compare`, `set` must true. The comparison will be ignored if `set` is
> false
The `op` (operations) currently supported in `kube-bench` are:
- `eq`: tests if the keyword is equal to the compared value.
- `noteq`: tests if the keyword is unequal to the compared value.
- `gt`: tests if the keyword is greater than the compared value.
- `gte`: tests if the keyword is greater than or equal to the compared value.
- `lt`: tests if the keyword is less than the compared value.
- `lte`: tests if the keyword is less than or equal to the compared value.
- `has`: tests if the keyword contains the compared value.
- `nothave`: tests if the keyword does not contain the compared value.
- `regex`: tests if the flag value matches the compared value regular expression.
When defining regular expressions in YAML it is generally easier to wrap them in
single quotes, for example `'^[abc]$'`, to avoid issues with string escaping.
2021-06-09 08:17:16 +00:00
- `bitmask` : tests if keyward is bitmasked with the compared value, common usege is for
comparing file permissions in linux.
## Omitting checks
If you decide that a recommendation is not appropriate for your environment, you can choose to omit it by editing the test YAML file to give it the check type `skip` as in this example:
```yaml
checks:
- id: 2.1.1
text: "Ensure that the --allow-privileged argument is set to false (Scored)"
type: "skip"
scored: true
```
No tests will be run for this check and the output will be marked [INFO].
## Configuration and Variables
Kubernetes component configuration and binary file locations and names
2019-10-24 08:15:29 +00:00
vary based on cluster deployment methods and Kubernetes distribution used.
For this reason, the locations of these binaries and config files are configurable
by editing the `cfg/config.yaml` file and these binaries and files can be
referenced in a `controls` file via variables.
The `cfg/config.yaml` file is a global configuration file. Configuration files
can be created for specific Kubernetes versions (distributions). Values in the
version-specific config overwrite similar values in `cfg/config.yaml`.
2019-10-24 08:15:29 +00:00
For example, the kube-apiserver in Red Hat OCP distribution is run as
`hypershift openshift-kube-apiserver` instead of the default `kube-apiserver`.
This difference can be specified by editing the `master.apiserver.defaultbin`
2021-06-09 08:17:16 +00:00
entry `cfg/rh-0.7/config.yaml`.
Below is the structure of `cfg/config.yaml`:
```
nodetype
|-- components
|-- component1
|-- component1
|-- bins
|-- defaultbin (optional)
|-- confs
|-- defaultconf (optional)
|-- svcs
|-- defaultsvc (optional)
|-- kubeconfig
|-- defaultkubeconfig (optional)
```
2019-10-25 18:58:41 +00:00
Every node type has a subsection that specifies the main configuration items.
- `components`: A list of components for the node type. For example master
will have an entry for **apiserver**, **scheduler** and **controllermanager**.
Each component has the following entries:
- `bins`: A list of candidate binaries for a component. `kube-bench` checks this
2021-06-09 08:17:16 +00:00
list and selects the **first** binary that is running on the node.
2019-10-24 08:15:29 +00:00
If none of the binaries in `bins` list is running, `kube-bench` checks if the
binary specified by `defaultbin` is running and terminates if none of the
binaries in both `bins` and `defaultbin` is running.
The selected binary for a component can be referenced in `controls` using a
variable in the form `$<component>bin`. In the example below, we reference
the selected API server binary with the variable `$apiserverbin` in an `audit`
command.
2019-10-24 08:15:29 +00:00
```yml
id: 1.1.1
text: "Ensure that the --anonymous-auth argument is set to false (Scored)"
audit: "ps -ef | grep $apiserverbin | grep -v grep"
2019-10-24 08:15:29 +00:00
# ...
```
- `confs`: A list of candidate configuration files for a component. `kube-bench`
2021-06-09 08:17:16 +00:00
checks this list and selects the **first** config file that is found on the node.
2019-10-25 18:58:41 +00:00
If none of the config files exists, `kube-bench` defaults conf to the value
of `defaultconf`.
The selected config for a component can be referenced in `controls` using a
variable in the form `$<component>conf`. In the example below, we reference the
selected API server config file with the variable `$apiserverconf` in an `audit`
command.
2019-10-24 08:15:29 +00:00
```yml
id: 1.4.1
text: "Ensure that the API server pod specification file permissions are
set to 644 or more restrictive (Scored)"
audit: "/bin/sh -c 'if test -e $apiserverconf; then stat -c %a $apiserverconf; fi'"
```
2019-10-25 18:58:41 +00:00
- `svcs`: A list of candidate unitfiles for a component. `kube-bench` checks this
2021-06-09 08:17:16 +00:00
list and selects the **first** unitfile that is found on the node. If none of the
unitfiles exists, `kube-bench` defaults unitfile to the value of `defaultsvc`.
The selected unitfile for a component can be referenced in `controls` via a
variable in the form `$<component>svc`. In the example below, the selected
kubelet unitfile is referenced with `$kubeletsvc` in the `remediation` of the
`check`.
2019-10-24 08:15:29 +00:00
```yml
id: 2.1.1
2019-10-24 08:15:29 +00:00
# ...
remediation: |
Edit the kubelet service file $kubeletsvc
on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.
--allow-privileged=false
Based on your system, restart the kubelet service. For example:
systemctl daemon-reload
systemctl restart kubelet.service
2019-10-24 08:15:29 +00:00
# ...
```
- `kubeconfig`: A list of candidate kubeconfig files for a component. `kube-bench`
2021-06-09 08:17:16 +00:00
checks this list and selects the **first** file that is found on the node. If none
of the files exists, `kube-bench` defaults kubeconfig to the value of
`defaultkubeconfig`.
The selected kubeconfig for a component can be referenced in `controls` with a variable in the form `$<component>kubeconfig`. In the example below, the
selected kubelet kubeconfig is referenced with `$kubeletkubeconfig` in the
`audit` command.
2019-10-24 08:15:29 +00:00
```yml
id: 2.2.1
text: "Ensure that the kubelet.conf file permissions are set to 644 or
more restrictive (Scored)"
audit: "/bin/sh -c 'if test -e $kubeletkubeconfig; then stat -c %a $kubeletkubeconfig; fi'"
2019-10-24 08:15:29 +00:00
# ...
```