This page shows how to install a custom resource into the Kubernetes API by creating a CustomResourceDefinition.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:
Your Kubernetes server must be at or later than version 1.16. To check the version, enterkubectl version
. If you are using an older version of Kubernetes that is still supported, switch to the documentation for that version to see advice that is relevant for your cluster. When you create a new CustomResourceDefinition (CRD), the Kubernetes API Server creates a new RESTful resource path for each version you specify. The CRD can be either namespaced or cluster-scoped, as specified in the CRD's scope
field. As with existing built-in objects, deleting a namespace deletes all custom objects in that namespace. CustomResourceDefinitions themselves are non-namespaced and are available to all namespaces.
For example, if you save the following CustomResourceDefinition to resourcedefinition.yaml
:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: crontabs.stable.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: stable.example.com
# list of versions supported by this CustomResourceDefinition
versions:
- name: v1
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
image:
type: string
replicas:
type: integer
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: crontabs
# singular name to be used as an alias on the CLI and for display
singular: crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: CronTab
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- ct
and create it:
kubectl apply -f resourcedefinition.yaml
Then a new namespaced RESTful API endpoint is created at:
/apis/stable.example.com/v1/namespaces/*/crontabs/...
This endpoint URL can then be used to create and manage custom objects. The kind
of these objects will be CronTab
from the spec of the CustomResourceDefinition object you created above.
It might take a few seconds for the endpoint to be created. You can watch the Established
condition of your CustomResourceDefinition to be true or watch the discovery information of the API server for your resource to show up.
After the CustomResourceDefinition object has been created, you can create custom objects. Custom objects can contain custom fields. These fields can contain arbitrary JSON. In the following example, the cronSpec
and image
custom fields are set in a custom object of kind CronTab
. The kind CronTab
comes from the spec of the CustomResourceDefinition object you created above.
If you save the following YAML to my-crontab.yaml
:
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
and create it:
kubectl apply -f my-crontab.yaml
You can then manage your CronTab objects using kubectl. For example:
kubectl get crontab
Should print a list like this:
NAME AGE
my-new-cron-object 6s
Resource names are not case-sensitive when using kubectl, and you can use either the singular or plural forms defined in the CRD, as well as any short names.
You can also view the raw YAML data:
kubectl get ct -o yaml
You should see that it contains the custom cronSpec
and image
fields from the YAML you used to create it:
apiVersion: v1
items:
- apiVersion: stable.example.com/v1
kind: CronTab
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{},"name":"my-new-cron-object","namespace":"default"},"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}
creationTimestamp: "2021-06-20T07:35:27Z"
generation: 1
name: my-new-cron-object
namespace: default
resourceVersion: "1326"
uid: 9aab1d66-628e-41bb-a422-57b8b3b1f5a9
spec:
cronSpec: '* * * * */5'
image: my-awesome-cron-image
kind: List
metadata:
resourceVersion: ""
selfLink: ""
When you delete a CustomResourceDefinition, the server will uninstall the RESTful API endpoint and delete all custom objects stored in it.
kubectl delete -f resourcedefinition.yaml
kubectl get crontabs
Error from server (NotFound): Unable to list {"stable.example.com" "v1" "crontabs"}: the server could not find the requested resource (get crontabs.stable.example.com)
If you later recreate the same CustomResourceDefinition, it will start out empty.
CustomResources store structured data in custom fields (alongside the built-in fields apiVersion
, kind
and metadata
, which the API server validates implicitly). With OpenAPI v3.0 validation a schema can be specified, which is validated during creation and updates, compare below for details and limits of such a schema.
With apiextensions.k8s.io/v1
the definition of a structural schema is mandatory for CustomResourceDefinitions. In the beta version of CustomResourceDefinition, the structural schema was optional.
A structural schema is an OpenAPI v3.0 validation schema which:
type
in OpenAPI) for the root, for each specified field of an object node (via properties
or additionalProperties
in OpenAPI) and for each item in an array node (via items
in OpenAPI), with the exception of: x-kubernetes-int-or-string: true
x-kubernetes-preserve-unknown-fields: true
allOf
, anyOf
, oneOf
or not
, the schema also specifies the field/item outside of those logical junctors (compare example 1 and 2).description
, type
, default
, additionalProperties
, nullable
within an allOf
, anyOf
, oneOf
or not
, with the exception of the two pattern for x-kubernetes-int-or-string: true
(see below).metadata
is specified, then only restrictions on metadata.name
and metadata.generateName
are allowed.Non-structural example 1:
allOf:
- properties:
foo:
...
conflicts with rule 2. The following would be correct:
properties:
foo:
...
allOf:
- properties:
foo:
...
Non-structural example 2:
allOf:
- items:
properties:
foo:
...
conflicts with rule 2. The following would be correct:
items:
properties:
foo:
...
allOf:
- items:
properties:
foo:
...
Non-structural example 3:
properties:
foo:
pattern: "abc"
metadata:
type: object
properties:
name:
type: string
pattern: "^a"
finalizers:
type: array
items:
type: string
pattern: "my-finalizer"
anyOf:
- properties:
bar:
type: integer
minimum: 42
required: ["bar"]
description: "foo bar object"
is not a structural schema because of the following violations:
foo
is missing (rule 1).bar
inside of anyOf
is not specified outside (rule 2).bar
's type
is within anyOf
(rule 3).anyOf
(rule 3).metadata.finalizers
might not be restricted (rule 4).In contrast, the following, corresponding schema is structural:
type: object
description: "foo bar object"
properties:
foo:
type: string
pattern: "abc"
bar:
type: integer
metadata:
type: object
properties:
name:
type: string
pattern: "^a"
anyOf:
- properties:
bar:
minimum: 42
required: ["bar"]
Violations of the structural schema rules are reported in the NonStructural
condition in the CustomResourceDefinition.
CustomResourceDefinitions store validated resource data in the cluster's persistence store, etcd. As with native Kubernetes resources such as ConfigMap, if you specify a field that the API server does not recognize, the unknown field is pruned (removed) before being persisted.
CRDs converted from apiextensions.k8s.io/v1beta1
to apiextensions.k8s.io/v1
might lack structural schemas, and spec.preserveUnknownFields
might be true
.
For legacy CustomResourceDefinition objects created as apiextensions.k8s.io/v1beta1
with spec.preserveUnknownFields
set to true
, the following is also true:
For compatibility with apiextensions.k8s.io/v1
, update your custom resource definitions to:
spec.preserveUnknownFields
to false
.If you save the following YAML to my-crontab.yaml
:
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
someRandomField: 42
and create it:
kubectl create --validate=false -f my-crontab.yaml -o yaml
your output is similar to:
apiVersion: stable.example.com/v1
kind: CronTab
metadata:
creationTimestamp: 2017-05-31T12:56:35Z
generation: 1
name: my-new-cron-object
namespace: default
resourceVersion: "285"
uid: 9423255b-4600-11e7-af6a-28d2447dc82b
spec:
cronSpec: '* * * * */5'
image: my-awesome-cron-image
Notice that the field someRandomField
was pruned.
This example turned off client-side validation to demonstrate the API server's behavior, by adding the --validate=false
command line option. Because the OpenAPI validation schemas are also published to clients, kubectl
also checks for unknown fields and rejects those objects well before they would be sent to the API server.
By default, all unspecified fields for a custom resource, across all versions, are pruned. It is possible though to opt-out of that for specifc sub-trees of fields by adding x-kubernetes-preserve-unknown-fields: true
in the structural OpenAPI v3 validation schema. For example:
type: object
properties:
json:
x-kubernetes-preserve-unknown-fields: true
The field json
can store any JSON value, without anything being pruned.
You can also partially specify the permitted JSON; for example:
type: object
properties:
json:
x-kubernetes-preserve-unknown-fields: true
type: object
description: this is arbitrary JSON
With this, only object
type values are allowed.
Pruning is enabled again for each specified property (or additionalProperties
):
type: object
properties:
json:
x-kubernetes-preserve-unknown-fields: true
type: object
properties:
spec:
type: object
properties:
foo:
type: string
bar:
type: string
With this, the value:
json:
spec:
foo: abc
bar: def
something: x
status:
something: x
is pruned to:
json:
spec:
foo: abc
bar: def
status:
something: x
This means that the something
field in the specified spec
object is pruned, but everything outside is not.
Nodes in a schema with x-kubernetes-int-or-string: true
are excluded from rule 1, such that the following is structural:
type: object
properties:
foo:
x-kubernetes-int-or-string: true
Also those nodes are partially excluded from rule 3 in the sense that the following two patterns are allowed (exactly those, without variations in order to additional fields):
x-kubernetes-int-or-string: true
anyOf:
- type: integer
- type: string
...
and
x-kubernetes-int-or-string: true
allOf:
- anyOf:
- type: integer
- type: string
- ... # zero or more
...
With one of those specification, both an integer and a string validate.
In Validation Schema Publishing, x-kubernetes-int-or-string: true
is unfolded to one of the two patterns shown above.
RawExtensions (as in runtime.RawExtension
defined in k8s.io/apimachinery) holds complete Kubernetes objects, i.e. with apiVersion
and kind
fields.
It is possible to specify those embedded objects (both completely without constraints or partially specified) by setting x-kubernetes-embedded-resource: true
. For example:
type: object
properties:
foo:
x-kubernetes-embedded-resource: true
x-kubernetes-preserve-unknown-fields: true
Here, the field foo
holds a complete object, e.g.:
foo:
apiVersion: v1
kind: Pod
spec:
...
Because x-kubernetes-preserve-unknown-fields: true
is specified alongside, nothing is pruned. The use of x-kubernetes-preserve-unknown-fields: true
is optional though.
With x-kubernetes-embedded-resource: true
, the apiVersion
, kind
and metadata
are implicitly specified and validated.
See Custom resource definition versioning for more information about serving multiple versions of your CustomResourceDefinition and migrating your objects from one version to another.
Finalizers allow controllers to implement asynchronous pre-delete hooks. Custom objects support finalizers similar to built-in objects.
You can add a finalizer to a custom object like this:
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
finalizers:
- stable.example.com/finalizer
Identifiers of custom finalizers consist of a domain name, a forward slash and the name of the finalizer. Any controller can add a finalizer to any object's list of finalizers.
The first delete request on an object with finalizers sets a value for the metadata.deletionTimestamp
field but does not delete it. Once this value is set, entries in the finalizers
list can only be removed. While any finalizers remain it is also impossible to force the deletion of an object.
When the metadata.deletionTimestamp
field is set, controllers watching the object execute any finalizers they handle and remove the finalizer from the list after they are done. It is the responsibility of each controller to remove its finalizer from the list.
The value of metadata.deletionGracePeriodSeconds
controls the interval between polling updates.
Once the list of finalizers is empty, meaning all finalizers have been executed, the resource is deleted by Kubernetes.
Custom resources are validated via OpenAPI v3 schemas, by x-kubernetes-validations when the Validation Rules feature is enabled, and you can add additional validation using admission webhooks.
Additionally, the following restrictions are applied to the schema:
definitions
,dependencies
,deprecated
,discriminator
,id
,patternProperties
,readOnly
,writeOnly
,xml
,$ref
.uniqueItems
cannot be set to true
.additionalProperties
cannot be set to false
.additionalProperties
is mutually exclusive with properties
.The x-kubernetes-validations
extension can be used to validate custom resources using Common Expression Language (CEL) expressions when the Validation rules feature is enabled and the CustomResourceDefinition schema is a structural schema.
The default
field can be set when the Defaulting feature is enabled, which is the case with apiextensions.k8s.io/v1
CustomResourceDefinitions. Defaulting is in GA since 1.17 (beta since 1.16 with the CustomResourceDefaulting
feature gate enabled, which is the case automatically for many clusters for beta features).
Refer to the structural schemas section for other restrictions and CustomResourceDefinition features.
The schema is defined in the CustomResourceDefinition. In the following example, the CustomResourceDefinition applies the following validations on the custom object:
spec.cronSpec
must be a string and must be of the form described by the regular expression.spec.replicas
must be an integer and must have a minimum value of 1 and a maximum value of 10.Save the CustomResourceDefinition to resourcedefinition.yaml
:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
schema:
# openAPIV3Schema is the schema for validating custom objects.
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
image:
type: string
replicas:
type: integer
minimum: 1
maximum: 10
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
and create it:
kubectl apply -f resourcedefinition.yaml
A request to create a custom object of kind CronTab is rejected if there are invalid values in its fields. In the following example, the custom object contains fields with invalid values:
spec.cronSpec
does not match the regular expression.spec.replicas
is greater than 10.If you save the following YAML to my-crontab.yaml
:
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * *"
image: my-awesome-cron-image
replicas: 15
and attempt to create it:
kubectl apply -f my-crontab.yaml
then you get an error:
The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
validation failure list:
spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
spec.replicas in body should be less than or equal to 10
If the fields contain valid values, the object creation request is accepted.
Save the following YAML to my-crontab.yaml
:
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
replicas: 5
And create it:
kubectl apply -f my-crontab.yaml
crontab "my-new-cron-object" created
Kubernetes v1.23 [alpha]
Validation rules are in alpha since 1.23 and validate custom resources when the CustomResourceValidationExpressions
feature gate is enabled. This feature is only available if the schema is a structural schema.
Validation rules use the Common Expression Language (CEL) to validate custom resource values. Validation rules are included in CustomResourceDefinition schemas using the x-kubernetes-validations
extension.
The Rule is scoped to the location of the x-kubernetes-validations
extension in the schema. And self
variable in the CEL expression is bound to the scoped value.
For example:
...
openAPIV3Schema:
type: object
properties:
spec:
type: object
x-kubernetes-validation-rules:
- rule: "self.minReplicas <= self.replicas"
message: "replicas should be greater than or equal to minReplicas."
- rule: "self.replicas <= self.maxReplicas"
message: "replicas should be smaller than or equal to maxReplicas."
properties:
...
minReplicas:
type: integer
replicas:
type: integer
maxReplicas:
type: integer
required:
- minReplicas
- replicas
- maxReplicas
will reject a request to create this custom resource:
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
minReplicas: 0
replicas: 20
maxReplicas: 10
with the response:
The CronTab "my-new-cron-object" is invalid:
* spec: Invalid value: map[string]interface {}{"maxReplicas":10, "minReplicas":0, "replicas":20}: replicas should be smaller than or equal to maxReplicas.
x-kubernetes-validations
could have multiple rules.
The rule
under x-kubernetes-validations
represents the expression which will be evaluated by CEL.
The message
represents the message displayed when validation fails. If message is unset, the above response would be:
The CronTab "my-new-cron-object" is invalid:
* spec: Invalid value: map[string]interface {}{"maxReplicas":10, "minReplicas":0, "replicas":20}: failed rule: self.replicas <= self.maxReplicas
Validation rules are compiled when CRDs are created/updated. The request of CRDs create/update will fail if compilation of validation rules fail. Compilation process includes type checking as well.
The compilation failure:
no_matching_overload
: this function has no overload for the types of the arguments.
e.g. Rule like self == true
against a field of integer type will get error:
Invalid value: apiextensions.ValidationRule{Rule:"self == true", Message:""}: compilation failed: ERROR: \<input>:1:6: found no matching overload for '_==_' applied to '(int, bool)'
no_such_field
: does not contain the desired field.
e.g. Rule like self.nonExistingField > 0
against a non-existing field will return the error:
Invalid value: apiextensions.ValidationRule{Rule:"self.nonExistingField > 0", Message:""}: compilation failed: ERROR: \<input>:1:5: undefined field 'nonExistingField'
invalid argument
: invalid argument to macros.
e.g. Rule like has(self)
will return error:
Invalid value: apiextensions.ValidationRule{Rule:"has(self)", Message:""}: compilation failed: ERROR: <input>:1:4: invalid argument to has() macro
Validation Rules Examples:
Rule | Purpose |
---|---|
self.minReplicas <= self.replicas && self.replicas <= self.maxReplicas | Validate that the three fields defining replicas are ordered appropriately |
'Available' in self.stateCounts | Validate that an entry with the 'Available' key exists in a map |
(size(self.list1) == 0) != (size(self.list2) == 0) | Validate that one of two lists is non-empty, but not both |
!('MY_KEY' in self.map1) || self['MY_KEY'].matches('^[a-zA-Z]*$') | Validate the value of a map for a specific key, if it is in the map |
self.envars.filter(e, e.name = 'MY_ENV').all(e, e.value.matches('^[a-zA-Z]*$') | Validate the 'value' field of a listMap entry where key field 'name' is 'MY_ENV' |
has(self.expired) && self.created + self.ttl < self.expired | Validate that 'expired' date is after a 'create' date plus a 'ttl' duration |
self.health.startsWith('ok') | Validate a 'health' string field has the prefix 'ok' |
self.widgets.exists(w, w.key == 'x' && w.foo < 10) | Validate that the 'foo' property of a listMap item with a key 'x' is less than 10 |
type(self) == string ? self == '100%' : self == 1000 | Validate an int-or-string field for both the the int and string cases |
self.metadata.name.startsWith(self.prefix) | Validate that an object's name has the prefix of another field value |
self.set1.all(e, !(e in self.set2)) | Validate that two listSets are disjoint |
size(self.names) == size(self.details) && self.names.all(n, n in self.details) | Validate the 'details' map is keyed by the items in the 'names' listSet |
Xref: Supported evaluation on CEL
If the Rule is scoped to the root of a resource, it may make field selection into any fields declared in the OpenAPIv3 schema of the CRD as well as apiVersion
, kind
, metadata.name
and metadata.generateName
. This includes selection of fields in both the spec
and status
in the same expression:
...
openAPIV3Schema:
type: object
x-kubernetes-validation-rules:
- rule: "self.status.availableReplicas >= self.spec.minReplicas"
properties:
spec:
type: object
properties:
minReplicas:
type: integer
...
status:
type: object
properties:
availableReplicas:
type: integer
If the Rule is scoped to an object with properties, the accessible properties of the object are field selectable via self.field
and field presence can be checked via has(self.field)
. Null valued fields are treated as absent fields in CEL expressions.
...
openAPIV3Schema:
type: object
properties:
spec:
type: object
x-kubernetes-validation-rules:
- rule: "has(self.foo)"
properties:
...
foo:
type: integer
If the Rule is scoped to an object with additionalProperties (i.e. a map) the value of the map are accessible via self[mapKey]
, map containment can be checked via mapKey in self
and all entries of the map are accessible via CEL macros and functions such as self.all(...)
.
...
openAPIV3Schema:
type: object
properties:
spec:
type: object
x-kubernetes-validation-rules:
- rule: "self['xyz'].foo > 0"
additionalProperties:
...
type: object
properties:
foo:
type: integer
If the Rule is scoped to an array, the elements of the array are accessible via self[i]
and also by macros and functions.
...
openAPIV3Schema:
type: object
properties:
...
foo:
type: array
x-kubernetes-validation-rules:
- rule: "size(self) == 1"
items:
type: string
If the Rule is scoped to a scalar, self
is bound to the scalar value.
...
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
...
foo:
type: integer
x-kubernetes-validation-rules:
- rule: "self > 0"
Examples:
type of the field rule scoped to | Rule example |
---|---|
root object | self.status.actual <= self.spec.maxDesired |
map of objects | self.components['Widget'].priority < 10 |
list of integers | self.values.all(value, value >= 0 && value < 100) |
string | self.startsWith('kube') |
The apiVersion
, kind
, metadata.name
and metadata.generateName
are always accessible from the root of the object and from any x-kubernetes-embedded-resource annotated objects. No other metadata properties are accessible.
Unknown data preserved in custom resources via x-kubernetes-preserve-unknown-fields
is not accessible in CEL expressions. This includes:
Only property names of the form [a-zA-Z_.-/][a-zA-Z0-9_.-/]*
are accessible. Accessible property names are escaped according to the following rules when accessed in the expression:
escape sequence | property name equivalent |
---|---|
__underscores__ | __ |
__dot__ | . |
__dash__ | - |
__slash__ | / |
__{keyword}__ | CEL RESERVED keyword |
Note: CEL RESERVED keyword needs to match the exact property name to be escaped (e.g. int in the word sprint would not be escaped).
Examples on escaping:
property name | rule with escaped property name |
---|---|
namespace | self.__namespace__ > 0 |
x-prop | self.x__dash__prop > 0 |
redact__d | self.redact__underscores__d > 0 |
string | self.startsWith('kube') |
Equality on arrays with x-kubernetes-list-type
of set
or map
ignores element order, i.e. [1, 2] == [2, 1]. Concatenation on arrays with x-kubernetes-list-type use the semantics of the list type:
set
: X + Y
performs a union where the array positions of all elements in X
are preserved and non-intersecting elements in Y
are appended, retaining their partial order.map
: X + Y
performs a merge where the array positions of all keys in X
are preserved but the values are overwritten by values in Y
when the key sets of X
and Y
intersect. Elements in Y
with non-intersecting keys are appended, retaining their partial order.Here is the declarations type mapping between OpenAPIv3 and CEL type:
OpenAPIv3 type | CEL type |
---|---|
'object' with Properties | object / "message type" |
'object' with AdditionalProperties | map |
'object' with x-kubernetes-embedded-type | object / "message type", 'apiVersion', 'kind', 'metadata.name' and 'metadata.generateName' are implicitly included in schema |
'object' with x-kubernetes-preserve-unknown-fields | object / "message type", unknown fields are NOT accessible in CEL expression |
x-kubernetes-int-or-string | dynamic object that is either an int or a string, type(value) can be used to check the type |
'array | list |
'array' with x-kubernetes-list-type=map | list with map based Equality & unique key guarantees |
'array' with x-kubernetes-list-type=set | list with set based Equality & unique entry guarantees |
'boolean' | boolean |
'number' (all formats) | double |
'integer' (all formats) | int (64) |
'null' | null_type |
'string' | string |
'string' with format=byte (base64 encoded) | bytes |
'string' with format=date | timestamp (google.protobuf.Timestamp) |
'string' with format=datetime | timestamp (google.protobuf.Timestamp) |
'string' with format=duration | duration (google.protobuf.Duration) |
xref: CEL types, OpenAPI types, Kubernetes Structural Schemas.
apiextensions.k8s.io/v1
. Defaulting allows to specify default values in the OpenAPI v3 validation schema:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
schema:
# openAPIV3Schema is the schema for validating custom objects.
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
default: "5 0 * * *"
image:
type: string
replicas:
type: integer
minimum: 1
maximum: 10
default: 1
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
With this both cronSpec
and replicas
are defaulted:
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
image: my-awesome-cron-image
leads to
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "5 0 * * *"
image: my-awesome-cron-image
replicas: 1
Defaulting happens on the object
Defaults applied when reading data from etcd are not automatically written back to etcd. An update request via the API is required to persist those defaults back into etcd.
Default values must be pruned (with the exception of defaults for metadata
fields) and must validate against a provided schema.
Default values for metadata
fields of x-kubernetes-embedded-resources: true
nodes (or parts of a default value covering metadata
) are not pruned during CustomResourceDefinition creation, but through the pruning step during handling of requests.
New in 1.20: null values for fields that either don't specify the nullable flag, or give it a false
value, will be pruned before defaulting happens. If a default is present, it will be applied. When nullable is true
, null values will be conserved and won't be defaulted.
For example, given the OpenAPI schema below:
type: object
properties:
spec:
type: object
properties:
foo:
type: string
nullable: false
default: "default"
bar:
type: string
nullable: true
baz:
type: string
creating an object with null values for foo
and bar
and baz
spec:
foo: null
bar: null
baz: null
leads to
spec:
foo: "default"
bar: null
with foo
pruned and defaulted because the field is non-nullable, bar
maintaining the null value due to nullable: true
, and baz
pruned because the field is non-nullable and has no default.
CustomResourceDefinition OpenAPI v3 validation schemas which are structural and enable pruning are published as part of the OpenAPI v2 spec from Kubernetes API server.
The kubectl command-line tool consumes the published schema to perform client-side validation (kubectl create
and kubectl apply
), schema explanation (kubectl explain
) on custom resources. The published schema can be consumed for other purposes as well, like client generation or documentation.
The OpenAPI v3 validation schema is converted to OpenAPI v2 schema, and show up in definitions
and paths
fields in the OpenAPI v2 spec.
The following modifications are applied during the conversion to keep backwards compatibility with kubectl in previous 1.13 version. These modifications prevent kubectl from being over-strict and rejecting valid OpenAPI schemas that it doesn't understand. The conversion won't modify the validation schema defined in CRD, and therefore won't affect validation in the API server.
allOf
, anyOf
, oneOf
and not
are removednullable: true
is set, we drop type
, nullable
, items
and properties
because OpenAPI v2 is not able to express nullable. To avoid kubectl to reject good objects, this is necessary.The kubectl tool relies on server-side output formatting. Your cluster's API server decides which columns are shown by the kubectl get
command. You can customize these columns for a CustomResourceDefinition. The following example adds the Spec
, Replicas
, and Age
columns.
Save the CustomResourceDefinition to resourcedefinition.yaml
:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
image:
type: string
replicas:
type: integer
additionalPrinterColumns:
- name: Spec
type: string
description: The cron spec defining the interval a CronJob is run
jsonPath: .spec.cronSpec
- name: Replicas
type: integer
description: The number of jobs launched by the CronJob
jsonPath: .spec.replicas
- name: Age
type: date
jsonPath: .metadata.creationTimestamp
Create the CustomResourceDefinition:
kubectl apply -f resourcedefinition.yaml
Create an instance using the my-crontab.yaml
from the previous section.
Invoke the server-side printing:
kubectl get crontab my-new-cron-object
Notice the NAME
, SPEC
, REPLICAS
, and AGE
columns in the output:
NAME SPEC REPLICAS AGE
my-new-cron-object * * * * * 1 7s
NAME
column is implicit and does not need to be defined in the CustomResourceDefinition. Each column includes a priority
field. Currently, the priority differentiates between columns shown in standard view or wide view (using the -o wide
flag).
0
are shown in standard view.0
are shown only in wide view.A column's type
field can be any of the following (compare OpenAPI v3 data types):
integer
– non-floating-point numbersnumber
– floating point numbersstring
– stringsboolean
– true
or false
date
– rendered differentially as time since this timestamp.If the value inside a CustomResource does not match the type specified for the column, the value is omitted. Use CustomResource validation to ensure that the value types are correct.
A column's format
field can be any of the following:
int32
int64
float
double
byte
date
date-time
password
The column's format
controls the style used when kubectl
prints the value.
Custom resources support /status
and /scale
subresources.
The status and scale subresources can be optionally enabled by defining them in the CustomResourceDefinition.
When the status subresource is enabled, the /status
subresource for the custom resource is exposed.
The status and the spec stanzas are represented by the .status
and .spec
JSONPaths respectively inside of a custom resource.
PUT
requests to the /status
subresource take a custom resource object and ignore changes to anything except the status stanza.
PUT
requests to the /status
subresource only validate the status stanza of the custom resource.
PUT
/POST
/PATCH
requests to the custom resource ignore changes to the status stanza.
The .metadata.generation
value is incremented for all changes, except for changes to .metadata
or .status
.
Only the following constructs are allowed at the root of the CRD OpenAPI validation schema:
description
example
exclusiveMaximum
exclusiveMinimum
externalDocs
format
items
maximum
maxItems
maxLength
minimum
minItems
minLength
multipleOf
pattern
properties
required
title
type
uniqueItems
When the scale subresource is enabled, the /scale
subresource for the custom resource is exposed. The autoscaling/v1.Scale
object is sent as the payload for /scale
.
To enable the scale subresource, the following fields are defined in the CustomResourceDefinition.
specReplicasPath
defines the JSONPath inside of a custom resource that corresponds to scale.spec.replicas
.
.spec
and with the dot notation are allowed.specReplicasPath
in the custom resource, the /scale
subresource will return an error on GET.statusReplicasPath
defines the JSONPath inside of a custom resource that corresponds to scale.status.replicas
.
.status
and with the dot notation are allowed.statusReplicasPath
in the custom resource, the status replica value in the /scale
subresource will default to 0.labelSelectorPath
defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector
.
.status
or .spec
and with the dot notation are allowed.labelSelectorPath
in the custom resource, the status selector value in the /scale
subresource will default to the empty string.In the following example, both status and scale subresources are enabled.
Save the CustomResourceDefinition to resourcedefinition.yaml
:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
image:
type: string
replicas:
type: integer
status:
type: object
properties:
replicas:
type: integer
labelSelector:
type: string
# subresources describes the subresources for custom resources.
subresources:
# status enables the status subresource.
status: {}
# scale enables the scale subresource.
scale:
# specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
specReplicasPath: .spec.replicas
# statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
statusReplicasPath: .status.replicas
# labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
labelSelectorPath: .status.labelSelector
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
And create it:
kubectl apply -f resourcedefinition.yaml
After the CustomResourceDefinition object has been created, you can create custom objects.
If you save the following YAML to my-crontab.yaml
:
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
replicas: 3
and create it:
kubectl apply -f my-crontab.yaml
Then new namespaced RESTful API endpoints are created at:
/apis/stable.example.com/v1/namespaces/*/crontabs/status
and
/apis/stable.example.com/v1/namespaces/*/crontabs/scale
A custom resource can be scaled using the kubectl scale
command. For example, the following command sets .spec.replicas
of the custom resource created above to 5:
kubectl scale --replicas=5 crontabs/my-new-cron-object
crontabs "my-new-cron-object" scaled
kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}'
5
You can use a PodDisruptionBudget to protect custom resources that have the scale subresource enabled.
Categories is a list of grouped resources the custom resource belongs to (eg. all
). You can use kubectl get <category-name>
to list the resources belonging to the category.
The following example adds all
in the list of categories in the CustomResourceDefinition and illustrates how to output the custom resource using kubectl get all
.
Save the following CustomResourceDefinition to resourcedefinition.yaml
:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
image:
type: string
replicas:
type: integer
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
# categories is a list of grouped resources the custom resource belongs to.
categories:
- all
and create it:
kubectl apply -f resourcedefinition.yaml
After the CustomResourceDefinition object has been created, you can create custom objects.
Save the following YAML to my-crontab.yaml
:
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
and create it:
kubectl apply -f my-crontab.yaml
You can specify the category when using kubectl get
:
kubectl get all
and it will include the custom resources of kind CronTab
:
NAME AGE
crontabs/my-new-cron-object 3s
Read about custom resources.
Serve multiple versions of a CustomResourceDefinition.
© 2022 The Kubernetes Authors
Documentation Distributed under CC BY 4.0.
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/