Kubernetes Advanced Concepts: Workloads, Security, Scheduling, and Extensibility: Part 2

If you have not read the Part 1 for this post then make sure you go and read that first as this part 2 is going to be much more heavy and technical on different types of things that can be configured and run in kubernetes

Basics

So you have already learnt about pods, deployments and services in part 1, if not you should go ahead and check it out.

These basic building blocks allow you to understand what Kubernetes works with at a fundamental level, what I will talk about are more basic components that are still important but you may not work with them as often or they are incorporated together with other services as they are more of a configuration in the basic components

Namespaces: A namespace is an isolated environment in a single cluster, you can create namespaces for specific sections like your media in 1 namespace and your ai is in another, this allows you to be able to isolate everything and can have multiple people working on the same cluster but never seeing anything outside of the namespaces they have access to. Super useful when deploying stuff as I can group different things in different namespaces making it easier to find everything.

ReplicaSets: So when I talked about having multiple pods for a deployment, where you desire 2 or 3 or more running instances of pods, these are what is used, you normally identify this in the yaml file when working with deployments. If you are just building a homelab and you may have maybe more nodes or such you can have maybe 2 or 3 but you won't need any more than that.

StatefulSets: These are just like deployments, but the pods here matter in which order and what configuration because the difference between both is running stateful or stateless application

Stateless applications are like web server or a stateless backend service, you do not need a persistent storage and the pods connecting to a single storage or interchanging and being deleted don't really matter but it does for stateful where things like a database can't just be connecting to any volume randomly

So statefulsets make sure to maintain the identity of each pod, for example in a usecase of running mysql with replica's

DaemonSets: This forces each node you have to have a copy of the node you need to run so it will force each node to run the pod that you want, this is for things like log collectors or a gpu plugin etc, things that affect every node

Jobs and CronJobs: A job is one or more pods that just runs once, you want to run a backup or a clean up, you don't need it to keep running so you use a job for that, cronjob is very similar to crontab in unix based systems where you can schedule a job to either run at a specific time or every certain hours or such

Configuration and Secrets:

ConfigMaps: Think of a ConfigMap as a way to store configuration data separately from your containerized applications in Kubernetes. Instead of hardcoding settings or environment variables inside your app images, ConfigMaps let you manage these configurations independently. This means if you need to tweak something like, an API endpoint, you don’t have to rebuild your app, just update the ConfigMap and reload or restart your pods. For example if you need to add a specific Jellyfin plugin which changes code in your pod, this will apply it without affecting the actual image

Secrets: Think of a Secret as a Vault and a way in Kubernetes to safely store sensitive data like passwords or API keys outside your container images. The data inside a Secret is encoded in base64 before being stored, which helps with safe handling though it’s not encryption. This means you can update or manage sensitive information without rebuilding your app or exposing it openly. Like if your app needs OpenAI api key, you don't want to give it to the app directly

Networking

Ingress: Manages external HTTP and HTTPS access to services inside your cluster. It lets you route traffic by hostname or path, handle SSL termination, and provide virtual hosting. For it to work, you need an Ingress Controller that does the actual load balancing and routing. You can use things like traefik or nginx reverse proxy. These are super important if you want people to access your service without having to use a VPN

Network Policies: NetworkPolicies act like rules that control which pods can talk to each other inside your Kubernetes cluster. By setting these rules, you limit network access so only the right pods can communicate, improving security. It helps you create isolated segments within your cluster, making it harder for issues to spread across apps. Example if you want your database to not talk to anything other than a specific app, these are policies that can be set

Endpoint slices: These keep track of the network endpoints for services more efficiently than the older Endpoints system. Endpoint is where the traffic will go, this is so if the pod restarts, name changes or there are more than 1, this remains the same. They split large sets of endpoints into smaller slices so Kubernetes can handle scale better, especially in big clusters with many pods.

Storage

Persistent Volume and Persistent Volume Claim: A Persistent Volume (PV) is a durable piece of storage in your Kubernetes cluster, like a network share or disk that outlives Pods, provisioned manually or dynamically via a Storage Class with details on size, access modes (e.g., ReadWriteOnce) like how many pods can use 1 PV , and type. A Persistent Volume Claim (PVC) acts as your app's request for that storage, specifying needs like space and rules, which Kubernetes binds to a matching PV for exclusive use until released. Together, they ensure data persistence;

A great example to understand is think of PV as a Parking Spot, and your PVC is your claim to the spot, only then you can use the spot.

Storage Class: A Storage Class defines the "flavor" of storage available in your cluster, specifying things like the provisioner (e.g., NFS or ISCSI), performance characteristics, and reclamation policies, allowing dynamic PV creation without manual setup. It acts as a template for provisioning storage on demand when a PVC is created. In a homelab, you could define a Storage Class for your local SSDs using things like Longhorn or Rook Ceph versus NFS shares from your NAS (slower but expandable), then use it to automatically provision storage for all your apps

Volume Snapshots: Volume Snapshots capture point-in-time copies of a PV's data for backups, integrated via CSI drivers (Related to ISCSI) to create restorable images of volumes. They allow quick recovery or cloning without full backups. In a homelab, you need these so that you can store them on a drive or on the cloud in case you break things like me 😜, you will be able to then just rollback to use a stable version

Security and Access Control

ServiceAccounts: ServiceAccounts are special accounts in Kubernetes that provide an identity for processes running inside Pods to interact with the cluster's API server, allowing them to authenticate and perform actions like accessing secrets or querying resources without using human user credentials. They can also be for tools like I have built a Service account just for my Claude so it doesn't have more permissions than it should

Considering it has broken my server because of that before

Roles and ClusterRoles: Roles define a set of permissions for actions on resources within a single namespace, like allowing "get" or "list" on Pods, while ClusterRoles extend this to cluster-wide resources or non-namespaced objects, such as Nodes or PersistentVolumes. They're essentially rule-based policies that specify verbs (e.g., create, delete) and resources without assigning them to anyone yet. You could use a Role for your Service Accounts for them to have specific permissions

RoleBindings and ClusterRoleBindings: RoleBindings attach a Role's permissions to specific users, groups, or ServiceAccounts within a namespace, while ClusterRoleBindings do the same for ClusterRoles at a cluster-wide level, effectively granting the defined access. They link "who" (subjects) to "what" (roles), enabling fine-grained RBAC (Role-Based Access Control). This is how you will assign the roles you create to the Service Account you created.

Scheduling and Node Management

Taints and Tolerations: Taints are properties applied to nodes that repel certain Pods from being scheduled there, while Tolerations are added to Pods to allow them to "tolerate" those taints and still be scheduled. Taints include a key, value, and effect (e.g., NoSchedule to block scheduling, NoExecute to evict non-tolerating Pods), helping manage node specialization or issues like maintenance. Like lets say you have a gpu node where you don't want non gpu pods to run so you can put a taint there and tolerations for the gpu workload

Node Affinity: Similar to taints and tolerations this does the opposite, where you can make pods schedule on the node based on things like hard or soft affinity, making it so it only schedules on that node, and nothing else, or prefer that node, which I do for my jellyfin so it never goes down, just won't have transcoding available

Pod Affinity and Anti-Affinity: This relates to having pods a affinity to other pods, so that they will tend to schedule either on the same node or not on the same node based on what you need. You can use this so that if you have 2 pods running heavy workloads you don't want to have them on the same node, this is super useful then

ResourceQuotas: A quota allows you to limit the cpu, memory, storage count etc, limiting how much can be run in a namespace. This forces no namespace to be hoggin the resources making sure all other things deployed in other namespaces get a fair amount

LimitRanges: LimitRanges define default, minimum, and maximum resource limits (e.g., CPU, memory) for containers in a namespace, automatically applying defaults if not specified and enforcing bounds to prevent over-allocation. They help standardize resource requests and avoid starvation. You will understand the need for this the more you deploy as then a single node cannot handle all the services so distribution is necessary. 

PriorityClasses: PriorityClasses assign a priority value to Pods, influencing scheduling order where higher-priority Pods are scheduled first, with global defaults possible for system-critical workloads. They use a numeric value (higher is more important) to preempt lower-priority Pods if needed. Things like your Ingress or database that should start first, because otherwise your services will just not start

PodDisruptionBudgets (PDB): PodDisruptionBudgets limit the number of Pods in a replicated application that can be voluntarily disrupted (e.g., during node drains or updates) at once, specifying min available or max unavailable to maintain availability. They protect against excessive downtime by controlling evictions. This is so if you are running something that should be on all the time, and you have 3 or 4, you can make it so at least 1 doesn't get disrupted or moved before the others start up to give you High Availability.

Advanced Workload Resources

HorizontalPodAutoscaler (HPA): The HorizontalPodAutoscaler (HPA) automatically scales the number of Pod replicas in a Deployment, ReplicaSet, or StatefulSet based on observed metrics like CPU utilization, memory, or custom metrics, adjusting between a min and max replica count to handle load changes. It works via a control loop that queries metrics (e.g., every 15 seconds) and scales up during high demand or down when load decreases. This is very much similar to AutoScaling in AWS if you are familiar

This is the main method you use to deal with more usage needed, because this is the easiest.

VerticalPodAutoscaler (VPA): The VerticalPodAutoscaler (VPA) automatically adjusts the CPU and memory requests/limits of Pods vertically to match their actual usage, recommending or applying changes to optimize resource allocation without changing the number of replicas. It operates in modes like recommendation (suggesting values) or auto (applying them), helping prevent over- or under-provisioning. This is useful if your node has a lot of resources or if you want to do this for a short period of time because there will always be a cap to how much hardware the node itself can provide

Custom Resources (CR): Custom Resources (CR) are extensions to the Kubernetes API that allow you to define and manage your own resource types beyond built-in ones like Pods or Deployments, created via CustomResourceDefinitions (CRDs) to store structured data and integrate with controllers. They enable domain-specific objects, like a "Database" resource, which can be manipulated with kubectl just like native resources.

Say if you are running CNPG which is cloud native postgres, you create a custom resource called a cluster that creates the cluster you need

Operators: Operators are custom controllers that extend Kubernetes to manage complex, stateful applications using Custom Resources and automation logic, often packaged as a pattern to handle tasks like installation, updates, backups, and scaling for apps like databases or message queues. They combine CRDs with reconciliation loops to ensure the desired state is maintained.

And thats all you need to know regarding kubernetes. This gives you a good idea of the complexity you can bring to a cluster you run, but its much more useful when you start off to have an idea of what kubernetes has that could be useful to you

You can now customise what you build to how you like with all the things present and be sure to share with me what you build on twitter or linkedin or in the comments below