Category Archives: Containers

New Pluralsight Course – Maintaining, Monitoring and Troubleshooting Kubernetes

My new course “Maintaining, Monitoring, and Troubleshooting Kubernetes” is now available on Pluralsight here! Check out the trailer here or if you want to dive right in head over to https://app.pluralsight.com/library/courses/maintaining-monitoring-troubleshooting-kubernetes/!
 
This course will teach you to maintain, monitor, and troubleshoot production Kubernetes clusters.  

This course targets IT professionals that design and maintain Kubernetes and container-based solutions. The course can be used by both the IT pro learning new skills and the system administrator or developer preparing for using Kubernetes both on-premises and in the Cloud. 

This course is part of my Learning Path covering the content needed to prepare for the Certified Kubernetes Administrator exam.

Let’s take your Kubernetes administration and configuration skills to the next level and get you started now!

The modules of the course are:

  • Maintaining Kubernetes Clusters – In this module you will learn core Kubernetes cluster maintenance tasks. We will start off with a closer look at what etcd is, the services it provides, and learn its backup and restore operations. Next, you will then learn the cluster upgrade process, enabling you to take advantage of new Kubernetes features. Then finally, you will learn how to facilitate Worker Node maintenance such as operating system upgrades with draining and cordoning.
  • Logging and Monitoring in Kubernetes Clusters – Monitoring and logging enable you to understand what’s happening inside your Kubernetes cluster and can tell you how things are performing and when things go wrong. In this module we will look at the Kubernetes logging architecture, learning where logs are stored for the Control Plane, Nodes, and Pods and how to access and review those logs. Then next, we’ll dive into how to monitor performance in your cluster with the Kubernetes Metrics Server and access performance data for Nodes and Pods running in your cluster.
  • Troubleshooting Kubernetes – It is inevitable, something will go wrong in your cluster. In this module, you will learn the tools and techniques needed to troubleshoot your Kubernetes cluster. We will start by introducing common troubleshooting methodologies and pain points in Kubernetes. Then you will learn how to debug and fix issues with your cluster, focusing on the control plane and worker nodes.

NewImage

Check out the course at Pluralsight!

New Pluralsight Course – Configuring and Managing Kubernetes Networking, Services, and Ingress

My new course “Configuring and Managing Kubernetes Networking, Services, and Ingress” is now available on Pluralsight here! Check out the trailer here or if you want to dive right in go here!
 
In this course you will learn Kubernetes cluster networking fundamentals and configuring and accessing applications in a Kubernetes Cluster with Services and Ingress.  

This course targets IT professionals that design and maintain Kubernetes and container-based solutions. The course can be used by both the IT pro learning new skills and the system administrator or developer preparing for using Kubernetes both on-premises and in the Cloud. 

Let’s take your Kubernetes administration and configuration skills to the next level and get you started now!

The modules of the course are:

  • Kubernetes Networking Fundamentals – In this module, you will learn Kubernetes networking fundamentals. We will start with the Kubernetes networking model and the motivation behind it, providing developers consistent and robust networking. You will learn cluster network topology, Pod networking internals and how CNI and network plugins implement the Kubernetes network model. Finally, we will learn how DNS is integrated into our cluster and how to configure the DNS Server and Pod DNS clients.
  • Configuring and Managing Application Access with Services Services are the core abstraction to access applications deployed in Kubernetes. In this module, you will learn the motivation for Services and how Services work. You will learn the types of Services available and when to choose which type for your application. We’ll dive deep and look at how Services are implemented in the cluster. You will then learn the key concepts of Service Discovery in a cluster, enabling applications you deploy to work together seamlessly. 
  • Configuring and Managing Application Access with Ingress – In this demo-heavy module you will learn how to expose applications outside of a Kubernetes cluster using Ingress. Starting with the core constructs Ingress and Ingress Controllers. You will learn how traffic flows from outside your cluster through the Ingress controller and to your Pod-based applications. We will learn how to define rules to access applications in several scenarios including single and multi-service access, name-based virtual hosts, and securing access to applications with TLS.

NewImage

Check out the course at Pluralsight!

Speaking at PowerShell Summit 2020!

I’m proud to announce that I will be speaking at PowerShell + DevOps Global Summit 2020 the conference runs from April 27th through April 30. This is an incredible event packed with fantastic content and speakers. Check out the amazing schedule! All the data you need on going is in this excellent brochure right here!

This year I have two sessions!

On Wednesday, April 29th at 09:00AM – I’m presenting “Inside Kubernetes – An Architectural Deep Dive

Here’s the abstract

In this session we will introduce Kubernetes, we’ll deep dive into cluster architecture and higher-level abstractions such as Services, Controllers, and Deployments and how they can be used to ensure the desired state of an application deployed in Kubernetes. In this session we will introduce Kubernetes, we’ll deep dive into each component and its responsibility in a cluster. We will also look at and demonstrate higher-level abstractions such as Services, Controllers, Deployments and Jobs and how they can be used to ensure the desired state of an application deployed in Kubernetes. By the end of this session, you will understand what’s needed to put your applications in production in a Kubernetes cluster

Session Objectives

  • Understand Kubernetes cluster architecture
  • Understand Services, Controllers, and Deployments
  • Designing Production-Ready Kubernetes Clusters
  • Learn to run PowerShell in Kubernetes Jobs.

I look forward to seeing you there.

Speaking at SQLBits 2020

I’m proud to announce that I will be speaking at SQLBits! I had the absolute pleasure of speaking at SQLBits last year for the first time and saw first hand how great this event is and cannot wait to get back and speak again! And this year, I have two sessions!!! One on building and deploying container based applications in Kubernetes and the other on deploying SQL Server in Kubernetes

If you haven’t been to SQLBits before, what are you waiting for! Sign up now!

 

SQL Bits Excel London

Here’s the details for my sessions!

Practical Container Scenarios in Azure – April 2 2020 – 12:40PM

You’ve heard the buzz about containers and Kubernetes, now let’s start your journey towards rapidly deploying and scaling your container-based applications in Azure. In this session, we will introduce containers and the container orchestrator Kubernetes. Then we’ll dive into how to build a container image, push it into our Azure Container Registry and deploy it to our Azure Kubernetes Services cluster. Once deployed, we’ll learn how to keep our applications available and how to scale them using Kubernetes.

Key topics introduced

  • Building a container based application
  • Publishing containers to Azure Container Registry
  • Deploying Azure Kubernetes Services Clusters
  • Scaling our container-based applications in Azure Kubernetes Services

Deploying SQL Server in Kubernetes – April 3 2020  4:50PM

Are you thinking about running SQL Server in Kubernetes and don’t know where to start…are you wondering what you really need to know? If so, then this is the session for you! When deploying SQL Server In Kubernetes key considerations include data persistency, Pod configuration, resource management, and high availability/disaster recovery scenarios. In this session, we’ll look closely at each of these elements and learn how to run SQL Server in Kubernetes.

Learning Objectives

  • Deploying SQL Server in Kubernetes
  • Allocating Persistent Data Storage and configuring advanced disk topologies
  • SQL Server Specific Pod Configuration
  • Near zero-downtime upgrades
  • High availability and Disaster Recovery Scenarios 

Be sure to come to both sessions, learn how to build and deploy containers based applications in Kubernetes and also how to deploy SQL Server in Kubernetes!

Speaking at SQLIntersection Orlando 2020

I’m very pleased to announce that I will be speaking at SQL Intersection April 2020!  This is my first time speaking at SQL Intersection and I’m very excited to be doing so!

Speaking at SQL Intersection means so much to me because in 2014 I got my first exposure to the SQL Server community via SQLskills and their training. Then to follow up on their training workshops I attended my very first IT conference, SQL Intersection and now I get to come back as a speaker. Let’s just say, I’m a little excited!!!

Now as for the sessions…lots of content here on SQL Server on Linux, Containers and Kubernetes…check them out! Click here to register!

Full Day Workshop

Kubernetes Zero to Here: Installation, Configuration and Application Deployment

Modern application deployment needs to be fast and consistent to keep up with business objectives and Kubernetes is quickly becoming the standard for deploying container-based applications, fast. In this day-long session, we will start with an architectural overview of a Kubernetes cluster and how it manages application state. Then we will learn how to build a production-ready cluster. With our cluster up and running, will learn how to interact with our cluster, common administrative tasks, then wrap up with how to deploy applications and SQL Server. At the end of the session, you will know how to set up a Kubernetes cluster, manage a cluster, deploy applications and databases, and how to keep everything up and running.

Workshop Objectives:

  • Introduce Kuberentes Cluster Components
  • Introduce Kubernetes API Objects and Controllers
  • Installing Kubernetes
  • Interacting with your cluster
  • Storing persistent data in Kubernetes
  • Deploying Applications in Kubernetes
  • Deploying SQL Server in Kubernetes
  • High Availability SQL Server scenarios in Kubernetes

General Sessions

Containers – It’s Time to Get on Board

Containers are taking over, changing the way systems are developed and deployed…and that’s not hyperbole. Just imagine if you could deploy SQL Server or even your whole application stack in just minutes? You can do that, leveraging containers! In this session, we’ll get your started on your container journey, learn some common container scenarios and introduce container orchestration with Kubernetes.

In this session we’ll look at

  • Container Fundamentals
  • Common Container Scenarios
  • Running SQL Server in a Container
  • Container Orchestration with Kubernetes

Containers – Continued!

You’ve been working with containers in development for a while, benefiting from the ease and speed of the deployments. Now it’s time to extend your container-based data platform’s capabilities for your production scenarios.
In this session, we’ll look at how to build custom containers, enabling you to craft a container image for your production system’s needs. We’ll also dive deeper into operationalizing your container-based data platform and learn how to provision advanced disk topologies, seed larger databases, implement resource control and understand performance concepts.

By the end of this session, you will learn what it takes to build containers and make them production ready for your environment.

  • Custom container builds with Features
  • Advanced disk configurations
  • Backups/restores
  • Seeding larger databases
  • Backup restore into the container from a mounted volume
  • Resource control
  • Container Restart Policy
  • Container based performance concepts

Linux OS Fundamentals for the SQL Admin

Do you manage SQL Server but have developers using Linux? It’s time to take the leap to understand and communicate better with your Linux peers! You might be a Windows / SQL Server Admin but both SQL Server and PowerShell are now available on Linux. You can manage ALL of these technologies more effectively now. Don’t fear! Linux is just an operating system! While it feels different, it still has all the same components as Windows! In this session, I’ll show you that. We will look at the Linux operating system architecture and show you how to interact with and manage a Linux system. By the end of this session, you’ll be ready to go back to the office and get started working with Linux with a fundamental understanding of how it works.

Monitoring Linux Performance for the SQL Server Admin

Taking what you learned in our Fundamentals session one step further, we will continue and focus on the performance data you’re used to collecting on Windows! We’ll dive into SQLPAL and how the Linux architecture / internals enable high performance for your SQL Server. By the end of this session you’ll be ready to go back to the office and have a solid understanding of performance monitoring Linux systems and SQL on Linux. We’ll look at the core system components of CPU, Disk, Memory, and Networking monitoring techniques for each and look some of the new tools available from DMVs to DBFS.

In this session we’ll cover the following

  • System resource management concepts, CPU, disk, memory and networking
  • Introduce SQLPAL architecture and internals and how its design enables high performance for SQL Server on Linux
  • Baselining and benchmarking 

 

SQLint20 1024x512 NOCENTINO

New Pluralsight Course – Configuring and Managing Kubernetes Storage and Scheduling

My new course “Configuring and Managing Kubernetes Storage and Scheduling” in now available on Pluralsight here! Check out the trailer here or if you want to dive right in go here! This course offers practical tips from my experiences managing Kubernetes Clusters and workloads for Centino Systems clients.

This course targets IT professionals that design and maintain Kubernetes and container based solutions.The course can be used by both the IT pro learning new skills and the system administrator or developer preparing for using Kubernetes both on premises and in the Cloud and is the fourth course in my Kubernetes Administration Learning Path.

Let’s take your Kubernetes administration and configuration skills to the next level and get you started now!

The modules of the course are:

  • Configuring and Managing Storage in Kubernetes – In this module, we will introduce the need for persistent storage in container based applications and then introduce Kubernetes storage objects that provide those services. We’ll dive into the storage lifecycle and how Pods use Persistent Volumes and Persistent Volume Claims to consume storage. We’ll look closely at the types of PVs available and controlling access to PVs with access modes. Once we have the fundamentals down we will learn how to use both Static and Dynamic Provisioning to map Pods to their underlying storage. 
  • Configuration as Data – Environment Variables, Secrets and ConfigMaps – In this demo-heavy module, we’ll look at how to configure Pods using environment variables, secrets and ConfigMaps. We’ll begin with Pod configuration using environment variables and learn how to leverage secrets to securely configure Pod/container based application. Next we’ll see how we can use ConfigMaps to decouple application and Pod configurations in our Pods.
  • Managing and Controlling the Kubernetes Scheduler – In this module we’ll learn, In Kubernetes the Scheduler has the responsibility of sheduling Pods to worker Nodes in the Cluster.  In this module, we will learn how scheduling works and how we can influence the scheduler to help meet application requirements. We will learn how to place Pods on specific nodes (or subsets of nodes) in the cluster, 

NewImage

Check out the course at Pluralsight!

Upgrading SQL Server 2017 Containers to 2019 non-root Containers with Data Volumes – Another Method

Yesterday in this post I described a method to correct permissions when upgrading a SQL Server 2017 container using Data Volumes to 2019’s non-root container on implementations that use the Moby or HyperKit VM. My friend Steve Jones’ on Twitter wondered if you could do this in one step by attaching a shell (bash) in the 2017 container prior to shutdown. Absolutely…let’s walk through that here in this post.  I opted to use an intermediate container in the prior post out of an abundance of caution so that I was not changing permissions on the SQL Server instance directory and all of the data files while they were in use. Technically this is a-ok, but again…just being paranoid there.

Start Up a Container with a Data Volume

Start up a container with a Data Volume (sqldata1) using the 2017 image. This will create the directories and files with root as the owner and group.

docker run \
    --name 'sql1' \
    -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD='$PASSWORD \
    -p 1433:1433 \
    -v sqldata1:/var/opt/mssql \
    -d mcr.microsoft.com/mssql/server:2017-latest
597652b61b22b27ff6d765b48196621a79dd2ffd7798328868d2296c7e953950 

Create a Database

Let’s create a database and confirm it’s there.

sqlcmd -S localhost,1433 -U sa -Q 'CREATE DATABASE TestDB1' -P $PASSWORD
sqlcmd -S localhost,1433 -U sa -Q 'SELECT name from sys.databases' -P $PASSWORD -W

name ---- master tempdb model msdb TestDB1 (5 rows affected) 

Get a Shell into the Container

Now, let’s get a shell into our running container. Logging in as root is great, isn’t it? :) 

docker exec -it sql1 /bin/bash
root@ed9051c6b5f3:/# 

Adjust the Permissions

Now while we’re in the running 2017 container we can adjust the permissions on the instance directory. The user mssql (uid 10001) doesn’t have to exist in the 2017 container. The key to the permissions is using the uid directly.

ls -laR /var/opt/mssql
chgrp -R 0 /var/opt/mssql
chmod -R g=u /var/opt/mssql
chown -R 10001:0 /var/opt/mssql
ls -laR /var/opt/mssql
exit

Stop our Container

Now to start the process of upgrading from 2017 to 2019, we’ll stop and remove the existing container.

docker stop sql1
docker rm sql1
sql1 

Start up a 2019 non-root Container

docker run \
    --name 'sql1' \
    -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD='$PASSWORD \
    -p 1433:1433 \
    -v sqldata1:/var/opt/mssql \
    -d mcr.microsoft.com/mssql/server:2019-GDR1-ubuntu-16.04 

Is Everything OK?

Are our database there? Yep! 

sqlcmd -S localhost,1433 -U sa -Q 'SELECT name from sys.databases' -P $PASSWORD
name
----
master
tempdb
model
msdb
TestDB1

(5 rows affected)

 

 

Upgrading SQL Server 2017 Containers to 2019 non-root Containers with Data Volumes

Recently Microsoft released a Non-Root SQL Server 2019 container and that’s the default if you’re pulling a new container image. But what if you’re using a 2017 container running as root and want to upgrade your system the SQL Server 2019 container…well something’s going to break. As you can see here, my friend Grant Fritchey came across this issue recently and asked for some help on Twitter’s #sqlhelp. This article describe a solution to getting things sorted and running again. The scenario below is if you’re using a Linux based SQL Server container on Windows or Mac host where the container volumes are backed by a Docker Moby or HyperKit virtual machine. If you’re using Linux container on Linux, you’ll adjust the file system permissions directly.

What’s the issue?

When you start up the 2017 container, the SQL Server (sqlservr) process is running as root (uid 0). Any files created by this process will have the user and group ownership of the root user. Now when we come along later and start up a 2019 container, the sqlservr process is running as the user msssql (uid 10001 by default). This new user doesn’t have permission to open the database files and other files used by SQL Server.

How do we fix this?

The way I fixed this issue is by stopping the SQL Server 2017 container and using another container, attaching the data volumes used by the 2017 container into this container then recursively adjusting the permissions to allow a user with the uid 10001 access to the files in the instance directory /var/opt/mssql. If you’re databases and log files are in other paths you’ll have to take that into account if using this process. Once we adjust the permissions, stop that ubuntu container and start up SQL Server’s 2019 non-root container and everything should be happy happy. Let’s do it together…

Start Up a Container with a Data Volume

Start up a container with a Data Volume (sqldata1) using the 2017 image. This will create the files with root as the owner and group.

docker run \
    --name 'sql1' \
    -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD='$PASSWORD \
    -p 1433:1433 \
    -v sqldata1:/var/opt/mssql \
    -d mcr.microsoft.com/mssql/server:2017-latest
597652b61b22b27ff6d765b48196621a79dd2ffd7798328868d2296c7e953950 

Create a Database

Let’s create a database and confirm it’s there.

sqlcmd -S localhost,1433 -U sa -Q 'CREATE DATABASE TestDB1' -P $PASSWORD
sqlcmd -S localhost,1433 -U sa -Q 'SELECT name from sys.databases' -P $PASSWORD -W

name ---- master tempdb model msdb TestDB1 (5 rows affected)  

Stop our Container

Now to start the process of upgrading from 2017 to 2019, we’ll stop and remove the existing container.

docker stop sql1
docker rm sql1
sql1 

Start a 2019 non-root Container

Create a new container pointing to that existing Data Volume (sqldata1), this time I’m not using -d so we can attach to stdout and see the error messages on the terminal. Here you can see that the sqlservr process is unable to open a file instance_id.

docker run \
    --name 'sql1' \
    -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD='$PASSWORD \
    -p 1433:1433 \
    -v sqldata1:/var/opt/mssql \
     mcr.microsoft.com/mssql/server:2019-GDR1-ubuntu-16.04

SQL Server 2019 will run as non-root by default. This container is running as user mssql. Your master database file is owned by root. To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216. sqlservr: Unable to open /var/opt/mssql/.system/instance_id: Permission denied (13) /opt/mssql/bin/sqlservr: Unable to open /var/opt/mssql/.system//instance_id: Permission denied (13) 

Since that was a bust, let’s go ahead and delete that container since it’s not usable. 

docker rm sql1
sql1 

Changing Permissions on the Files

Let’s create an intermediate container, in this case using an Ubuntu image, and mount that data volume (sqldata1), and then change the permissions on the files SQL Server needs to work with. 

docker run \
    --name 'permissionsarehard' \
    -v sqldata1:/var/opt/mssql \
    -it ubuntu:latest

If we look at the permissions of the instance directory (/var/opt/mssql/) we can see the files user and group owner are root. This is just a peek at the instance directory, we’ll need to adjust permissions on all of the file SQL Server needs to work with and recursively within this directory.

ls -la /var/opt/mssql
/var/opt/mssql:
total 24
drwxr-xr-x 6 root root 4096 Nov 20 13:43 .
drwxr-xr-x 1 root root 4096 Nov 20 13:46 ..
drwxr-xr-x 5 root root 4096 Nov 20 13:43 .system
drwxr-xr-x 2 root root 4096 Nov 20 13:43 data
drwxr-xr-x 2 root root 4096 Nov 20 13:43 log
drwxr-xr-x 2 root root 4096 Nov 20 13:43 secrets

Let’s adjust the permissions on the directories and files sqlservr needs access to…again I want to point out, that this is against the default instance directory which is /var/opt/mssql…if you have files in other locations they will need their permissions updated too. Check out the Microsoft Docs article here for more information on this.

ls -laR /var/opt/mssql
chgrp -R 0 /var/opt/mssql
chmod -R g=u /var/opt/mssql
chown -R 10001:0 /var/opt/mssql
ls -laR /var/opt/mssql
exit

Here’s some output from a directory listing of our instance directory after we’ve made the permissions changed…now they have the owner of 10001 and a group owner of root.

ls -la /var/opt/mssql
/var/opt/mssql:
total 24
drwxrwxr-x 6 10001 root 4096 Nov 20 13:43 .
drwxr-xr-x 1 root  root 4096 Nov 20 13:46 ..
drwxrwxr-x 5 10001 root 4096 Nov 20 13:43 .system
drwxrwxr-x 2 10001 root 4096 Nov 20 13:43 data
drwxrwxr-x 2 10001 root 4096 Nov 20 13:43 log
drwxrwxr-x 2 10001 root 4096 Nov 20 13:43 secrets

Let’s start up a 2019 non-root container now

Start up our 2019 container now…should work eh? Woot!

docker run \
    --name 'sql1' \
    -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD='$PASSWORD \
    -p 1433:1433 \
    -v sqldata1:/var/opt/mssql \
    -d mcr.microsoft.com/mssql/server:2019-GDR1-ubuntu-16.04 

Why UID 10001?

Let’s hop into the container now that it’s up and running…and we’ll see sqlservr is running as mssql which has a uid of 10001. This is the default uid used inside non-root container. If you’re using a system that doesn’t have this user defined, like the intermediate ubuntu container, you’ll need to adjust permissions using the uid directly. That permission information is written into the directory and files and when we start up the 2019 container again the correct permissions are in place since the uid of the mssql user matches the uid of the permissions on the files and directories.

docker exec -it sql1 /bin/bash

ps -aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND mssql 1 8.4 0.3 148820 22768 ? Ssl 13:49 0:00 /opt/mssql/bin/ mssql 9 96.5 9.3 7470104 570680 ? Sl 13:49 0:03 /opt/mssql/bin/ mssql 140 2.0 0.0 18220 3060 pts/0 Ss 13:49 0:00 /bin/bash mssql 148 0.0 0.0 34420 2792 pts/0 R+ 13:49 0:00 ps -aux
id mssql uid=10001(mssql) gid=0(root) groups=0(root)
exit

Is Everything OK?

Are our database there? Yep! 

sqlcmd -S localhost,1433 -U sa -Q 'SELECT name from sys.databases' -P $PASSWORD
name
----
master
tempdb
model
msdb
TestDB1

(5 rows affected)

Another Method

If you like living on the edge you can correct the permissions logging into the running 2017 container prior to shutdown and not using an intermediate container, check out this post here

 

 

Speaking at PASS Summit 2019!

I’m very pleased to announce that I will be speaking at PASS Summit 2019!  This is my second time speaking at PASS Summit and I’m very excited to be doing so! What’s more, is I get to help blaze new ground with an emerging technology, Kubernetes and how to run SQL Server in Kubernetes!

My session is Inside Kubernetes – An Architectural Deep Dive if you’re a just getting started in the container space and want to learn how Kubernetes works and dive into how to deploy SQL Server in Kubernetes this is the session for you. I hope to see you there!

Inside Kubernetes – An Architectural Deep Dive

Abstract

In this session we will introduce Kubernetes, we’ll deep dive into each component and its responsibility in a cluster. We will also look at and demonstrate higher-level abstractions such as Services, Controllers, and Deployments, and how they can be used to ensure the desired state of an application and data platform deployed in Kubernetes. Next, we’ll look at Kubernetes networking and intercluster communication patterns. With that foundation, we will then introduce various cluster scenarios and high availability designs. By the end of this session, you will understand what’s needed to put your applications and data platform in production in a Kubernetes cluster. 

In addition to my session be sure to check out the following sessions on Kubernetes by my friends Bob Ward and Hamish Watson, I’m certainly going to be at both of these sessions!

 

 

 

 

PASS Summit 2019

Updated: Getting Started with Installing Kubernetes

Let’s get you started on your Kubernetes journey with installing Kubernetes and creating a cluster in virtual machines.

Kubernetes is a distributed system, you will be creating a cluster which will have a master node that is in charge of all operations in your cluster. In this walkthrough we’ll create three workers which will run our applications. This cluster topology is, by no means, production ready. If you’re looking for production cluster builds check out Kubernetes documentation. Here and here. The primary components that need high availability in a Kubernetes cluster are the API Server which controls the state of the cluster and the etcd database which persists the state of the cluster. You can learn more about Kubernetes cluster components here. If you want to dive into Kubernetes more check out my Pluralsight Courses here! Where I have a dedicated course on Installation and Configuration.

In our demonstration here, the master is where the API Server, etcd, and the other control plan functions will live. The workers/nodes, will be joined to the cluster and run our application workloads. 

Get your infrastructure sorted

I’m using 4 Ubuntu Virtual machines in VMware Fusion on my Mac. Each with 2vCPUs and 2GB of RAM running Ubuntu 16.04.5. Ubuntu 18 requires a slightly different install. Documented here. In there you will add the Docker repository, then install Docker from there. The instructions below get Docker from Ubuntu’s repository. You will also need to disable the swap on any system which you will run the kubelet, which in our case is all systems. To do so you need to turn swap off with sudo swapoff -a and edit /etc/fstab removing or commenting out the swap volume entry. 

  • c1-master1 – 172.16.94.15
  • c1-node1 – DHCP
  • c1-node2 – DHCP
  • c1-node3 – DHCP

Ensure that each host has a unique name and that all nodes can have network reachability between each other. Take note of the IPs, because you will need to log into each node with SSH. If you need assistance getting your environment ready, check out my training on Pluralsight to get you started here! I have courses on installation, command line basics all the way up through advanced topics on networking and performance.

Overview of the cluster creation process

  • Install Kubernetes packages on all nodes
    • Add Kubernetes’ apt repositories
    • Install the required software packages for Kubernetes
  • Download deployment files for your Pod Network
  • Create a Kubernetes cluster on the Master
    • We’re going to use a utility called kubeadm to create our cluster with a basic configuration
  • Install a Pod Network
  • Join our three worker nodes to our cluster

Install Kubernetes Packages

Let’s start off with installing the required Kubernetes packages on to all of the nodes in our system. This is going to require logging into each server via SSH (or console), adding the Kubernetes apt repositories and installing the required packages. Perform the following tasks on ALL nodes in your cluster, the master and the three workers. If you add more nodes, you will need to install these packages on those nodes too.

Add the gpg key for the Kubernetes apt repository to your local system

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
Add the Kubernetes apt repository to your local repository locations
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
Next, we’ll update our apt package lists
sudo apt-get update
Install the required packages
sudo apt-get install -y docker.io kubelet kubeadm kubectl
Then we need to tell apt to not update these packages. 
sudo apt-mark hold docker.io kubelet kubeadm kubectl
With Docker installed, we need to make one adjustment to its configuration changing the cgroup driver to systemd.
sudo bash -c 'cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF'
With that file created, go ahead and reload the systemd configuration and restart the docker daemon.
sudo systemctl daemon-reload
sudo systemctl restart docker
Here’s what you just installed
  • kubelet – On each node in the cluster, this is in charge of starting and stopping pods in response to the state defined on the API Server on the master 
  • kubeadm – Primary command line utility for creating your cluster
  • kubectl – Primary command line utility for working with your cluster
  • docker – Remember, that Kubernetes is a container orchestrator so we’ll need a container runtime to run your containers. We’re using Docker. You can use other container runtimes if required

Download the YAML files for your Pod Network

Now, only on the Master, let’s download the YAML deployment file for your Pod network and get our cluster created. Networking in Kubernetes is different than what you’d expect. For Pods to be on different nodes to be able to communicate with each other on the same IP network, you’ll want to create a Pod network. Which essentially is an overlay network that gives you a uniform address space for Pods to operate in. The decision of which Pod network to use, or even if you need one is very dependent on your local or cloud infrastructure. For this demo, I’m going to use the Calico Pod network overlay. The code below will download the Pod manifest in YAML and we’ll deploy those into our cluster. This creates a DaemonSet. A DaemonSet is a Kubernetes Controller that will start the specified Pod on all or some of the nodes in the cluster. In this case, the Calico network Pod will be deployed on all nodes in our cluster. So as we join nodes, you might see some delay in nodes becoming Ready…this is because the container is being pulled and started on the node.
 
Download the YAML for the Pod network
wget https://docs.projectcalico.org/master/manifests/calico.yaml
If you need to change the address of your Pod network edit calico.yaml, look for the name: CALICO_IPV4POOL_CIDR and set the value: to your specified CIDR range. It’s 192.168.0.0/16 by default. 

Creating a Kubernetes Cluster

Now we’re ready to create our Kubernetes cluster, we’re going to use kubeadm to help us get this done. It’s a community-based tool that does a lot of the heavy lifting for you.
 
To create a cluster do this, here we’re specifying a CIDR range to match that in our calico.yaml file.
sudo kubeadm init --pod-network-cidr=192.168.0.0/16
What’s happening behind the scenes with kubeadm init:
  • Creates a certificate authority – Kubernetes uses certificates to secure communication between components, verify the identity of Nodes in the cluster and authenticate users.
  • Creates kubeconfig files – On the Master, this will create configuration files for various Kubernetes cluster components
  • Pulls Control Plane container images – the services implementing the cluster components are deployed into the cluster as containers. Very cool! You can, of course, run these as local system daemons on the hosts, but Kubernetes suggests keeping them inside containers
  • Bootstraps the Control Plane Pods – starts up the pods and creates static manifests on the master start automatically when the master node starts up
  • Taints the Master to just system pods – this means the master will run (schedule) only system Pods, not user Pods. This is ideal for production. In testing, you may want to untaint the master, you’ll really want to do this if you’re running a single node cluster. See this link for details on that.
  • Generates a bootstrap token – used to join worker nodes to the cluster
  • Starts any add-ons – the most common add-ons are the DNS pod and the master’s kube-proxy
If you see this output, you’re good to go! Keep that join command handy. We’ll need it in a second.
[init] Using Kubernetes version: v1.16.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
…output omitted… Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:  /docs/concepts/cluster-administration/addons/ 
Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.16.94.20:6443 --token czpkcj.ncl6p005orlie95h \ --discovery-token-ca-cert-hash sha256:3e21bb225c0986330ba11dd37c51fcd6542928964832705e13b84354872270bd

The output from your cluster creation is very important, it’s going to give you the code needed to access your cluster, the code needed to create your Pod network and also the code needed to join worker nodes to your cluster (just go ahead and copy this into a text file right now). Let’s go through each of those together.

Configuring your cluster for access from the Master node as a non-privileged user

This will allow you to log into your system with a regular account and administer your cluster.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Create your Pod network

Now that your cluster is created, you can deploy the YAML files for your Pod network. You must do this prior to adding more nodes to your cluster and certainly before starting any Pods on those nodes. We are going to use kubectl apply -f calico.yaml to deploy the Pod network from the YAML manifest we downloaded earlier. 

kubectl apply -f calico.yaml
configmap/calico-config created customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created clusterrole.rbac.authorization.k8s.io/calico-node created clusterrolebinding.rbac.authorization.k8s.io/calico-node created daemonset.apps/calico-node created serviceaccount/calico-node created deployment.apps/calico-kube-controllers created serviceaccount/calico-kube-controllers created
Before moving forward, check for the creation of the Calico pods and also the DNS pods, once these are created and the STATUS is Running then you can proceed. In this output here you can also see the other components of your Kubernetes cluster. You see the Pods running etcd, API Server, the Controller Manager, kube-proxy and the Scheduler.
kubectl get pods --all-namespaces
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-7594bb948-4mgqd   1/1     Running   0          2m58s
kube-system   calico-node-qpcv7                         1/1     Running   0          2m58s
kube-system   coredns-5644d7b6d9-2lxgt                  1/1     Running   0          3m42s
kube-system   coredns-5644d7b6d9-g5tfc                  1/1     Running   0          3m42s
kube-system   etcd-c2-master1                           1/1     Running   0          2m50s
kube-system   kube-apiserver-c2-master1                 1/1     Running   0          2m41s
kube-system   kube-controller-manager-c2-master1        1/1     Running   0          3m5s
kube-system   kube-proxy-d2c6s                          1/1     Running   0          3m42s
kube-system   kube-scheduler-c2-master1                 1/1     Running   0          2m44s

Joining worker nodes to your cluster

Now on each of the worker nodes, let’s use kubeadm join to join the worker nodes to the cluster. Go back to the output of kubeadm init and copy the string from that output be sure to put a sudo on the front before you do this on each node. The process below is called a TLS bootstrap. This securely joins the node to the cluster over TLS and authenticates the host with server certificates.
sudo kubeadm join 172.16.94.20:6443 \
>     --token czpkcj.ncl6p005orlie95h \
>     --discovery-token-ca-cert-hash sha256:3e21bb225c0986330ba11dd37c51fcd6542928964832705e13b84354872270bd
[sudo] password for aen:
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 
If you didn’t keep the token or the CA Cert Hash in the earlier steps, go back to the master and run these commands. Also note, that join token is only valid for 24 hours. 
 
To get the current join token
kubeadm token list
To get the CA Cert Hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
Back on the master, check on the status of your nodes joining the cluster. These nodes are currently NotReady, behind the scenes they’re pulling the Calico Pods and setting up the Pod network.
kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
c1-master1   Ready      master   11m   v1.16.1
c1-node1     NotReady   <none>   63s   v1.16.1
c1-node2     NotReady   <none>   57s   v1.16.1
C1-node3     NotReady   <none>   33s   v1.16.1
And here we are with a fully functional Kubernetes cluster! All nodes joined and Ready.
kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
c1-master1   Ready    master   12m     v1.16.1
c1-node1     Ready    <none>   3m04s   v1.16.1
c1-node2     Ready    <none>   2m31s   v1.16.1
C1-node3     Ready    <none>   1m28s   v1.16.1
Please feel free to contact me with any questions regarding Kubernetes, Linux and other SQL Server related issues at: aen@centinosystems.com