Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Kubernetes components 

A Kubernetes cluster consists of the components that represent the control plane and a set of machines called nodes.

Control Plane Components 

• kube-api server – The API server is a component of the Kubernetes control plane that exposes the Kubernetes API. The API server is the front end for the Kubernetes control plane
• etcd – Consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data
• kube-scheduler – Control plane component that watches for newly created Pods with no assigned node, and selects a node for them to run on
• kube-control-manager – Control Plane component that runs controller processes
• cloud-controller-manager

Nodes Components 

• kublet – An agent that runs on each node in the cluster. It makes sure that containers are running in a pod.
• kube-proxy – kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Serviceconcept.
• container runtime – The container runtime is the software that is responsible for running containers.

Cluster Networking

Managing a network where containers can interoperate efficiently is very important. Kubernetes has adopted the Container Network Interface(CNI) specification for managing network resources on a cluster. This relatively simple specification makes it easy for Kubernetes to interact with a wide range of CNI-based software solutions.

Networking is a central part of Kubernetes, but it can be challenging to understand exactly how it is expected to work.

There are 4 distinct networking problems :

1. container-to-container communications: this is solved by pods and localhost communications.
2. Pod-to-Pod communications
3. Pod-to-Service communications: this is covered by services.
4. External-to-Service communications: this is covered by services.

 

Ingress 

An API object that manages external access to the services in a cluster, typically HTTP.

Ingress may provide load balancing, SSL termination and name-based virtual hosting

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

Kubernetes network 

Kubernetes objects 

Kubernetes objects are persistent entities in the Kubernetes system. Kubernetes uses these entities to represent the state of your cluster. Specifically, they can describe:

• What containerized applications are running (and on which nodes)
• The resources available to those applications
• The policies around how those applications behave, such as restart policies, upgrades, and fault-tolerance

 

• The kubelet uses liveness probes to know when to restart a container
• The kubelet uses readiness probes to know when a container is ready to start accepting traffic
• The kubelet uses startup probes to know when a container application has started

 

 

Kubernetes cluster

Install Kubernetes – Master node

Install Kubernetes – Minios / Nodes

#kubectl version

#kubectl get nodes –help

#kubectl get nodes

 

# kubectl run

 

Reference

https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

 

Apache Nifi,

Amazon Redshift is a fast, fully managed data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing Business Intelligence (BI) tools. It allows you to run complex analytic queries against petabytes of structured data, using sophisticated query optimization, columnar storage on high-performance local disks, and massively parallel query execution. Most results come back in seconds.

 

 

Reference:

https://aws.amazon.com/redshift/

 

 

Hypervisors

 

Type I Hypervisor 

The hypervisor runs directly on top of hardware

Type II Hypervisor

The hypervisor acts like separate layer often on top of a base operating system

 

KVM

KVM is a Type II Hypervisor, which means that it is one layer above the OS.  On a bare metal machine, install an OS, and THEN install KVM. In Linux it is installed in the form of a Kernel, and this kernel then turns the bare metal machine into a Hypervisor.

KVM will offer better performance with the Linux OS.

XEN

Xen is a Type I Hypervisor – which means it sits a single layer above bare metal.  It’s like a stripped down operating system, and it uses a function called “pass-through” to connect directly to PCI devices like RAM/CPU/NICs.

 

Reference :

https://onapp.com/2016/09/06/hypervisor-choice-xen-or-kvm/