Prometheus is a systems and service monitoring system. It collects metrics
from configured targets at given intervals, evaluates rule expressions,
displays the results, and can trigger alerts if some condition is observed
to be true.

Redis is an open source, in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster

 

reference

https://redis.io/

 

Linux Container

Hue is a browser-based environment that enables you to interact with a Hadoop cluster. Hue includes several easy to use applications that help you work with Hadoop MapReduce jobs, Hive queries, Hadoop files and user accounts. The Hue applications run in a Web browser and require no client installation.

Hue is a Web UI for Hadoop.

Hue supports the following features:

Web access console

http://localhost:8000/

ovi

I…09

 

AWS Schema Conversion Tool makes heterogeneous database migrations easy by automatically converting the source database schema and a majority of the custom code to a format compatible with the target database. The custom code that the tool converts includes views, stored procedures, and functions

Grafana

AWS – EBS dasboard

 

AWS billing

Visualize estimated AWS charges per AWS resource (EC2, S3, …)

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Kubernetes components 

A Kubernetes cluster consists of the components that represent the control plane and a set of machines called nodes.

Control Plane Components 

• kube-api server – The API server is a component of the Kubernetes control plane that exposes the Kubernetes API. The API server is the front end for the Kubernetes control plane
• etcd – Consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data
• kube-scheduler – Control plane component that watches for newly created Pods with no assigned node, and selects a node for them to run on
• kube-control-manager – Control Plane component that runs controller processes
• cloud-controller-manager

Nodes Components 

• kublet – An agent that runs on each node in the cluster. It makes sure that containers are running in a pod.
• kube-proxy – kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Serviceconcept.
• container runtime – The container runtime is the software that is responsible for running containers.

Cluster Networking

Managing a network where containers can interoperate efficiently is very important. Kubernetes has adopted the Container Network Interface(CNI) specification for managing network resources on a cluster. This relatively simple specification makes it easy for Kubernetes to interact with a wide range of CNI-based software solutions.

Networking is a central part of Kubernetes, but it can be challenging to understand exactly how it is expected to work.

There are 4 distinct networking problems :

1. container-to-container communications: this is solved by pods and localhost communications.
2. Pod-to-Pod communications
3. Pod-to-Service communications: this is covered by services.
4. External-to-Service communications: this is covered by services.

 

Ingress 

An API object that manages external access to the services in a cluster, typically HTTP.

Ingress may provide load balancing, SSL termination and name-based virtual hosting

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

Kubernetes network 

Kubernetes objects 

Kubernetes objects are persistent entities in the Kubernetes system. Kubernetes uses these entities to represent the state of your cluster. Specifically, they can describe:

• What containerized applications are running (and on which nodes)
• The resources available to those applications
• The policies around how those applications behave, such as restart policies, upgrades, and fault-tolerance

 

• The kubelet uses liveness probes to know when to restart a container
• The kubelet uses readiness probes to know when a container is ready to start accepting traffic
• The kubelet uses startup probes to know when a container application has started

 

 

Kubernetes cluster

Install Kubernetes – Master node

Install Kubernetes – Minios / Nodes

#kubectl version

#kubectl get nodes –help

#kubectl get nodes

 

# kubectl run

 

Reference

https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

 

Apache Nifi,

Amazon Redshift is a fast, fully managed data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing Business Intelligence (BI) tools. It allows you to run complex analytic queries against petabytes of structured data, using sophisticated query optimization, columnar storage on high-performance local disks, and massively parallel query execution. Most results come back in seconds.

 

 

Reference:

https://aws.amazon.com/redshift/