Year: 2016

AWS – EFS

techhadoop

AWS – EFS

Amazon EFS is a fully-managed service that makes it easy to set up and scale file storage in the Amazon cloud

Amazon EFS is a file storage service for use with Amazon EC2. Amazon EFS provides a file system interface, file system access semantics (such as strong consistency and file locking), and concurrently-accessible storage for up to thousands of Amazon EC2 instances.

Amazon EBS is a block level storage service for use with Amazon EC2. Amazon EBS can deliver performance for workloads that require the lowest-latency access to data from a single EC2 instance.

Amazon S3 is an object storage service. Amazon S3 makes data available through an Internet API that can be accessed anywhere.

 

Amazon EFS uses the NFSv4.1 protocol

From AWS Console, go to EFS

Step 1 : Configure file system access
Step 2 : Configure optional settings
Step 3 : Review and create

An Amazon EFS file system is accessed by EC2 instances running inside one of your VPCs. Instances connect to a file system via a network interface called a mount target. Each mount target has an IP address, which we assign automatically or you can specify.

Create mount targets

Instances connect to a file system via mount targets you create. We recommend creating a mount target in each of your VPC’s Availability Zones so that EC2 instances across your VPC can access the file system.

Mount target – To access your file system, you must create mount targets in your VPC. Each mount target has the following properties: the mount target ID, the subnet ID in which it is created, the file system ID for which it is created, an IP address at which the file system may be mounted, and the mount target state. You can use the IP address or the DNS name in your mount command. Each mount target has a DNS name of the following form:

availability-zone.file-system-id.efs.aws-region.amazonaws.com

On First EC2 Instance :

#yum install nfs-utils

Create a local directory ( e.g efs )

# mkdir efs

With mount command – mount the target you can use DNS or IP ( I use IP ) please see attached

#mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 172.30.zzyy:/ /efs

[root@ efs]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 992M 80K 992M 1% /dev
tmpfs 1002M 0 1002M 0% /dev/shm
/dev/xvda1 7.8G 1.2G 6.6G 15% /
/dev/xvdb 25G 2.5G 21G 11% /data
/dev/xvdh 79G 19G 56G 26% /data3
172.30.yy.zz:/ 8.0E 0 8.0E 0% /efs

[root@ efs]# ls -l /efs
total 16
drwxr-xr-x 2 root root 4096 Nov 14 20:44 data
drwxr-xr-x 2 root root 4096 Nov 14 20:43 data2
drwxr-xr-x 2 root root 4096 Nov 14 20:43 data3
drwxr-xr-x 2 root root 4096 Nov 14 19:09 test_efs

On second Ec2 instance
#yum install nfs-utils

mount target mount under local folder efs_2

#mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 172.30.yy.zz:/ /efs_2

[root@ip-172-30 efs_2]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 992M 68K 992M 1% /dev
tmpfs 1002M 0 1002M 0% /dev/shm
/dev/xvda1 7.8G 1.1G 6.7G 14% /
172.30.yy.zz:/ 8.0E 0 8.0E 0% /efs_2

[root@ /]# ls -l /efs_2/
total 16
drwxr-xr-x 2 root root 4096 Nov 14 20:44 data
drwxr-xr-x 2 root root 4096 Nov 14 20:43 data2
drwxr-xr-x 2 root root 4096 Nov 14 20:43 data3
drwxr-xr-x 2 root root 4096 Nov 14 19:09 test_efs

install boto

techhadoop

Boto is a Python package that provides interfaces to AWS including Amazon S3

boto – the AWS SDK for Python. Boto3 makes it easy to integrate your Python application, library, or script with AWS services including Amazon S3,

Amazon EC2, Amazon DynamoDB, and more.

[root@ip-172-…-126 ~]# pip list | grep boto
You are using pip version 6.1.1, however version 9.0.1 is available.
You should consider upgrading via the ‘pip install –upgrade pip’ command.
boto (2.42.0)
botocore (1.4.86)

[root@ip-172-30- ~]# pip install -U boto
You are using pip version 6.1.1, however version 9.0.1 is available.
You should consider upgrading via the ‘pip install –upgrade pip’ command.
Collecting boto
Downloading boto-2.43.0-py2.py3-none-any.whl (1.3MB)
100% |████████████████████████████████| 1.3MB 354kB/s
Installing collected packages: boto
Found existing installation: boto 2.42.0
Uninstalling boto-2.42.0:
Successfully uninstalled boto-2.42.0
Successfully installed boto-2.43.0

 

pip is a package management system used to install and manage software packages written in Python. Many packages can be found in the Python Package Index (PyPI). Python 2.7.9 and later (on the python2 series), and Python 3.4 and later include pip (pip3 for Python 3) by default.

 

[root@ip-172-30 ~]# pip install –upgrade pip
You are using pip version 6.1.1, however version 9.0.1 is available.
You should consider upgrading via the ‘pip install –upgrade pip’ command.
Collecting pip
Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
100% |████████████████████████████████| 1.3MB 372kB/s
Installing collected packages: pip
Found existing installation: pip 6.1.1
Uninstalling pip-6.1.1:
Successfully uninstalled pip-6.1.1
Successfully installed pip-9.0.1

 

 

 

/etc/boto.cfg
[root@ etc]# more boto.cfg
[Credentials]
aws_access_key_id = AKIA************************
aws_secret_access_key = oH7JxIljhY**************

 

 

simple script to upload a file to AWS S3

#!/usr/bin/python

import boto
from boto.s3.key import Key

keyId = “AKIA**************”
sKeyId= “eOCZ4********************”

fileName=”abcd.txt”
bucketName=”ovi-test”
file = open(fileName)

conn = boto.connect_s3(keyId,sKeyId)
bucket = conn.get_bucket(bucketName)

#Get the Key object of the bucket
k = Key(bucket)

#Crete a new key with id as the name of the file
k.key=fileName

#Upload the file
result = k.set_contents_from_file(file)
#result contains the size of the file uploaded

You can test if file was uploaded properly from aws cli

#aws s3 ls s3://ovi-test
PRE aws_doc/
PRE test/
2016-11-11 15:24:10 14 abcd.txt
2016-10-06 14:30:07 14 ovi2.txt
2016-10-06 12:01:16 13 test

AWS- clean up

techhadoop

to keep costs under control for AWS we will clean up below :

  • clean up idle EC2 Instance
  • clean up EBS Volumes which are unattached / all the volumes available to remove
  • clean up snapshots orphans / delete all Snapshots which are not in use
  • all Snapshots not in use by any AMI’s
  • Release elastic IPs when they’re not in use

 

 

#aws ec2 describe-regions | grep -i regionname
“RegionName”: “ap-south-1”
“RegionName”: “eu-west-1”
“RegionName”: “ap-northeast-2”
“RegionName”: “ap-northeast-1”
“RegionName”: “sa-east-1”
“RegionName”: “ap-southeast-1”
“RegionName”: “ap-southeast-2”
“RegionName”: “eu-central-1”
“RegionName”: “us-east-1”
“RegionName”: “us-east-2”
“RegionName”: “us-west-1”
“RegionName”: “us-west-2”

 

# aws ec2 describe-volumes > describe_volumes.txt

# aws ec2 describe-volumes –region us-west-1 > describe_volumes_us-west1.txt

# aws ec2 describe-snapshots > describe_snapshots

 

 

➜ # more describe_snapshots | grep -i SNAPSHOT | awk ‘{print $2}’ | sort | uniq | wc -l
15445

# aws ec2 describe-volumes –region us-west-1 | grep -i available
“State”: “available”,
“State”: “available”,
“State”: “available”,
“State”: “available”,
“State”: “available”,
“State”: “available”,
“State”: “available”,
“State”: “available”,
“State”: “available”,
“State”: “available”,
“State”: “available”,
“State”: “available”,

 

 

 

Reference :

aws clean up

http://www.robertsindall.co.uk/blog/how-to-clean-up-amazon-ebs-volumes-and-snapshots/

Detect useless Snapshots and Volumes in the Amazon EC2 Cloud

http://cloudacademy.com/blog/how-to-manage-ebs-volumes-snapshots-in-aws/

aws – Storage Gateway

techhadoop

gateway-cached volumes

Gateway-cached volumes allow you to utilize Amazon S3 for your primarily data, while retaining some portion of it locally in a cache for

frequently accessed data

 

gateway-stored volumes

gateway -stored volumes store your primary data locally, while asynchronously backing up data to AWS.

 

gateway-virtual tape Library

aws – metadata

techhadoop

Instance metadata is data about your instance tah you can use to configure or manage the running instance

[root@ip-10-192- ]# curl http://169.254.169.254/latest/meta-data/

ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-keys/
reservation-id
security-groups

services/

[root@ip-10-] curl http://169.254.169.254/latest/meta-data/ami-id ; echo
ami-de347abc

 

[root@ip-10-192-10]# curl http://169.254.169.254/
1.0
2007-01-19
2007-03-01
2007-08-29
2007-10-10
2007-12-15
2008-02-01
2008-09-01
2009-04-04
2011-01-01
2011-05-01
2012-01-12
2014-02-25
2014-11-05
2015-10-20
2016-04-19
2016-06-30

root@ip-10-192]# curl http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key

Check public IP behind NAT

[root@ip-10-192- ]# wget -qO- http://ipecho.net/plain ; echo
50.18.yyy.yy

EBS – volume in linux

techhadoop

After you attach an Amazon EBS volume to your instance, it is exposed as a block device. You can format the volume with any file system and then mount it

[root@ip-172-30… //]# lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
└─xvda1 202:1    0   8G  0 part /
xvdf    202:80   0   2G  0 disk

[root@ip-172-30 //]# file -s /dev/xvdf
/dev/xvdf: data

If the output of the previous command shows simply data for the device, then there is no file system on the device and you need to create

[root@ip-172-30-0-59 //]# mkfs -t ext4 /dev/xvdf
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 524288 4k blocks and 131072 inodes
Filesystem UUID: 33193f80-886e-41ad-858e-6be5a4dde19e
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

after format, check again

[root@ip-172-30-//]# file -s /dev/xvdf
/dev/xvdf: Linux rev 1.0 ext4 filesystem data, UUID=33193f80-886e-41ad-858e-6be5a4dde19e (extents) (large files) (huge files)

 

[root@ip-172-30- /]# ls -al /dev/disk/by-uuid/
total 0
drwxr-xr-x 2 root root  80 Oct  4 14:16 .
drwxr-xr-x 7 root root 140 Oct  4 14:16 ..
lrwxrwxrwx 1 root root  10 Oct  4 14:16 33193f80-886e-41ad-858e-6be5a4dde19e -> ../../xvdf
lrwxrwxrwx 1 root root  11 Oct  4 14:17 43c07df6-e944-4b25-8fd1-5ff848b584b2 -> ../../xvda1

edit /etc/fstab

[root@ip-172-30-0-235 /]# cat /etc/fstab
#
LABEL=/     /           ext4    defaults,noatime  1   1
tmpfs       /dev/shm    tmpfs   defaults        0   0
devpts      /dev/pts    devpts  gid=5,mode=620  0   0
sysfs       /sys        sysfs   defaults        0   0
proc        /proc       proc    defaults        0   0
/dev/xvdf   /apps       ext4    defaults        0   0

 

create a directory apps

# mkdir  apps

#mount -a

test

 

[root@ip-172-30- /]# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8123812 3819192   4204372  48% /
devtmpfs          498816      60    498756   1% /dev
tmpfs             509664       0    509664   0% /dev/shm
/dev/xvdf        1998672    3076   1874356   1% /apps

 

With Amazon EBS encryption, you can now create an encrypted EBS volume and attach it to a supported instance type. Data on the volume, disk I/O,
and snapshots created from the volume are then all encrypted. The encryption occurs on the servers that host the EC2 instances, providing
encryption of data as it moves between EC2 instances and EBS storage. EBS encryption is based on the industry standard AES-256
cryptographic algorithm.
** Snapshots that are taken from encrypted volumes are automatically encrypted.
** Volumes that are created from encrypted snapshots are also automatically encrypted.

Public snapshots of encrypted volumes are not supported, but you can share an encrypted snapshot with specific accounts if you
take the following steps:

– Use a custom CMK, not your default CMK, to encrypt your volume.
– Give the specific accounts access to the custom CMK.
– Create the snapshot.
– Give the specific accounts access to the snapshot.

– You cannot snapshot an EC2 instance store volume.