HAWQ is the new benchmark for SQL on Hadoop,
HAWQ is a parallel SQL query engine . HAWQ has been designed from the ground up to be a massively parallel SQL processing engine optimized specifically for analytics with full transaction support
Pivotal HAWQ is a Massively Parallel Processing (MPP) database using several Postgres database instances and HDFS storage
Hawq Physical Architecture
- Hawq Master server
- HDFS NameNode
- Segment server
- Interconnect Switch
How to locate the logs:
HAWQ master logs.
[gpadmin@uphdmst02 gpseg-1]$ psql
psql (8.4.20, server 8.2.15)
WARNING: psql version 8.4, server version 8.2.
Some psql features might not work.
Type “help” for help.
gpadmin=# show data_directory;
data_directory
———————-
/data/master/gpseg-1
(1 row)
[gpadmin@uphdmst02 ~]$ psql
psql (8.4.20, server 8.2.15)
WARNING: psql version 8.4, server version 8.2.
Some psql features might not work.
Type “help” for help.
gpadmin=# select version();
version
————————————————————————-
————————————————-
PostgreSQL 8.2.15 (Greenplum Database 4.2.0 build 1) (HAWQ 1.2.1.0 build 10335) on x86_64-unknown-linux-gnu, comp
iled by GCC gcc (GCC) 4.4.2 compiled on Aug 8 2014 16:31:48
(1 row)
Create some external table in uat
gpadmin=# CREATE EXTERNAL TABLE person ( id int, name text)
gpadmin-# location(‘gpfdist://phdmst03.uat.mydev.com:8000/Test/person.txt’) FORMAT ‘text’ (delimiter ‘|’)
gpadmin-# ENCODING ‘UTF8’;
CREATE EXTERNAL TABLE
gpadmin=# select count(*) from person;
count
———
1000000
(1 row)
gpadmin=#
— External Table: ext_sim_result_value_f
— DROP EXTERNAL TABLE ext_sim_result_value_f;
CREATE EXTERNAL TABLE ext_sim_result_value_f
(
sim_result_id ,
md_point_id ,
path_num ,
value
)
LOCATION (
‘gpfdist://uphdmst03.uat.mydev.com:8000/500041_Sim_Result_Value_F.csv.gz’
)
FORMAT ‘text’ (delimiter ‘|’ null ‘\\N’ escape ‘\\’)
ENCODING ‘UTF8’;
ALTER TABLE ext_sim_result_value_f
OWNER TO gpadmin;
shutdown
3
20150731:16:56:16:570576 gpstop:uphdmst02:gpadmin-[INFO]:-Commencing parallel segment instance shutdown, please wait…
…………….
20150731:16:56:32:570576 gpstop:uphdmst02:gpadmin-[INFO]:—————————————————–
20150731:16:56:32:570576 gpstop:uphdmst02:gpadmin-[INFO]:- Segments stopped successfully = 24
20150731:16:56:32:570576 gpstop:uphdmst02:gpadmin-[INFO]:- Segments with errors during stop = 0
20150731:16:56:32:570576 gpstop:uphdmst02:gpadmin-[INFO]:—————————————————–
20150731:16:56:32:570576 gpstop:uphdmst02:gpadmin-[INFO]:-Successfully shutdown 24 of 24 segment instances
20150731:16:56:32:570576 gpstop:uphdmst02:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
Hawq – pga_conf
vim /data/master/gpseg-1/pg_hba.conf
source /usr/local/hawq/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/data/master/gpseg-1
reload the configuration
[gpadmin@uphdmst02 ~]$ gpstop -u
20150805:11:05:40:233745 gpstop:uphdmst02:gpadmin-[INFO]:-Starting gpstop with args: -u
20150805:11:05:40:233745 gpstop:uphdmst02:gpadmin-[INFO]:-Gathering information and validating the environment…
20150805:11:05:40:233745 gpstop:uphdmst02:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20150805:11:05:40:233745 gpstop:uphdmst02:gpadmin-[INFO]:-Obtaining Segment details from master…
20150805:11:05:41:233745 gpstop:uphdmst02:gpadmin-[INFO]:-Greenplum Version: ‘postgres (HAWQ) 4.2.0 build 1’
20150805:11:05:41:233745 gpstop:uphdmst02:gpadmin-[INFO]:-Signalling all postmaster processes to reload
pg_hba.conf
host all gpadmin 192.168.68.135/32 trust
host all gpadmin 192.168.68.135/32 trust
host all all 192.193.68.132/32 trust
host all gpadmin 10.115.yyy.0/24 trust
host all gpadmin 10.115.yyy.0/24 trust
host all aafsh02 10.115.zzz.0/24 trust
#host all all 10.115.xx.xx/32 trust
local all gpadmin ident
host all gpadmin 127.0.0.1/28 trust
host all gpadmin 192.193.68.134/32 trust
host all gpadmin ::1/128 trust
host all gpadmin fe80::3aea:a7ff:fe35:c0c/128 trust
host all gpadmin 192.168.68.136/32 trust
host all gpadmin 10.110.xxx.0/24 trust
#host all gpadmin 10.115.xxx.0/24 trust
#host all gpadmin 10.115.xxx.0/24 trust
host all all 10.115.xxx.0/24 ldap ldapserver=xxx.225.227.15 ldapprefix=”OFFICE\”
host all all 10.110.xxx.0/24 ldap ldapserver=xxx.225.227.15 ldapprefix=”OFFICE\”
host all all 10.202.xxx.0/24 ldap ldapserver=xxx.225.227.15 ldapprefix=”OFFICE\”
host benchmark fsh02 10.115.xxx.20/32 ldap ldapserver=xxx.225.227.yyy ldapprefix=”OFFICE\”
host benchmark fsh02 10.193.xxx.yyy/32 ldap ldapserver=xxx.225.227.yyy ldapprefix=”OFFICE\”
Create user in hawq
padmin=# CREATE USER ong01 WITH LOGIN ;
gpadmin=# CREATE USER user1 WITH LOGIN ;
gpadmin=# \du
List of roles
Role name | Attributes | Member of
———— —–+———————————————+————-
CM_Admin | Cannot login |
RD_Admin | Cannot login |
RD_Users | Cannot login |
ovi02 | Superuser, Create DB |
ong01 | |
gpadmin | Superuser, Create role, Create DB |
user2 | |
ovi | |
rd_user | | {RD_Users}
user1 | |
gpadmin-# \l
List of databases
Name | Owner | Encoding | Access privileges
———–+———-+———-+———————–
APMS | gpadmin | UTF8 |
B_POC | gpadmin | UTF8 | =Tc/gpadmin
: gpadmin=CTc/gpadmin
: gorodn=CTc/gpadmin
: asi=CTc/gpadmin
CCD | gpadmin | UTF8 |
CI_DB | usertest | UTF8 | usertest=CTc/usertest
: ovi=CTc/usertest
Mike_Test | gpadmin | UTF8 | =Tc/gpadmin
: gpadmin=CTc/gpadmin
: ovi=CTc/gpadmin
GRANT ALL PRIVILEGES
ON TABLE sim_result_d2, sim_result_value_f
TO PUBLIC
So for now they can access the tables I created.
Infra Cloud Solutions 