GCP DataProc 설치

DASHBoard 에 가면 확인 가능 username : student-03-db9b109b5ecf@qwiklabs.net project id : qwiklabs-gcp-00-cb7bb719eb9a project Number : 378084473336

1. ranger 관리자 비밀번호 설정 gcloud projects add-iam-policy-binding [project-id]
   –member=serviceAccount:[project-number]-compute@developer.gserviceaccount.com
   –role=roles/cloudkms.cryptoKeyDecrypter ==> project-id, project-number 를 나의 환경에 맞게 수정한다 gcloud projects add-iam-policy-binding qwiklabs-gcp-00-cb7bb719eb9a
   –member=serviceAccount:378084473336-compute@developer.gserviceaccount.com
   –role=roles/cloudkms.cryptoKeyDecrypter

student_03_db9b109b5ecf@cloudshell:~ (qwiklabs-gcp-00-cb7bb719eb9a)$ gcloud projects add-iam-policy-binding qwiklabs-gcp-00-cb7bb719eb9a \
    --member=serviceAccount:378084473336-compute@developer.gserviceaccount.com \
    --role=roles/cloudkms.cryptoKeyDecrypter
Updated IAM policy for project [qwiklabs-gcp-00-cb7bb719eb9a].
bindings:
- members:
  - user:student-03-db9b109b5ecf@qwiklabs.net
  role: roles/aiplatform.admin
- members:
  - serviceAccount:qwiklabs-gcp-00-cb7bb719eb9a@qwiklabs-gcp-00-cb7bb719eb9a.iam.gserviceaccount.com
  - user:student-03-db9b109b5ecf@qwiklabs.net
  role: roles/bigquery.admin
- members:
  - serviceAccount:378084473336@cloudbuild.gserviceaccount.com
  role: roles/cloudbuild.builds.builder
- members:
  - serviceAccount:service-378084473336@gcp-sa-cloudbuild.iam.gserviceaccount.com
  role: roles/cloudbuild.serviceAgent
- members:
  - serviceAccount:378084473336-compute@developer.gserviceaccount.com
  role: roles/cloudkms.cryptoKeyDecrypter
- members:
  - serviceAccount:service-378084473336@compute-system.iam.gserviceaccount.com
  role: roles/compute.serviceAgent
- members:
  - serviceAccount:service-378084473336@container-engine-robot.iam.gserviceaccount.com
  role: roles/container.serviceAgent
- members:
  - serviceAccount:service-378084473336@dataproc-accounts.iam.gserviceaccount.com
  role: roles/dataproc.serviceAgent
- members:
  - serviceAccount:378084473336-compute@developer.gserviceaccount.com
  - serviceAccount:378084473336@cloudservices.gserviceaccount.com
  role: roles/editor
- members:
  - user:student-03-db9b109b5ecf@qwiklabs.net
  role: roles/iam.serviceAccountUser
- members:
  - serviceAccount:service-378084473336@gcp-sa-notebooks.iam.gserviceaccount.com
  role: roles/notebooks.serviceAgent
- members:
  - serviceAccount:admiral@qwiklabs-services-prod.iam.gserviceaccount.com
  - serviceAccount:qwiklabs-gcp-00-cb7bb719eb9a@qwiklabs-gcp-00-cb7bb719eb9a.iam.gserviceaccount.com
  - user:student-03-db9b109b5ecf@qwiklabs.net
  role: roles/owner
- members:
  - serviceAccount:qwiklabs-gcp-00-cb7bb719eb9a@qwiklabs-gcp-00-cb7bb719eb9a.iam.gserviceaccount.com
  - user:student-03-db9b109b5ecf@qwiklabs.net
  role: roles/storage.admin
- members:
  - user:student-03-db9b109b5ecf@qwiklabs.net
  role: roles/viewer
- members:
  - serviceAccount:service-378084473336@gcp-sa-websecurityscanner.iam.gserviceaccount.com
  role: roles/websecurityscanner.serviceAgent
etag: BwXaTBU8B9Y=
version: 1

1. KMS 사용하여 관리자 비밀번호를 암호화한다. 비밀번호는 최소 8자 이상, 영문, 숫자 조합이여야 한다 2.1 키링 생성 gcloud kms keyrings create my-keyring –location=global

2.2 키 생성 gcloud kms keys create my-key
–location=global
–keyring=my-keyring
–purpose=encryption

2.3 Ranger 관리자 비밀번호를 암호화 한다. echo “test12345” |
gcloud kms encrypt
–location=global
–keyring=my-keyring
–key=my-key
–plaintext-file=-
–ciphertext-file=admin-password.encrypted

2.4 생성된 비밀번호 파일을 프로젝트의 GCS 에 업로드 한다 gsutil cp admin-password.encrypted gs://my-bucket/ranger/

gsutil cp admin-password.encrypted gs://qwiklabs-gcp-00-cb7bb719eb9a/ranger/

1. Dataproc 설정

properties 설정 방법 –properties=”dataproc:ranger.kms.key.uri=projects/[project-id]/locations/global/keyRings/my-keyring/cryptoKeys/my-key,dataproc:ranger.admin.password.uri=gs://my-bucket/admin-password.encrypted” \

gcloud dataproc clusters create cluster-test
–enable-component-gateway
–region us-central1
–zone us-central1-c
–master-machine-type n1-standard-4
–master-boot-disk-size 500
–num-workers 2
–worker-machine-type n1-standard-4
–worker-boot-disk-size 500
–image-version 2.0-ubuntu18
–optional-components HIVE_WEBHCAT,JUPYTER,ZOOKEEPER,RANGER,HBASE,SOLR
–project qwiklabs-gcp-00-cb7bb719eb9a
–properties=”dataproc:ranger.kms.key.uri=projects/qwiklabs-gcp-00-cb7bb719eb9a/locations/global/keyRings/my-keyring/cryptoKeys/my-key,dataproc:ranger.admin.password.uri=gs://qwiklabs-gcp-00-cb7bb719eb9a/ranger/admin-password.encrypted” –initialization-actions gs://qwiklabs-gcp-00-cb7bb719eb9a/dataproc/hue/hue.sh

gcloud dataproc clusters create cluster-7403
–region us-central1
–initialization-actions gs://qwiklabs-gcp-00-cb7bb719eb9a/dataproc/hue/hue.sh gs://qwiklabs-gcp-00-cb7bb719eb9a/dataproc/hue/hue.sh

ERROR: (gcloud.dataproc.clusters.create) INVALID_ARGUMENT: Insufficient ‘DISKS_TOTAL_GB’ quota. Requested 3000.0, available 2596.0.

log into your Dataproc master node: gcloud compute –project “yourprojectname” ssh –zone “us-west1-a” “gcp_admin@my-cluster123” sudo su - wget https://raw.githubusercontent.com/GoogleCloudPlatform/dataproc-initialization-actions/master/oozie/oozie.sh . bash oozie.sh wget https://raw.githubusercontent.com/GoogleCloudPlatform/dataproc-initialization-actions/master/hue/hue.sh . bash hue.sh

Hue Web UI: http://your master_node_IP:8888 Oozie Web UI: http://your_master_node_IP:11000/oozie