您的位置：首页 > 数据库 > Oracle

OGG TO HDFS,及Kerberose验证方法

2016-10-21 00:00 615 查看

一、准备工作

1、源端ORACLE RDBMS 12c,及Oracle GoldenGate 12c for Oracle。安装需要有图形界面，需要准备好VNC或其他。

2、目的端Oracle GoldenGate 12 for Big Data 12 ,HADOOP 2.6

二、安装步骤。

1、源端ORACLE数据库安装及GoldenGate安装都要用到./runInstaller。ORACLE安装忽略，本文安装在/data1/oracle/。OGG FOR ORACLE安装在/data/oracle/oggora。

2、源端 ORACLE需要做的配置。

$sqlplus / as sysdba

SQL>SELECT log_mode FROM

$database;

看看是否是归档模式，若不是要做如下操作。

SQL> alter database

mount

Database altered.

SQL> alter databasearchivelog;

Database altered.

SQL> alter database

open

SQL> alter databaseadd supplemental log data;

Database altered.

SQL> alter databaseforce logging;

Database altered.

SQL>SELECT log_mode FROM

$database;

LOG_MODE

------------

ARCHIVELOG

开启ORACLE中相关OGG的参数

SQL> alter system set enable_goldengate_replication=

true

sid=

'*'

scope=both;

System altered.

创建ogg表空间，用户，并赋予DBA权值，不然，后面可能无法执行

SQL> CREATE TABLESPACE OGG DATAFILE

'/data1/oracle/app/oracle/oradata/orcl/ogg.dbf'

size 10M autoextend on;

SQL> create user ogg identified by ogg

default

tablespace ogg temporary tablespace temp;

SQL> EXEC DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE(

'OGGUSER'

);

SQL> grant connect,dba to ogg;

Grant succeeded.

3、源端GOLDENGATE要做配置

安装后OGG FOR ORACLE后，默认已经create subdirs了，并且已经配置启动mgr。

运行oggora$./ggsci进行入ogg命令行

GGSCI (TEST-BDD-

>edit params mgr

PORT

userid ogg,password ogg

purgeoldextracts /data1/oracle/oggora/*,usecheckpoints

accessrule,prog server,allow

重启MGR

GGSCI (TEST-BDD-

>  stop mgr

Manager process is required by other GGS processes.

Are you sure you want to stop it (y/n)?y

Sending STOP request to MANAGER ...

Request processed.

Manager stopped.

GGSCI (TEST-BDD-

> start mgr

Manager started.

GGSCI (TEST-BDD-

> info mgr detail

Manager is running (IP port host01.example.com.

,Process ID

).

GGSCI (TEST-BDD-

4、需要配置目标端MGR，才能回过来进行源端的EXTRACT的下一步操作。因此，开始安装目标端的OGG FOR BIGDATA。

$mkdir ogg

$tar xvpf ../ggs_Adapters_Linux_x64.tar

$./ggsci

GGSCI (NM-ITC-NF8460M3-

> create subdirs

GGSCI (NM-ITC-NF8460M3-

> edit params mgr

PORT

DYNAMICPORTLIST

AUTORESTART REPLICAT *,RETRIES

,WAITMINUTES

,RESETMINUTES

PURGEOLDEXTRACTS ./dirdat/*,usecheckpoints,minkeepdays

LAGREPORTHOURS

LAGINFOMINUTES

GGSCI (NM-ITC-NF8460M3-

> start manager

Manager started.

5、回到源端进行初始化EXTRACT，并在目标端生成文件。

$vi dirprm/ini_ext.prm

SOURCEISTABLE

userid ogg,password ogg

RMTHOST

10.142

78.36

,MGRPORT

RMTFILE /usr/bdusr01/ogg/dirdat/initld,megabytes

,purge

table ggtest.*;

-bash-

4.1

$  ./extract paramfile dirprm/ini_ext.prm reportfile dirrpt/ini_ext.rpt

6、切到目标端，查看是否生成文件

[bdusr01

@NM

-ITC-NF8460M3-

ogg]$ ls dirdat

initld

7、在源端配置extract。

$./ggsci

Oracle GoldenGate Command Interpreter

for

Oracle

Version

12.2

0.1

OGGCORE_12.

2.0

.0_PLATFORMS_151211.1401_FBO

Linux,x64,64bit (optimized),Oracle 12c on Dec

Operating system character set identified as UTF-

Copyright (C)

,Oracle and/or its affiliates. All rights reserved.

GGSCI (TEST-BDD-

> edit params ggext

extract ggext

userid ogg,password ogg

--RMTHOSTOPTIONS

RMTHOSTsandbox,MGRPORT

RMTFILE  /usr/bdusr01/ogg/dirdat/or,MEGABYTES

,PURGE

DDL include objname ogg.*

TABLE ogg.*;

GGSCI (TEST-BDD-

> dblogin userid ogg,password ogg

Successfully logged into database.

GGSCI (TEST-BDD-

> register extract GGEXT database

INFOOGG-

Extract GGEXT successfully registered with database at SCN

17151616

GGSCI (TEST-BDD-

> add extract ggext,INTEGRATED TRANLOG,BEGIN NOW

EXTRACT (Integrated)added.

8、在目标端，编辑hdfs.props。

[bdusr01

@NM

-ITC-NF8460M3-

ogg]$ vi dirprm/hdfs.props

gg.handlerlist=hdfs

gg.handler.hdfs.type=hdfs

gg.handler.hdfs.includeTokens=

true

gg.handler.hdfs.maxFileSize=256m

gg.handler.hdfs.rootFilePath=/data/hjpt/test/jl

gg.handler.hdfs.fileRollInterval=

gg.handler.hdfs.inactivityRollInterval=

gg.handler.hdfs.fileSuffix=.txt

gg.handler.hdfs.partitionByTable=

true

gg.handler.hdfs.rollOnMetadataChange=

true

#gg.handler.hdfs.authType=none

gg.handler.hdfs.authType=kerberos

gg.handler.hdfs.kerberosKeytabFile=/usr/bdusr01/bdusr01.keytab

gg.handler.hdfs.kerberosPrincipal=krbtgt/HADOOP.CHINATELECOM.CN

@HADOOP

.CHINATELECOM.CN

gg.handler.hdfs.format=delimitedtext

gg.handler.hdfs.format.includeColumnNames=

false

gg.handler.hdfs.mode=tx

gg.handler.hdfs.format.includeOpTimestamp=

true

gg.handler.hdfs.format.includeCurrentTimestamp=

true

gg.handler.hdfs.format.insertOpKey=I

gg.handler.hdfs.format.updateOpKey=U

gg.handler.hdfs.format.deleteOpKey=D

gg.handler.hdfs.format.truncateOpKey=T

gg.handler.hdfs.format.encoding=UTF-

gg.handler.hdfs.format.fieldDelimiter=CDATA[\u0001]

#gg.handler.hdfs.format.fieldDelimiter=

'|'

gg.handler.hdfs.format.lineDelimiter=CDATA[\n]

gg.handler.hdfs.format.includeTableName=

true

gg.handler.hdfs.format.keyValueDelimiter=CDATA[=]

#gg.handler.hdfs.format.kevValuePairDelimiter=CDATA[,]

gg.handler.hdfs.format.pkUpdateHandling=update

gg.handler.hdfs.format.nullValueRepresentation=

gg.handler.hdfs.format.missingValueRepresentation=CDATA[]

gg.handler.hdfs.format.includePosition=

false

goldengate.userexit.timestamp=utc

goldengate.userexit.writers=javawriter

javawriter.stats.display=TRUE

javawriter.stats.full=TRUE

gg.log=log4j

gg.log.level=INFO

gg.report.time=30sec

gg.classpath=/etc/hadoop/conf.empty:/usr/lib/hadoop/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop/lib/*

javawriter.bootoptions=-Xmx8200m –Xms8192m -Djava.

class

.path=ggjava/ggjava.jar

9、在目标端配置JAVA_HOME,LD_LIBRARY_PATH，必需要JAVA 1.7，不然后面运行可能出错。

export JAVA_HOME=/usr/lib/jvm/java-

1.7

-openjdk-

1.7

0.45

.x86_64

export LD_LIBRARY_PATH=/usr/lib/jvm/java-

1.7

-openjdk-

1.7

0.45

.x86_64/jre/lib/amd64:/usr/lib/jvm/java-

1.7

-openjdk-

1.7

0.45

.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-

1.7

-openjdk-

1.7

0.45

.x86_64/jre/lib/amd64/libjsig.so:/usr/lib/jvm/java-

1.7

-openjdk-

1.7

0.45

.x86_64/jre/lib/amd64/server/libjvm.so

10、在目标端初始化hdfs

[bdusr01

@NM

-ITC-NF8460M3-

ogg]$./replicat paramfile dirprm/irhdfs.prm reportfile dirrpt/ini_rhdfs.rpt

[bdusr01

@NM

-ITC-NF8460M3-

ogg]$ hadoop dfs -ls /data/hjpt/test/jl

Found

items

...

10、在源端启动extract

GGSCI (TEST-BDD-

>start extract ggext

Sending START request to MANAGER ...

EXTRACT GGEXT starting

11、在目标端配置REPLICAT。

[bdusr01

@NM

-ITC-NF8460M3-

ogg]$ vi dirprm/rhdfs.prm

REPLICAT rhdfs

setenv HADOOP_COMMON_LIB_NATIVE_DIR=/usr/lib/hadoop/lib/

native

DDL include all

TARGETDB LIBFILE libggjava.so SET property=dirprm/hdfs.props

REPORTCOUNT EVERY

MINUTES,RATE

GROUPTRANSOPS

MAP ogg.*,TARGET bdtest.*;

[bdusr01

@NM

-ITC-NF8460M3-

ogg]$./ggsci

GGSCI (NM-ITC-NF8460M3-

> add replicat rhdfs,exttrail dirdat/or

REPLICAT added.

GGSCI (NM-ITC-NF8460M3-

> start replicat rhdfs

Sending START request to MANAGER ...

REPLICAT RHDFS starting

GGSCI (NM-ITC-NF8460M3-

12、目标端HIVE建表

hive> CREATE EXTERNAL TABLE BDTEST.TEST_TAB_1  (tran_flag string,tab_name string,tran_time_utc timestamp,tran_time_loc string,something string,something1 string,

> PK_ID INT,RND_STR VARCHAR(

),USE_DATE string,RND_STR_1 string,ACC_DATE string)

> stored as textfile location

'/data/hjpt/test/jl'

OK

Time taken:

0.327

seconds

hive> select * from BDTEST.TEST_TAB_1;

...

13、在源端插入数据，再到目标端查看变化，即可。

参考文档：

http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/goldengate/12c/OGG12c_Integrated_Replicat/index.html

http://www.tuicool.com/articles/IzqAZnI

内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理

标签： Oracle Golden Gate

相关文章推荐

新的分享

章节导航