Day#2 Sqoop: Import data from MySql into HDFS

1) Download Sqoop

Download Apache Sqoop from sqoop.apache.org

$sudo mkdir /usr/local/sqoop
$cd $HOME/Downloads
$cd  sqoop-1.4.4.bin__hadoop-1.0.0
$sudo cp -R * /usr/local/sqoop
$cd /usr/local/
$sudo chown -R butik sqoop

2) Setup Sqoop home

$cd $HOME
$vi .bashrc

$exec bash  # export values

3) Set configuration files

$cd $SQOOP_PREFIX/conf

Edit sqoop-env.sh

export HADOOP_COMMON_HOME=/usr/local/hadoop

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/usr/local/hadoop

#set the path to where bin/hbase is available
#export HBASE_HOME=/usr/local/hbase

#Set the path to where bin/hive is available
#export HIVE_HOME=/usr/local/hive


4) install mySql and mysql Driver



$sudo apt-get install mysql-server

When prompted give password for user root.

 
$sudo apt-get install libmysql-java



5) create mysql database and tables

Connect to mysql and crate objects
$mysql -u root -p

mysql>create database chunky;
mysql>use chunky;

6 ) copy mysql driver jar into Sqoop library

Link or copy mysql jar to Sqoop lib

$cp mysql-connector-java-5.1.28.jar  /usr/local/sqoop/lib/.

7) Connect to Mysql from Sqoop 


$sqoop list-tables --connect jdbc:mysql://localhost/chunky  --username root  -P

$sqoop list-databases --connect jdbc:mysql://localhost --username root  -P

8) Import Mysql Table data into HDFS

$sqoop import --connect jdbc:mysql://localhost/chunky --table currency --fields-terminated-by '\t' --username root -P

9) Check the files in HDFS