Sqoop --target-dir & ---warehouse-dir

 Sqoop store imported data by 2 way:

When we import record from RDBMS to HDFS using Sqoop then Sqoop use two way to store imported record on HDFS. 1. --target-dir directory path here 2. --warehouse-dir directory path here Import data using --target-dir in Sqoop: By using --target-dir path of directory, Sqoop create a directory which work as table name (directory path) with imported files. Example -1: Sqoop import --connect jdbc:mysql://localhost/sales --username root --password password --table students --target-dir /user/YT/tarData --where country_name = ‘India’ -m 1 This import statement stored all those students in target directory location, who belongs to India.

Import data using --warehouse-dir in Sqoop: Sqoop create a directory which works as database directory and table name directory automatically created with imported files with in warehouse dir . So warehouse directory work like database directory for importing table. Example: Sqoop import --connect jdbc:mysql://localhost/sales --username root --password sqoop_password --table students -m 1 --warehouse-dir /salesDB --where country_name = ‘India’ --field-terminated-by ‘\t’ It create a salesDB directory at root (along with user) and put students directory with in salesDB. ~]$ hadoop fs –lsr /salesDB Output of the above command will be as follow: Found 3 items drw-r--r--  3 YT     0 2020-04-11 23:15 /salesDB/students-rw-r--r--   3 YT      0 2020-04-10 23:11 /salesDB/students/_SUCCESS-rw-r--r--   3 YT 110 2020-04-10 23:11 /salesDB/students/part-m-00000

Comments

Post a Comment

Popular posts from this blog

SQOOP file format

  How many file  formats  in Sqoop?   Sqoop supports the following 4 file  formats  for import operation.    1. Text file format.    2. Avro file format.    3. Sequence file format.    4. Parquet file format.    1st is  text file format  and  other three are binary file format .    Binary formats are Avro, Sequence and Parquet file.     Text file format in Sqoop:     1. Text file format: Apache Sqoop uses text file format as the default file format for importing the data from SQL to HDFS.    If we want to import data into text file then we need to specify --as- textfile  in  sqoop  command.  
Read more

Why do we use $CONDITIONS in Apache Sqoop?

Usage of Sqoop:  By default 4 mapper running in sqoop. When we are going to import data and if a table has a primary key then by default 4 mapper otherwise 1 mapper runs. If more than 1 mapper will run then sqoop distributes the data equally among the mappers to get high performance. Number of mappers = Number of part files on the HDFS.
Read more

Sqoop where condition , Sqoop join two tables

  1. Import data from MySQL using JOIN in Sqoop Example: Sqoop import --connect jdbc:mysql://localhost/sales --username root --password password --target-dir /user/YT/joinedData --query `select s.name, d.name from students s join department d on d.dept_id = s.dept_id where s.country_name = ‘India’ and $CONDITIONS order by d.name` -m 2 Above SQL will display all those student who belongs to India with the student name and department name. 2. Where Clause without query in Sqoop: Here we are not using the query, using only table name and where clause. It will return all the matching rows. Example: Sqoop import --connect jdbc:mysql://localhost/sales --username root --password password --table students --target-dir /user/YT/studentsCountryWiseData --where country_name = ‘India’ -m 1
Read more