Compression technique in Sqoop:

 Compression technique in Sqoop:

While saving data to HDFS, decrease the overall size occupied on HDFS by using the compression technique. Mainly 3 compression techniques are available in sqoop. 1. Gzip Compression 2. Snappy Compression 3. Bzip2 Compression --compress is used to enable the compression. --compression-codec is used with a specific compression algorithm. Compression technique in Sqoop:
When using the --compress parameter in sqoop command, output files will be compressed using the Gzip codec, and all files will end up with a .gz extension. Gzip Files Extension .gz. Bzip2 Files Extension .bz2. Snappy Files Extension .snappy. Gzip Compression Speed medium. Bzip2 Compression Speed slow. Snappy Compression Speed fast.
if 100GB size medium 50 ,high 40)
Gzip Degree of Compression medium. Bzip2 Degree of Compression high. Snappy Degree of Compression medium. You need to make sure the compressed map output is allowed in your Hadoop configuration file. Configure Compression technique in Sqoop:

in core-site.xml find below propoeties In mapred-site.xml: name: mapreduce.map.output.compress value: true name: mapreduce.map.output.compress.codec value: org.apache.hadoop.io.compress.GzipCodec name: mapreduce.map.output.compress.codec value: org.apache.hadoop.io.compress.SnappyCodec name: mapreduce.map.output.compress.codec value: org.apache.hadoop.io.compress.BZip2Codec Gzip Compressing technique: Compress imported data with Gzip compression technique. By default sqoop uses GzipCodec compress technique. To enable compress, parameter is: --compress 1.1 . Gzip Compression using --compress
if we give --compress its Gzip compress in sqoop. Example : $sqoop import --connect jdbc:mysql://localhost/database-name --username root --password mypassword --table cities --target-dir /user/YT/cities_gz --compress
1.2. Gzip Compression using --compression-codec GzipCodec Example : $sqoop import --connect jdbc:mysql://localhost/database-name --username root --password mypassword --table cities --target-dir /user/YT/cities_gz --compression-codec org.apache.hadoop.io.compress.GzipCodec -m 1
1.3 Snappy Compressing technique: Compress imported data with a snappy compression technique. Example : $sqoop import --connect jdbc:mysql://localhost/database-name --username root --password mypassword --table cities --target-dir /user/YT/cities_snappy --compression-codec org.apache.hadoop.io.compress. SnappyCodec 1.4 Bzip2 Compressing technique:
Compress imported data with Bzip2 compression technique. Example : $sqoop import --connect jdbc:mysql://localhost/database-name --username root --password mypassword --table cities --target-dir /user/YT/cities_bz2 --compression-codec org.apache.hadoop.io.compress. BZip2Codec

Comments

Post a Comment

Popular posts from this blog

SQOOP file format

  How many file  formats  in Sqoop?   Sqoop supports the following 4 file  formats  for import operation.    1. Text file format.    2. Avro file format.    3. Sequence file format.    4. Parquet file format.    1st is  text file format  and  other three are binary file format .    Binary formats are Avro, Sequence and Parquet file.     Text file format in Sqoop:     1. Text file format: Apache Sqoop uses text file format as the default file format for importing the data from SQL to HDFS.    If we want to import data into text file then we need to specify --as- textfile  in  sqoop  command.  
Read more

Why do we use $CONDITIONS in Apache Sqoop?

Usage of Sqoop:  By default 4 mapper running in sqoop. When we are going to import data and if a table has a primary key then by default 4 mapper otherwise 1 mapper runs. If more than 1 mapper will run then sqoop distributes the data equally among the mappers to get high performance. Number of mappers = Number of part files on the HDFS.
Read more

Sqoop where condition , Sqoop join two tables

  1. Import data from MySQL using JOIN in Sqoop Example: Sqoop import --connect jdbc:mysql://localhost/sales --username root --password password --target-dir /user/YT/joinedData --query `select s.name, d.name from students s join department d on d.dept_id = s.dept_id where s.country_name = ‘India’ and $CONDITIONS order by d.name` -m 2 Above SQL will display all those student who belongs to India with the student name and department name. 2. Where Clause without query in Sqoop: Here we are not using the query, using only table name and where clause. It will return all the matching rows. Example: Sqoop import --connect jdbc:mysql://localhost/sales --username root --password password --table students --target-dir /user/YT/studentsCountryWiseData --where country_name = ‘India’ -m 1
Read more