Hazards of Small Files and How to Handle Them

The Hazards of Small Files and How to Handle Them

1. Handling Small Files in HDFS

Impact:

Storage Layer

One file block takes up approximately 150 bytes in the NameNode, and 128G can store roughly 900 million file blocks (small files consume a significant amount of this).

Compute Layer

Each small file generates a MapTask, consuming a large amount of computational resources.

Solutions:

Use the HAR archiving method to archive small files;
Use CombineTextInputFormat;
Enable JVM reuse in scenarios with small files; if there are no small files, do not enable JVM reuse, as it will keep occupying the task slots in use until the job completes.

JVM reuse allows a JVM instance to be reused N times within the same job, and the value of N can be configured in Hadoop's mapred-site.xml file. It is typically set between 10 and 20.

<property>

  <name>mapreduce.job.jvm.numtasks</name>

  <value>10</value>

  <description>
  How many tasks to run per jvm,if set to -1 ,there is  no limit
  </description>

</property>

2. Small File Issues in Hive

Issue 1:

There are too many input files, and Hive has a limit on the total number of files it can create. This limit is determined by the parameter: hive.exec.max.created.files, with a default value of 10000. If your table currently has 60 partitions and you have a total of 2000 mappers, during execution, each mapper will create 60 files, one for each partition. So 60*2000 > 120000, which will trigger an error: exceeds 100000.Killing the job. The simplest solution is to increase the hive.exec.max.created.files parameter. However, if the data files only total 400G, and you adjust this parameter to, say, 40000, then on average each file would be about 10.24MB. This would result in over 40000 small files, which is not a good thing.

Solution 1:

Set the mapper input file merging parameters 
-- Merge small files before the mapper executes

hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;

-- Maximum input size for each mapper

mapred.max.split.size = 256000000

-- Minimum split size on a node (this value determines whether files on multiple DataNodes need to be merged)

mapred.min.split.size.per.node = 100000000

-- Minimum split size under a rack (this value determines whether files under that rack need to be merged)

mapred.min.split.size.per.rack = 100000000

Issue 2:

Too many files are generated during the intermediate process of Hive execution.

Solution 2:

Set the intermediate process merging parameters 

-- Merge small files at the end of Map-only tasks

hive.merge.mapfiles = true

-- Merge small files at the end of Map-Reduce tasks

hive.merge.mapredfiles = true

-- Size of merged files

hive.merge.size.per.task = 25610001000

-- When the average size of output files is smaller than this value, start a separate map-reduce task to merge files

hive.merge.smallfiles.avgsize=16000000

Issue 3:

Too many result files in Hive.

Solution 3:
 
 Set the reducer parameters (one approach is to adjust the number of reducers, and the other is to adjust the reducer size)
 -- Set the number of reducers

set mapreduce.job.reduces=500;
insert into table xxx
select
  *
from
 xxx
distribute by rand();

-- Set the reducer size
set hive.exec.reducers.bytes.per.reducer=5120000000;
insert into table xxx
select
  *
from
 xxx
distribute by rand();

-- distribute by rand() ensures that the data in reducers is randomly distributed and roughly equal in size