[Hive - LanguageManual ] Explain (待)
2015-01-26 12:49
281 查看
EXPLAIN Syntax
The use of
A Hive query gets converted into a sequence (it is more an Directed Acyclic Graph) of stages. These stages may be map/reduce stages or they may even be stages that do metastore or file system operations like move and rename. The explain output comprises of three parts:
The Abstract Syntax Tree for the query
The dependencies between the different stages of the plan
The description of each of the stages
The description of the stages itself shows a sequence of operators with the metadata associated with the operators. The metadata may comprise of things like filter expressions for the FilterOperator or the select expressions for the SelectOperator or the output file names for the FileSinkOperator.
As an example, consider the following
The output of this statement contains the following parts:
The Abstract Syntax Tree
The Dependency Graph
This shows that Stage-1 is the root stage, Stage-2 is executed after Stage-1 is done and Stage-0 is executed after Stage-2 is done.
The plans of each Stage
In this example there are 2 map/reduce stages (Stage-1 and Stage-2) and 1 File System related stage (Stage-0). Stage-0 basically moves the results from a temporary directory to the directory corresponding to the table dest_g1.
A map/reduce stage itself comprises of 2 parts:
A mapping from table alias to Map Operator Tree - This mapping tells the mappers which operator tree to call in order to process the rows from a particular table or result of a previous map/reduce stage. In Stage-1 in the above example, the rows from src table are processed by the operator tree rooted at a Reduce Output Operator. Similarly, in Stage-2 the rows of the results of Stage-1 are processed by another operator tree rooted at another Reduce Output Operator. Each of these Reduce Output Operators partitions the data to the reducers according to the criteria shown in the metadata.
A Reduce Operator Tree - This is the operator tree which processes all the rows on the reducer of the map/reduce job. In Stage-1 for example, the Reducer Operator Tree is carrying out a partial aggregation where as the Reducer Operator Tree in Stage-2 computes the final aggregation from the partial aggregates computed in Stage-1
The use of
the following output is produced:
The inputs contain both the tables and the partitions. Note that the table is present even if none of the partitions is accessed in the query.
The dependencies show the parents in case a table is accessed via a view. Consider the following queries:
The following output is produced:
As above, the inputs contain the view V1 and the table 'src' that the view V1 refers to.
All the outputs are shown if a table is being accessed via multiple parents.
The following output is produced.
As can be seen, src is being accessed via parents v1 and v4.
The use of
the following output is produced:
With the
EXPLAIN Syntax
Hive provides anEXPLAINcommand that shows the execution plan for a query. The syntax for this statement is as follows:
AUTHORIZATIONis supported from HIVE 0.14.0 via HIVE-5961.[/code]
The use of
EXTENDEDin the
EXPLAINstatement produces extra information about the operators in the plan. This is typically physical information like file names.
A Hive query gets converted into a sequence (it is more an Directed Acyclic Graph) of stages. These stages may be map/reduce stages or they may even be stages that do metastore or file system operations like move and rename. The explain output comprises of three parts:
The Abstract Syntax Tree for the query
The dependencies between the different stages of the plan
The description of each of the stages
The description of the stages itself shows a sequence of operators with the metadata associated with the operators. The metadata may comprise of things like filter expressions for the FilterOperator or the select expressions for the SelectOperator or the output file names for the FileSinkOperator.
As an example, consider the following
EXPLAINquery:
The Abstract Syntax Tree
The plans of each Stage
A map/reduce stage itself comprises of 2 parts:
A mapping from table alias to Map Operator Tree - This mapping tells the mappers which operator tree to call in order to process the rows from a particular table or result of a previous map/reduce stage. In Stage-1 in the above example, the rows from src table are processed by the operator tree rooted at a Reduce Output Operator. Similarly, in Stage-2 the rows of the results of Stage-1 are processed by another operator tree rooted at another Reduce Output Operator. Each of these Reduce Output Operators partitions the data to the reducers according to the criteria shown in the metadata.
A Reduce Operator Tree - This is the operator tree which processes all the rows on the reducer of the map/reduce job. In Stage-1 for example, the Reducer Operator Tree is carrying out a partial aggregation where as the Reducer Operator Tree in Stage-2 computes the final aggregation from the partial aggregates computed in Stage-1
The use of
DEPENDENCYin the
EXPLAINstatement produces extra information about the inputs in the plan. It shows various attributes for the inputs. For example, for a query like:
The dependencies show the parents in case a table is accessed via a view. Consider the following queries:
All the outputs are shown if a table is being accessed via multiple parents.
The use of
AUTHORIZATIONin the
EXPLAINstatement shows all entities needed to be authorized to execute the query and authorization failures if exists. For example, for a query like:
FORMATTEDkeyword, it will be returned in JSON format.
相关文章推荐
- [Hive - LanguageManual] Archiving for File Count Reduction
- [Hive - LanguageManual] DML: Load, Insert, Update, Delete
- [HIve - LanguageManual] Hive Operators and User-Defined Functions (UDFs)
- [Hive - LanguageManual] Import/Export
- [HIve - LanguageManual] XPathUDF
- [HIve - LanguageManual] Joins
- [HIve - LanguageManual] Join Optimization (不懂)
- [Hive - LanguageManual] Select base use
- [HIve - LanguageManual] Union
- [Hive - LanguageManual] GroupBy
- [HIve - LanguageManual] LateralView
- [HIve - LanguageManual] Subqueries
- [Hive - LanguageManual] Sampling
- [Hive - LanguageManual] Create/Drop/Alter Database Create/Drop/Truncate Table
- [Hive - LanguageManual] VirtualColumns
- [Hive - LanguageManual] Alter Table/Partition/Column
- [Hive - LanguageManual ] Windowing and Analytics Functions (待)
- [Hive - LanguageManual] Create/Drop/Alter -View、 Index 、 Function
- [Hive - LanguageManual] Create/Drop/Grant/Revoke Roles and Privileges / Show Use
- [Hive - LanguageManual] Hive Concurrency Model (待)