Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Modeler   Code  
»áÔ±   
 
   
 
 
     
   
 ¶©ÔÄ
  ¾èÖú
Spark SQL CatalystÔ´Âë·ÖÎöÖ®Analyzer
 
×÷Õß OopsOutOfMemory £¬»ðÁú¹ûÈí¼þ    ·¢²¼ÓÚ 2014-08-21
  3295  次浏览      28
 

Ç°Ãæ¼¸ÆªÎÄÕ½²½âÁËSpark SQLµÄºËÐÄÖ´ÐÐÁ÷³ÌºÍSpark SQLµÄCatalyst¿ò¼ÜµÄSql ParserÊÇÔõÑù½ÓÊÜÓû§ÊäÈësql£¬¾­¹ý½âÎöÉú³ÉUnresolved Logical PlanµÄ¡£ÎÒÃǼǵÃSpark SQLµÄÖ´ÐÐÁ÷³ÌÖÐÁíÒ»¸öºËÐĵÄ×é¼þʽAnalyzer£¬±¾ÎĽ«»á½éÉÜAnalyzerÔÚSpark SQLÀïÆðµ½ÁËʲô×÷Óá£

AnalyzerλÓÚCatalystµÄanalysis packageÏ£¬Ö÷ÒªÖ°ÔðÊǽ«Sql Parser δÄÜResolvedµÄLogical Plan ¸øResolvedµô¡£

Ò»¡¢Analyzer¹¹Ôì

Analyzer»áʹÓÃCatalogºÍFunctionRegistry½«UnresolvedAttributeºÍUnresolvedRelationת»»ÎªcatalystÀïÈ«ÀàÐ͵ĶÔÏó¡£
AnalyzerÀïÃæÓÐfixedPoint¶ÔÏó£¬Ò»¸öSeq[Batch].

class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Boolean)  
extends RuleExecutor[LogicalPlan] with HiveTypeCoercion {

// TODO: pass this in as a parameter.
val fixedPoint = FixedPoint(100)

val batches: Seq[Batch] = Seq(
Batch("MultiInstanceRelations", Once,
NewRelationInstances),
Batch("CaseInsensitiveAttributeReferences", Once,
(if (caseSensitive) Nil else LowercaseAttributeReferences :: Nil) : _*),
Batch("Resolution", fixedPoint,
ResolveReferences ::
ResolveRelations ::
NewRelationInstances ::
ImplicitGenerate ::
StarExpansion ::
ResolveFunctions ::
GlobalAggregates ::
typeCoercionRules :_*),
Batch("AnalysisOperators", fixedPoint,
EliminateAnalysisOperators)
)

AnalyzerÀïµÄһЩ¶ÔÏó½âÊÍ£º

FixedPoint£ºÏ൱ÓÚµü´ú´ÎÊýµÄÉÏÏÞ¡£

/** A strategy that runs until fix point or maxIterations times, whichever comes first. */  
case class FixedPoint(maxIterations: Int) extends Strategy

Batch: Åú´Î£¬Õâ¸ö¶ÔÏóÊÇÓÉһϵÁÐRule×é³ÉµÄ£¬²ÉÓÃÒ»¸ö²ßÂÔ£¨²ßÂÔÆäʵÊǵü´ú¼¸´ÎµÄ±ðÃû°É£¬eg£ºOnce£©

/** A batch of rules. */£¬  
protected case class Batch(name: String, strategy: Strategy, rules: Rule[TreeType]*)

Rule£ºÀí½âΪһÖÖ¹æÔò£¬ÕâÖÖ¹æÔò»áÓ¦Óõ½Logical Plan ´Ó¶ø½«UnResolved ת±äΪResolved

abstract class Rule[TreeType <: TreeNode[_]] extends Logging {  

/** Name for this rule, automatically inferred based on class name. */
val ruleName: String = {
val className = getClass.getName
if (className endsWith "$") className.dropRight(1) else className
}

def apply(plan: TreeType): TreeType
}

Strategy£º×î´óµÄÖ´ÐдÎÊý£¬Èç¹ûÖ´ÐдÎÊýÔÚ×î´óµü´ú´ÎÊý֮ǰ¾Í´ïµ½ÁËfix point£¬²ßÂԾͻáÍ£Ö¹£¬²»ÔÙÓ¦ÓÃÁË¡£

/** 
* An execution strategy for rules that indicates the maximum number of executions. If the
* execution reaches fix point (i.e. converge) before maxIterations, it will stop.
*/
abstract class Strategy { def maxIterations: Int }

Analyzer½âÎöÖ÷ÒªÊǸù¾ÝÕâЩBatchÀïÃæ¶¨ÒåµÄ²ßÂÔºÍRuleÀ´¶ÔUnresolvedµÄÂß¼­¼Æ»®½øÐнâÎöµÄ¡£

ÕâÀïAnalyzerÀà±¾Éí²¢Ã»Óж¨ÒåÖ´Ðеķ½·¨£¬¶øÊÇÒª´ÓËüµÄ¸¸ÀàRuleExecutor[LogicalPlan]ѰÕÒ£¬AnalyzerҲʵÏÖÁËHiveTypeCosercion£¬Õâ¸öÀàÊDzο¼HiveµÄÀàÐÍ×Ô¶¯¼æÈÝת»»µÄÔ­Àí¡£Èçͼ£º

RuleExecutor£ºÖ´ÐÐRuleµÄÖ´Ðл·¾³£¬Ëü»á½«°üº¬ÁËһϵÁеÄRuleµÄBatch½øÐÐÖ´ÐУ¬Õâ¸ö¹ý³Ì¶¼ÊÇ´®Ðеġ£

¾ßÌåµÄÖ´Ðз½·¨¶¨ÒåÔÚapplyÀ

¿ÉÒÔ¿´µ½ÕâÀïÊÇÒ»¸öwhileÑ­»·£¬Ã¿¸öbatchϵÄrules¶¼¶Ôµ±Ç°µÄplan½øÐÐ×÷Óã¬Õâ¸ö¹ý³ÌÊǵü´úµÄ£¬Ö±µ½´ïµ½Fix Point»òÕß×î´óµü´ú´ÎÊý¡£

def apply(plan: TreeType): TreeType = {  
var curPlan = plan

batches.foreach { batch =>
val batchStartPlan = curPlan
var iteration = 1
var lastPlan = curPlan
var continue = true

// Run until fix point (or the max number of iterations as specified in the strategy.
while (continue) {
curPlan = batch.rules.foldLeft(curPlan) {
case (plan, rule) =>
val result = rule(plan) //ÕâÀォµ÷Óø÷¸ö²»Í¬RuleµÄapply·½·¨£¬½«UnResolved Relations£¬AttrubuteºÍFunction½øÐÐResolve
if (!result.fastEquals(plan)) {
logger.trace(
s"""
|=== Applying Rule ${rule.ruleName} ===
|${sideBySide(plan.treeString, result.treeString).mkString("\n")}
""".stripMargin)
}

result //·µ»Ø×÷ÓúóµÄresult plan
}
iteration += 1
if (iteration > batch.strategy.maxIterations) { //Èç¹ûµü´ú´ÎÊýÒѾ­´óÓڸòßÂÔµÄ×î´óµü´ú´ÎÊý£¬¾Íֹͣѭ»·
logger.info(s"Max iterations ($iteration) reached for batch ${batch.name}")
continue = false
}

if (curPlan.fastEquals(lastPlan)) { //Èç¹ûÔÚ¶à´Îµü´úÖв»Ôٱ仯£¬ÒòΪplanÓиöunique id£¬¾Íֹͣѭ»·¡£
logger.trace(s"Fixed point reached for batch ${batch.name} after $iteration iterations.")
continue = false
}
lastPlan = curPlan
}

if (!batchStartPlan.fastEquals(curPlan)) {
logger.debug(
s"""
|=== Result of Batch ${batch.name} ===
|${sideBySide(plan.treeString, curPlan.treeString).mkString("\n")}
""".stripMargin)
} else {
logger.trace(s"Batch ${batch.name} has no effect.")
}
}

curPlan //·µ»ØResolvedµÄLogical Plan
}

¶þ¡¢Rules½éÉÜ

ĿǰSpark SQL 1.0.0µÄRule¶¼¶¨ÒåÔÚÁËAnalyzer.scalaµÄÄÚ²¿Àà¡£

ÔÚbatchesÀïÃæ¶¨ÒåÁË4¸öBatch¡£

MultiInstanceRelations¡¢CaseInsensitiveAttributeReferences¡¢Resolution¡¢AnalysisOperators Ëĸö¡£

Õâ4¸öBatchÊǽ«²»Í¬µÄRule½øÐйéÀ࣬ÿÖÖÀà±ð²ÉÓò»Í¬µÄ²ßÂÔÀ´½øÐÐResolve¡£

2.1¡¢MultiInstanceRelation

Èç¹ûÒ»¸öʵÀýÔÚLogical PlanÀï³öÏÖÁ˶à´Î£¬Ôò»áÓ¦ÓÃNewRelationInstancesÕâ¶ùRule

Batch("MultiInstanceRelations", Once,  
NewRelationInstances)

trait MultiInstanceRelation {  
def newInstance: this.type
}

object NewRelationInstances extends Rule[LogicalPlan] {   
def apply(plan: LogicalPlan): LogicalPlan = {
val localRelations = plan collect { case l: MultiInstanceRelation => l} //½«logical planÓ¦ÓÃpartial functionµÃµ½ËùÓÐMultiInstanceRelationµÄplanµÄ¼¯ºÏ
val multiAppearance = localRelations
.groupBy(identity[MultiInstanceRelation]) //group by²Ù×÷
.filter { case (_, ls) => ls.size > 1 } //Èç¹ûֻȡsize´óÓÚ1µÄ½øÐкóÐø²Ù×÷
.map(_._1)
.toSet

//¸üÐÂplan£¬Ê¹µÃÿ¸öʵÀýµÄexpIdÊÇΨһµÄ¡£
plan transform {
case l: MultiInstanceRelation if multiAppearance contains l => l.newInstance
}
}
}

2.2¡¢LowercaseAttributeReferences

ͬÑùÊÇpartital function£¬¶Ôµ±Ç°planÓ¦Ó㬽«ËùÓÐÆ¥ÅäµÄÈçUnresolvedRelationµÄ±ðÃûaliseת»»ÎªÐ¡Ð´£¬½«SubqueryµÄ±ðÃûҲת»»ÎªÐ¡Ð´¡£
×ܽ᣺ÕâÊÇÒ»¸öʹÊôÐÔÃû´óСд²»Ãô¸ÐµÄRule£¬ÒòΪËü½«ËùÓÐÊôÐÔ¶¼to lower caseÁË¡£

object LowercaseAttributeReferences extends Rule[LogicalPlan] {  
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
case UnresolvedRelation(databaseName, name, alias) =>
UnresolvedRelation(databaseName, name, alias.map(_.toLowerCase))
case Subquery(alias, child) => Subquery(alias.toLowerCase, child)
case q: LogicalPlan => q transformExpressions {
case s: Star => s.copy(table = s.table.map(_.toLowerCase))
case UnresolvedAttribute(name) => UnresolvedAttribute(name.toLowerCase)
case Alias(c, name) => Alias(c, name.toLowerCase)()
case GetField(c, name) => GetField(c, name.toLowerCase)
}
}
}

2.3¡¢ResolveReferences

½«Sql parser½âÎö³öÀ´µÄUnresolvedAttributeÈ«²¿¶¼×ªÎª¶ÔÓ¦µÄʵ¼ÊµÄcatalyst.expressions.AttributeReference AttributeReferences
ÕâÀïµ÷ÓÃÁËlogical plan µÄresolve·½·¨£¬½«ÊôÐÔתΪNamedExepression¡£

object ResolveReferences extends Rule[LogicalPlan] {  
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
case q: LogicalPlan if q.childrenResolved =>
logger.trace(s"Attempting to resolve ${q.simpleString}")
q transformExpressions {
case u @ UnresolvedAttribute(name) =>
// Leave unchanged if resolution fails. Hopefully will be resolved next round.
val result = q.resolve(name).getOrElse(u)//ת»¯ÎªNamedExpression
logger.debug(s"Resolving $u to $result")
result
}
}
}

2.4¡¢ ResolveRelations

Õâ¸ö±È½ÏºÃÀí½â£¬»¹¼ÇµÃÇ°ÃæSql parserÂ𣬱ÈÈçselect * from src£¬Õâ¸ösrc±íparseºó¾ÍÊÇÒ»¸öUnresolvedRelation½Úµã¡£

ÕâÒ»²½ResolveRelationsµ÷ÓÃÁËcatalogÕâ¸ö¶ÔÏó¡£Catalog¶ÔÏóÀïÃæÎ¬»¤ÁËÒ»¸ötableName,Logical PlanµÄHashMap½á¹û¡£

ͨ¹ýÕâ¸öCatalogĿ¼À´Ñ°ÕÒµ±Ç°±íµÄ½á¹¹£¬´Ó¶ø´ÓÖнâÎö³öÕâ¸ö±íµÄ×ֶΣ¬ÈçUnResolvedRelations »áµÃµ½Ò»¸ötableWithQualifiers¡££¨¼´±íºÍ×ֶΣ©
ÕâÒ²½âÊÍÁËΪʲôÁ÷³ÌͼÄÇ£¬Îһửһ¸öcatalogÔÚÉÏÃæ£¬ÒòΪËüÊÇAnalyzer¹¤×÷ʱÐèÒªµÄmeta data¡£

object ResolveRelations extends Rule[LogicalPlan] {  
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
case UnresolvedRelation(databaseName, name, alias) =>
catalog.lookupRelation(databaseName, name, alias)
}
}

2.5¡¢ImplicitGenerate

Èç¹ûÔÚselectÓï¾äÀïÖ»ÓÐÒ»¸ö±í´ïʽ£¬¶øÇÒÕâ¸ö±í´ïʽÊÇÒ»¸öGenerator£¨GeneratorÊÇÒ»¸ö1Ìõ¼Ç¼Éú³Éµ½NÌõ¼Ç¼µÄÓ³É䣩

µ±ÔÚ½âÎöÂß¼­¼Æ»®Ê±£¬Óöµ½Project½ÚµãµÄʱºò£¬¾Í¿ÉÒÔ½«Ëüת»»ÎªGenerateÀࣨGenerateÀàÊǽ«ÊäÈëÁ÷Ó¦ÓÃÒ»¸öº¯Êý£¬´Ó¶øÉú³ÉÒ»¸öеÄÁ÷£©¡£

object ImplicitGenerate extends Rule[LogicalPlan] {  
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
case Project(Seq(Alias(g: Generator, _)), child) =>
Generate(g, join = false, outer = false, None, child)
}
}

2.6 StarExpansion

ÔÚProject²Ù×÷·ûÀÈç¹ûÊÇ*·ûºÅ£¬¼´select * Óï¾ä£¬¿ÉÒÔ½«ËùÓеÄreferences¶¼Õ¹¿ª£¬¼´½«select * ÖеÄ*Õ¹¿ª³Éʵ¼ÊµÄ×ֶΡ£

object StarExpansion extends Rule[LogicalPlan] {  
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
// Wait until children are resolved
case p: LogicalPlan if !p.childrenResolved => p
// If the projection list contains Stars, expand it.
case p @ Project(projectList, child) if containsStar(projectList) =>
Project(
projectList.flatMap {
case s: Star => s.expand(child.output) //Õ¹¿ª£¬½«ÊäÈëµÄAttributeexpand(input: Seq[Attribute]) ת»¯ÎªSeq[NamedExpression]
case o => o :: Nil
},
child)
case t: ScriptTransformation if containsStar(t.input) =>
t.copy(
input = t.input.flatMap {
case s: Star => s.expand(t.child.output)
case o => o :: Nil
}
)
// If the aggregate function argument contains Stars, expand it.
case a: Aggregate if containsStar(a.aggregateExpressions) =>
a.copy(
aggregateExpressions = a.aggregateExpressions.flatMap {
case s: Star => s.expand(a.child.output)
case o => o :: Nil
}
)
}
/**
* Returns true if `exprs` contains a [[Star]].
*/
protected def containsStar(exprs: Seq[Expression]): Boolean =
exprs.collect { case _: Star => true }.nonEmpty
}
}

2.7 ResolveFunctions

Õâ¸öºÍResolveReferences²î²»¶à£¬ÕâÀïÖ÷ÒªÊǶÔudf½øÐÐresolve¡£

½«ÕâЩUDF¶¼ÔÚFunctionRegistryÀï½øÐвéÕÒ¡£

object ResolveFunctions extends Rule[LogicalPlan] {  
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
case q: LogicalPlan =>
q transformExpressions {
case u @ UnresolvedFunction(name, children) if u.childrenResolved =>
registry.lookupFunction(name, children) //¿´ÊÇ·ñ×¢²áÁ˵±Ç°udf
}
}
}

2.8 GlobalAggregates

È«¾ÖµÄ¾ÛºÏ£¬Èç¹ûÓöµ½ÁËProject¾Í·µ»ØÒ»¸öAggregate.

object GlobalAggregates extends Rule[LogicalPlan] {  
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
case Project(projectList, child) if containsAggregates(projectList) =>
Aggregate(Nil, projectList, child)
}

def containsAggregates(exprs: Seq[Expression]): Boolean = {
exprs.foreach(_.foreach {
case agg: AggregateExpression => return true
case _ =>
})
false
}
}

2.9 typeCoercionRules

Õâ¸öÊÇHiveÀïµÄ¼æÈÝSQLÓï·¨£¬±ÈÈ罫StringºÍInt»¥Ïàת»»£¬²»ÐèÒªÏÔʾµÄµ÷ÓÃcast xxx as yyyÁË¡£ÈçStringToIntegerCasts¡£

val typeCoercionRules =  
PropagateTypes ::
ConvertNaNs ::
WidenTypes ::
PromoteStrings ::
BooleanComparisons ::
BooleanCasts ::
StringToIntegralCasts ::
FunctionArgumentConversion ::
CastNulls ::
Nil

2.10 EliminateAnalysisOperators

½«·ÖÎöµÄ²Ù×÷·ûÒÆ³ý£¬ÕâÀï½öÖ§³Ö2ÖÖ£¬Ò»ÖÖÊÇSubqueryÐèÒªÒÆ³ý£¬Ò»ÖÖÊÇLowerCaseSchema¡£ÕâЩ½Úµã¶¼»á´ÓLogical PlanÀïÒÆ³ý¡£

object EliminateAnalysisOperators extends Rule[LogicalPlan] {  
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
case Subquery(_, child) => child //Óöµ½Subquery,²»·´»Ú±¾Éí£¬·µ»ØËüµÄChild£¬¼´É¾³ýÁ˸ÃÔªËØ
case LowerCaseSchema(child) => child
}
}

Èý¡¢Êµ¼ù

²¹³ä×òÌìDEBUGµÄÒ»¸öÀý×Ó£¬Õâ¸öÀý×Ó֤ʵÁËÈçºÎ½«ÉÏÃæµÄ¹æÔòÓ¦Óõ½Unresolved Logical Plan£º

µ±´«µÝsqlÓï¾äµÄʱºò£¬µÄÈ·µ÷ÓÃÁËResolveReferences½«mobile½âÎö³ÉNamedExpression¡£

¿ÉÒÔ¶ÔÕÕÕâ¿´Ö´ÐÐÁ÷³Ì£¬×ó±ßÊÇUnresolved Logical Plan£¬ÓÒ±ßÊÇResoveld Logical Plan¡£

ÏÈÊÇÖ´ÐÐÁËBatch Resolution£¬eg: µ÷ÓÃResovelRalationÕâ¸öRUleÀ´Ê¹ Unresovled Relation ת»¯Îª SparkLogicalPlan²¢Í¨¹ýCatalogÕÒµ½ÁËÆä¶ÔÓÚµÄ×Ö¶ÎÊôÐÔ¡£

È»ºóÖ´ÐÐÁËBatch Analysis Operator¡£eg£ºµ÷ÓÃEliminateAnalysisOperatorsÀ´½«SubQuery¸øremoveµôÁË¡£

¿ÉÄܸñʽÏÔʾµÄ²»Ì«ºÃ£¬¿ÉÒÔÏòÓÒ±ßÍ϶¯Ï¹ö¶¯ÖῴϽá¹û¡£ £º£©

val exec = sqlContext.sql("select mobile as mb, sid as id, mobile*2 multi2mobile,
 count(1) times from (select * from temp_shengli_mobile)a where pfrom_id=0.0 group by mobile, sid,  mobile*2")  
14/07/21 18:23:32 DEBUG SparkILoop$SparkILoopInterpreter: Invoking: public static java.lang.String $line47.$eval.$print()
14/07/21 18:23:33 INFO Analyzer: Max iterations (2) reached for batch MultiInstanceRelations
14/07/21 18:23:33 INFO Analyzer: Max iterations (2) reached for batch CaseInsensitiveAttributeReferences
14/07/21 18:23:33 DEBUG Analyzer$ResolveReferences$: Resolving 'pfrom_id to pfrom_id#5
14/07/21 18:23:33 DEBUG Analyzer$ResolveReferences$: Resolving 'mobile to mobile#2
14/07/21 18:23:33 DEBUG Analyzer$ResolveReferences$: Resolving 'sid to sid#1
14/07/21 18:23:33 DEBUG Analyzer$ResolveReferences$: Resolving 'mobile to mobile#2
14/07/21 18:23:33 DEBUG Analyzer$ResolveReferences$: Resolving 'mobile to mobile#2
14/07/21 18:23:33 DEBUG Analyzer$ResolveReferences$: Resolving 'sid to sid#1
14/07/21 18:23:33 DEBUG Analyzer$ResolveReferences$: Resolving 'mobile to mobile#2
14/07/21 18:23:33 DEBUG Analyzer:
=== Result of Batch Resolution ===
!Aggregate ['mobile,'sid,('mobile * 2) AS c2#27], ['mobile AS mb#23,'sid AS id#24,('mobile * 2) AS multi2mobile#25,COUNT(1) AS times#26L] Aggregate [mobile#2,sid#1,(CAST(mobile#2, DoubleType) * CAST(2, DoubleType)) AS c2#27], [mobile#2 AS mb#23,sid#1 AS id#24,(CAST(mobile#2, DoubleType) * CAST(2, DoubleType)) AS multi2mobile#25,COUNT(1) AS times#26L]
! Filter ('pfrom_id = 0.0) Filter (CAST(pfrom_id#5, DoubleType) = 0.0)
Subquery a Subquery a
! Project [*] Project [data_date#0,sid#1,mobile#2,pverify_type#3,create_time#4, pfrom_id#5,p_status#6,pvalidate_time#7,feffect_time#8,plastupdate_ip#9,update_time#10,status#11,preserve_int#12]
! UnresolvedRelation None, temp_shengli_mobile, None Subquery temp_shengli_mobile
! SparkLogicalPlan (ExistingRdd [data_date#0,sid#1,mobile#2,pverify_type#3,create_time#4, pfrom_id#5,p_status#6,pvalidate_time#7,feffect_time#8,plastupdate_ip#9,update_time#10,status#11, preserve_int#12], MapPartitionsRDD[4] at mapPartitions at basicOperators.scala:174)
14/07/21 18:23:33 DEBUG Analyzer:
=== Result of Batch AnalysisOperators ===
!Aggregate ['mobile,'sid,('mobile * 2) AS c2#27], ['mobile AS mb#23,'sid AS id#24,('mobile * 2) AS multi2mobile#25,COUNT(1) AS times#26L] Aggregate [mobile#2,sid#1,(CAST(mobile#2, DoubleType) * CAST(2, DoubleType)) AS c2#27], [mobile#2 AS mb#23,sid#1 AS id#24,(CAST(mobile#2, DoubleType) * CAST(2, DoubleType)) AS multi2mobile#25,COUNT(1) AS times#26L]
! Filter ('pfrom_id = 0.0) Filter (CAST(pfrom_id#5, DoubleType) = 0.0)
! Subquery a Project [data_date#0,sid#1,mobile#2,pverify_type#3,create_time#4, pfrom_id#5,p_status#6,pvalidate_time#7,feffect_time#8,plastupdate_ip#9,update_time#10,status#11,preserve_int#12]
! Project [*] SparkLogicalPlan (ExistingRdd [data_date#0,sid#1,mobile#2,pverify_type#3, create_time#4,pfrom_id#5,p_status#6,pvalidate_time#7,feffect_time#8,plastupdate_ip#9, update_time#10,status#11,preserve_int#12], MapPartitionsRDD[4] at mapPartitions at basicOperators.scala:174)
! UnresolvedRelation None, temp_shengli_mobile, None

ËÄ¡¢×ܽá

±¾ÎÄ´ÓÔ´´úÂë½Ç¶È·ÖÎöÁËAnalyzerÔÚ¶ÔSql Parser½âÎö³öµÄUnResolve Logical Plan ½øÐÐanalyzeµÄ¹ý³ÌÖУ¬ËùÖ´ÐеÄÁ÷³Ì¡£

Á÷³ÌÊÇʵÀý»¯Ò»¸öSimpleAnalyzer£¬¶¨ÒåһЩBatch£¬È»ºó±éÀúÕâЩBatchÔÚRuleExecutorµÄ»·¾³Ï£¬Ö´ÐÐBatchÀïÃæµÄRules£¬Ã¿¸öRule»á¶ÔUnresolved Logical Plan½øÐÐResolve£¬ÓÐЩ¿ÉÄܲ»ÄÜÒ»´Î½âÎö³ö£¬ÐèÒª¶à´Îµü´ú£¬Ö±µ½´ïµ½maxµü´ú´ÎÊý»òÕß´ïµ½fix point¡£ÕâÀïRuleÀï±È½Ï³£ÓõľÍÊÇResolveReferences¡¢ResolveRelations¡¢StarExpansion¡¢GlobalAggregates¡¢typeCoercionRulesºÍEliminateAnalysisOperators¡£

   
3295 ´Îä¯ÀÀ       28
Ïà¹ØÎÄÕÂ

»ùÓÚEAµÄÊý¾Ý¿â½¨Ä£
Êý¾ÝÁ÷½¨Ä££¨EAÖ¸ÄÏ£©
¡°Êý¾Ýºþ¡±£º¸ÅÄî¡¢ÌØÕ÷¡¢¼Ü¹¹Óë°¸Àý
ÔÚÏßÉ̳ÇÊý¾Ý¿âϵͳÉè¼Æ ˼·+Ч¹û
 
Ïà¹ØÎĵµ

GreenplumÊý¾Ý¿â»ù´¡Åàѵ
MySQL5.1ÐÔÄÜÓÅ»¯·½°¸
ijµçÉÌÊý¾ÝÖÐ̨¼Ü¹¹Êµ¼ù
MySQL¸ßÀ©Õ¹¼Ü¹¹Éè¼Æ
Ïà¹Ø¿Î³Ì

Êý¾ÝÖÎÀí¡¢Êý¾Ý¼Ü¹¹¼°Êý¾Ý±ê×¼
MongoDBʵս¿Î³Ì
²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿âÉè¼ÆÓëÓÅ»¯
PostgreSQLÊý¾Ý¿âʵսÅàѵ
×îл¼Æ»®
DeepSeek´óÄ£ÐÍÓ¦Óÿª·¢ 6-12[ÏÃÃÅ]
È˹¤ÖÇÄÜ.»úÆ÷ѧϰTensorFlow 6-22[Ö±²¥]
»ùÓÚ UML ºÍEA½øÐзÖÎöÉè¼Æ 6-30[±±¾©]
ǶÈëʽÈí¼þ¼Ü¹¹-¸ß¼¶Êµ¼ù 7-9[±±¾©]
Óû§ÌåÑé¡¢Ò×ÓÃÐÔ²âÊÔÓëÆÀ¹À 7-25[Î÷°²]
ͼÊý¾Ý¿âÓë֪ʶͼÆ× 8-23[±±¾©]

MySQLË÷Òý±³ºóµÄÊý¾Ý½á¹¹
MySQLÐÔÄܵ÷ÓÅÓë¼Ü¹¹Éè¼Æ
SQL ServerÊý¾Ý¿â±¸·ÝÓë»Ö¸´
ÈÃÊý¾Ý¿â·ÉÆðÀ´ 10´óDB2ÓÅ»¯
oracleµÄÁÙʱ±í¿Õ¼äдÂú´ÅÅÌ
Êý¾Ý¿âµÄ¿çƽ̨Éè¼Æ


²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿â
¸ß¼¶Êý¾Ý¿â¼Ü¹¹Éè¼ÆÊ¦
HadoopÔ­ÀíÓëʵ¼ù
Oracle Êý¾Ý²Ö¿â
Êý¾Ý²Ö¿âºÍÊý¾ÝÍÚ¾ò
OracleÊý¾Ý¿â¿ª·¢Óë¹ÜÀí


GE Çø¿éÁ´¼¼ÊõÓëʵÏÖÅàѵ
º½Ìì¿Æ¹¤Ä³×Ó¹«Ë¾ Nodejs¸ß¼¶Ó¦Óÿª·¢
ÖÐÊ¢Òæ»ª ׿Խ¹ÜÀíÕß±ØÐë¾ß±¸µÄÎåÏîÄÜÁ¦
ijÐÅÏ¢¼¼Êõ¹«Ë¾ PythonÅàѵ
ij²©²ÊITϵͳ³§ÉÌ Ò×ÓÃÐÔ²âÊÔÓëÆÀ¹À
ÖйúÓÊ´¢ÒøÐÐ ²âÊÔ³ÉÊì¶ÈÄ£Ðͼ¯³É(TMMI)
ÖÐÎïÔº ²úÆ·¾­ÀíÓë²úÆ·¹ÜÀí