[SPARK-22938][SQL][FOLLOWUP] Assert that SQLConf.get is accessed only on the driver
authorWenchen Fan <wenchen@databricks.com>
Fri, 11 May 2018 01:01:40 +0000 (09:01 +0800)
committerhyukjinkwon <gurwls223@apache.org>
Fri, 11 May 2018 01:01:40 +0000 (09:01 +0800)
commita4206d58e05ab9ed6f01fee57e18dee65cbc4efc
treecb8c8edda68eee591749badb9984b0fc68571ba4
parentd3c426a5b02abdec49ff45df12a8f11f9e473a88
[SPARK-22938][SQL][FOLLOWUP] Assert that SQLConf.get is accessed only on the driver

## What changes were proposed in this pull request?

This is a followup of https://github.com/apache/spark/pull/20136 . #20136 didn't really work because in the test, we are using local backend, which shares the driver side `SparkEnv`, so `SparkEnv.get.executorId == SparkContext.DRIVER_IDENTIFIER` doesn't work.

This PR changes the check to `TaskContext.get != null`, and move the check to `SQLConf.get`, and fix all the places that violate this check:
* `InMemoryTableScanExec#createAndDecompressColumn` is executed inside `rdd.map`, we can't access `conf.offHeapColumnVectorEnabled` there. https://github.com/apache/spark/pull/21223 merged
* `DataType#sameType` may be executed in executor side, for things like json schema inference, so we can't call `conf.caseSensitiveAnalysis` there. This contributes to most of the code changes, as we need to add `caseSensitive` parameter to a lot of methods.
* `ParquetFilters` is used in the file scan function, which is executed in executor side, so we can't can't call `conf.parquetFilterPushDownDate` there. https://github.com/apache/spark/pull/21224 merged
* `WindowExec#createBoundOrdering` is called on executor side, so we can't use `conf.sessionLocalTimezone` there. https://github.com/apache/spark/pull/21225 merged
* `JsonToStructs` can be serialized to executors and evaluate, we should not call `SQLConf.get.getConf(SQLConf.FROM_JSON_FORCE_NULLABLE_SCHEMA)` in the body. https://github.com/apache/spark/pull/21226 merged

## How was this patch tested?

existing test

Author: Wenchen Fan <wenchen@databricks.com>

Closes #21190 from cloud-fan/minor.
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTables.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala
sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala