LaVOZs

The World’s Largest Online Community for Developers

'; scala - Spark 2.0 - Split Dataframe into subDataframes without using createOrReplaceTempView - LavOzs.Com

I have a DF as show below , I need to convert this into sub-Dataframes without using createOrReplaceTempView (and doing sub-queries in sql) as the sub-Dataframes will be used for performing multiple left joins with Master Table

I tried listvaluesDF.createOrReplaceTempView("listvaluesDF") && var dfYear = spark.sql("select key, value from listvaluesDF where internal = 'year'"), but when I do explain its creating a View which can conflict when doing big left join

+--------------------+--------------------+----------------+
|                 key|               value|        internal|
+--------------------+--------------------+----------------+
|         accessories|   D31 - Accessories|  wmt0Department|
|         annualEvent|        Annual Event|       wmt0Event|
|             apparel|             Apparel|         wmt0SBU|
|          automotive|    D10 - Automotive|  wmt0Department|
|              baking|              baking|wmt0DeptCategory|
|              baking|              Baking|      seasonType|
|              baking|              Baking|wmt0DeptCategory|
|       bathandShower|D20 - Bath and Sh...|  wmt0Department|
|             bedding|       D22 - Bedding|  wmt0Department|
|               betty|               Betty|wmt0DeptCategory|
|          boxedCards|         Boxed Cards|wmt0DeptCategory|
|            boyswear|            Boyswear|wmt0DeptCategory|
|            boyswear|      D24 - Boyswear|  wmt0Department|
|        buildaBasket|      Build a Basket|wmt0DeptCategory|
|  camerasAndSupplies|D06 - Cameras & S...|  wmt0Department|
|       cardsAndGifts|       Cards & Gifts|      seasonType|
|       cardsAndGifts|       Cards & Gifts|wmt0DeptCategory|
|             carving|             Carving|wmt0DeptCategory|
|         celebration|   D67 - Celebration|  wmt0Department|
|      chineseNewYear|    Chinese New Year|      seasonType|
|           christmas|           Christmas|      seasonType|
|         cookandDine| D14 - Cook and Dine|  wmt0Department|
|            costumes|            Costumes|wmt0DeptCategory|
|              crafts|        D44 - Crafts|  wmt0Department|
|               decor|               Décor|wmt0DeptCategory|
|    decortiveGarland|Decorative Wreath...|wmt0DeptCategory|
|          dotcomOnly|         Dotcom Only|  wmt0DotcomOnly|
|             dressUp|            Dress Up|wmt0DeptCategory|
|              easter|              Easter|      seasonType|
|         electronics|   D72 - Electronics|  wmt0Department|
|       entertainment|       Entertainment|         wmt0SBU|
|                fall|           S3 - Fall|      seasonType|
|           fallGMPad|         Fall GM Pad|       wmt0Event|
|         fallModular|        Fall Modular|      seasonType|
|         familySocks|  D27 - Family Socks|  wmt0Department|
|         familySocks|        Family Socks|wmt0DeptCategory|
|          fathersDay|        Father's Day|      seasonType|
|             feature|             Feature|       wmt0Event|
|           featureQ1|          Feature Q1|      seasonType|
|           featureQ2|          Feature Q2|      seasonType|
|           featureQ3|          Feature Q3|      seasonType|
|           featureQ4|          Feature Q4|      seasonType|
|      featureSeason1|    Feature Season 1|      seasonType|
|      featureSeason2|    Feature Season 2|      seasonType|
|      featureSeason3|    Feature Season 3|      seasonType|
|      featureSeason4|    Feature Season 4|      seasonType|
|           firstHalf|     H1 – First Half|      seasonType|
Related
How to change column types in Spark SQL's DataFrame?
How to convert rdd object to dataframe in spark
Spark - load CSV file as DataFrame?
Concatenate columns in Apache Spark DataFrame
Difference between DataFrame, Dataset, and RDD in Spark
Extract column values of Dataframe as List in Apache Spark
How to check if spark dataframe is empty?
Renaming column names of a DataFrame in Spark Scala
Spark 2.0 Dataset vs DataFrame
How does createOrReplaceTempView work in Spark?