Skip to content
Big Data Interview
The Interview Hacker and Technical guide
  • Home
  • Blogs
  • About Us
  • Contact Us
  • Privacy Policy

Category: Spark

Save action in Spark takes too long time/Save operation spills huge data on to disk and fails with the error “No space left on device”

August 12, 2021 admin Leave a comment

It is very common in Big Data environment to deal with data of size Terabytes. We might observer some times…

Continue Reading →

Posted in: Spark Filed under: SparkOptimizations, SparkPerformanceEnhancementTechniques

How to set configuration to start Reduce jobs after completion of certain proportion of the Map jobs in Hive or Hadoop?

June 8, 2021 admin Leave a comment

Within the MapReduce framework in Platform Symphony, you can specify the proportion of the total number of map tasks in a…

Continue Reading →

Posted in: Big Data, Hive, MapReduce

Lead and Lag using Spark Scala

May 13, 2021 admin Leave a comment

Sometimes while processing data we will come across some situations where we need to find the difference of price a…

Continue Reading →

Posted in: Scala, Spark Filed under: Lag, Lead

How to process JSON data or file in HIVE without using JsonSerDe?

December 20, 2020 admin Leave a comment

It is very rare that the usage of HIVE with JSON. But sometimes business requirements might force the developers to…

Continue Reading →

Posted in: Big Data, Hive Filed under: HIVEJSON

How to add unique index or unique row number to reach row of a DataFrame?

December 7, 2020 admin Leave a comment

There are multiple ways to do this Spark. Here we have discussed two of the approaches to accomplish this task.…

Continue Reading →

Posted in: Big Data, Spark

Advanced performance enhancement techniques in Spark.

December 6, 2020 admin Leave a comment

Design choices: Language choice This impossible to answer and highly depends on your requirement. If you want to perform some…

Continue Reading →

Posted in: Big Data, Spark Filed under: Spark, SparkPerformanceEnhancementTechniques

zip, zipWithIndex and zipWithUniqueId in Spark

admin Leave a comment

These functions are little rarely used in Spark as they confined to be used with RDDs only and RDDs are…

Continue Reading →

Posted in: Big Data, Spark Filed under: zip, zipWithIndex, zipWithUniqueId

Which is the best programming language to use with Spark?

December 5, 2020 admin Leave a comment

Spark supports multiple programming languages. Out of them most used languages are Scala, Python and Java. But which is best…

Continue Reading →

Posted in: Big Data, Spark Filed under: JAva, Python, Scala, Spark

When do Spark spills the cached RDD or DataFrame onto disk? (or) What is the threshold to spill the cached data onto disk in Spark?

admin Leave a comment

Caching is one of the best optimization techniques available on Spark. When we cache any RDD or DataFrame, Spark will…

Continue Reading →

Posted in: Big Data, Spark

Miscellaneous Spark interview questions – Part I

admin Leave a comment

1). Spark jargon – Job A Spark application will have a number of sub-processes and each of them can be…

Continue Reading →

Posted in: Spark

Post navigation

Page 1 of 5
1 2 … 5 Next →

Recent Posts

  • Save action in Spark takes too long time/Save operation spills huge data on to disk and fails with the error “No space left on device”
  • How to set configuration to start Reduce jobs after completion of certain proportion of the Map jobs in Hive or Hadoop?
  • HDFS commands

Recent Comments

  • curry 7 sour patch on Spark groupByKey vs reduceByKey vs aggregateByKey
  • jordan 4 on Hive – Order By vs Sort By vs Cluster By vs Distribute By
  • louboutin shoes on Spark RDD vs Dataframe vs Dataset

Archives

  • August 2021
  • June 2021
  • May 2021
  • January 2021
  • December 2020
  • October 2020
  • July 2020
  • May 2020
  • April 2020
  • March 2020
  • November 2019
  • July 2019
  • June 2019
  • May 2019

Follow Us

Contact Us

  • Email
    sparkandbigdatainterview@gmail.com
Privacy Policy
Copyright © 2023 Big Data Interview