Caching is one of the best optimization techniques available on Spark. When we cache any RDD or DataFrame, Spark will…
1). Spark jargon – Job A Spark application will have a number of sub-processes and each of them can be…
What is Minus operation? Below is a picture that shows Venn diagram of result of minus operation between two tables…
In this I have given all miscellaneous interview questions related to UNIX. For Unix interview questions we should try to…
In Big Data applications we rarely get the requirement to pivot the data because making transpose of billions of rows…
EXISTS EXISTS operator will be used when we need to check if there is any row exists with a condition.…
To delete duplicate records in a DataFrame we can either use distinct or dropDuplicates method. But dropDuplicates method comes with…
While working with Big Data applications we might have used the methods withColumn and select(). They both are few of…
Spark has lot of performance enhancement techniques. Two of them are cache() and broadcast variables. Although they both used to…
In this post we will see how we can extract unique records from a Hive table. This can be achieved…