Assume that we have an RDD of m partitions and another RDD of n partitions. When we perform union operation on first RDD with second RDD we will get a new RDD of partitions m+n.
Let's take an example scenario and do it. We have an RDD of some number of partitions. Here I have a repartitioned the RDD to 1 partition and stored in a variable. And I have repartitioned the same RDD to 4 partitions. Then I have performed union operation on these two RDDs. Then I got a new RDD of 5 partitions.
Look at the screenshot of the operations performed.
If you liked this post or if you feel anything can be enhanced in this post please let us know..
5 thoughts on “What happens to number of partitions when we union two RDDs?”
My brother recommended I may like this blog. He was totally right. This submit actually made my day. You cann’t consider simply how so much time I had spent for this info! Thank you!| а
Like!! I blog frequently and I really thank you for your content. The article has truly peaked my interest.
Like!! I blog quite often and I genuinely thank you for your information. The article has truly peaked my interest.
Thanks for ones marvelous posting! I really enjoyed reading
it, you can be a great author. I will ensure that I bookmark your blog and will eventually come back in the future.
I want to encourage you to ultimately continue your great posts,
have a nice morning!
Thank you so much. Please comment if we have missed any Bigdata concepts on our blog.