What is RDD in spark

Question

Definition says: RDD is immutable distributed collection of objects I don't quite understand what does it mean. Is it like data (partitioned objects) stored on hard disk If so then how come RDD's can have user-defined classes (Such as java, scala or python) From this link: https://www.safaribooksonline.com/library/view/learning-spark/9781449359034/ch03.html It mentions: Users create RDDs in two ways: by loading an external dataset, or by distributing a collection of objects (e.g., a list or set) in their driver program I am really confused understanding RDD in general and in relation to spark and hadoop. Can some one please help. Select the correct answer from above options

Answer 1

RDD i.e. Resilient Distributed Datasets is an immutable distributed collection of objects. It is fundamental data structure of Spark. There are logical partitions for each dataset in RDD, which can be computed on different nodes of the cluster. RDDs can contain objects of any type Python, Java, or Scala, also it can include user-defined classes. An RDD is a read-only, partitioned collection of records. RDD is a collection of elements which is fault-tolerant and can be operated on in parallel. There are two ways to create RDDs − 1. Parallelizing an existing collection in your driver program, or 2. Referencing a dataset in an external storage system, such as a shared file system, HDFS, HBase, or any data source offering a Hadoop Input Format. Spark makes use of the concept of RDD. RDD makes Spark achieve faster and efficient MapReduce operations.