80 multiple-choice questions provides an in-depth exploration of core Big Data technologies in the Hadoop ecosystem. Covering MapReduce for parallel data processing, HDFS for scalable distributed storage, and YARN for efficient resource management, these MCQs are designed to test and reinforce foundational knowledge for aspiring data engineers and analysts.
1. What does HDFS stand for in the context of Big Data?
✅ Correct Answer: a) Hadoop Distributed File System
📝 Explanation:
HDFS is the primary storage system used by Hadoop for storing large datasets across multiple machines in a distributed manner.
2. In HDFS, what is the default block size for files?
✅ Correct Answer: b) 128 MB
📝 Explanation:
The default block size in HDFS is 128 MB, which allows for efficient storage and processing of large files by dividing them into manageable chunks.
3. Which component in HDFS is responsible for managing the namespace and regulating access to files?
✅ Correct Answer: b) NameNode
📝 Explanation:
The NameNode maintains the file system namespace and metadata, controlling access to files and directories in HDFS.
4. In HDFS, DataNodes store and retrieve data in response to instructions from which component?
✅ Correct Answer: a) NameNode
📝 Explanation:
DataNodes perform read/write operations on blocks as directed by the NameNode and report their status periodically.
5. What is the role of the Secondary NameNode in HDFS?
✅ Correct Answer: b) It periodically checkpoints the fsimage and edits log files
📝 Explanation:
The Secondary NameNode merges the fsimage and edit logs to create a new checkpoint, reducing recovery time for the NameNode.
6. HDFS is designed to run on which type of hardware?
✅ Correct Answer: a) Low-cost commodity hardware
📝 Explanation:
HDFS is built to tolerate frequent hardware failures and operate on inexpensive, commodity hardware for scalability.
7. By default, how many replicas of each block are maintained in HDFS?
✅ Correct Answer: c) 3
📝 Explanation:
HDFS maintains three replicas of each block by default to ensure high availability and fault tolerance.
8. What is the purpose of the Rack Awareness feature in HDFS?
✅ Correct Answer: a) To optimize data locality and fault tolerance
📝 Explanation:
Rack Awareness places replicas across different racks to minimize network traffic and improve resilience against rack failures.
9. Which HDFS command is used to create a directory?
✅ Correct Answer: a) hdfs dfs -mkdir
📝 Explanation:
The 'hdfs dfs -mkdir' command creates directories in the HDFS namespace, similar to the Unix mkdir command.
10. In HDFS, what happens if the NameNode fails without a proper checkpoint?
✅ Correct Answer: c) Manual recovery from edit logs is required, which can be time-consuming
📝 Explanation:
Without checkpoints, the NameNode recovery involves replaying the entire edit log, which can delay cluster availability.
11. HDFS supports which type of streaming access for data?
✅ Correct Answer: a) High-throughput, low-latency access
📝 Explanation:
HDFS is optimized for batch processing with high-throughput streaming access to large files, not low-latency random access.
12. What is the maximum file size supported by HDFS?
✅ Correct Answer: b) Unlimited (limited by available storage)
📝 Explanation:
HDFS can theoretically handle petabyte-scale files, constrained only by the total cluster storage capacity.
13. Which permission model does HDFS use for access control?
✅ Correct Answer: a) POSIX-like permissions
📝 Explanation:
HDFS implements a permission model similar to POSIX, with owner, group, and others categories for read/write/execute.
14. In HDFS, blocks are stored as files on the local file system of which node?
✅ Correct Answer: c) DataNode
📝 Explanation:
DataNodes store HDFS blocks as regular files on their local file systems and handle I/O operations for them.
15. What is the fsimage file in HDFS?
✅ Correct Answer: a) A persistent checkpoint of the file system metadata
📝 Explanation:
The fsimage is a serialized representation of the NameNode's in-memory namespace and block data.
16. HDFS Federation allows multiple NameNodes to manage which aspect?
✅ Correct Answer: a) Separate namespaces
📝 Explanation:
HDFS Federation enables horizontal scaling by allowing multiple independent NameNodes, each managing its own namespace.
17. What command lists the contents of a directory in HDFS?
✅ Correct Answer: a) hdfs dfs -ls
📝 Explanation:
'hdfs dfs -ls' displays the list of files and directories in the specified HDFS path.
18. In HDFS, what is the edit log used for?
✅ Correct Answer: a) Recording namespace modifications between checkpoints
📝 Explanation:
The edit log captures every change to the file system metadata since the last fsimage snapshot.
19. HDFS is best suited for which workload?
✅ Correct Answer: a) Write-once, read-many-times
📝 Explanation:
HDFS appends data to files and supports multiple reads, making it ideal for batch processing workloads.
20. What is the purpose of Balancer in HDFS?
✅ Correct Answer: a) To distribute data evenly across DataNodes
📝 Explanation:
The HDFS Balancer tool rebalances data blocks to ensure even distribution and prevent hotspots.
21. Which HDFS feature provides fault tolerance through data replication?
✅ Correct Answer: a) Block replication
📝 Explanation:
Replication ensures multiple copies of data blocks, allowing the system to recover from node failures seamlessly.
22. What is the default port for the NameNode web UI in HDFS?
✅ Correct Answer: a) 50070
📝 Explanation:
The NameNode's web interface runs on port 50070 for monitoring cluster status.
23. In HDFS, can files smaller than the block size be stored?
✅ Correct Answer: a) Yes, they occupy the full block space
📝 Explanation:
Small files in HDFS still reserve a full block, which can lead to inefficiency in name space usage.
24. What is Erasure Coding in HDFS used for?
✅ Correct Answer: a) To reduce storage overhead compared to replication
📝 Explanation:
Erasure Coding encodes data into parity blocks, providing fault tolerance with less storage than triple replication.
25. Which command copies a file from local to HDFS?
✅ Correct Answer: a) hdfs dfs -put
📝 Explanation:
'hdfs dfs -put' uploads files from the local file system to HDFS.
26. HDFS supports symmetric or asymmetric replication policies?
✅ Correct Answer: c) Both
📝 Explanation:
HDFS allows configuration of both symmetric (same replicas everywhere) and asymmetric (different counts) replication.
27. What is the heartbeat interval for DataNodes to NameNode?
✅ Correct Answer: a) 3 seconds
📝 Explanation:
DataNodes send heartbeats every 3 seconds to indicate liveness and report block reports every 6 seconds.
28. In HDFS, what is a 'block scan'?
✅ Correct Answer: a) Periodic verification of block integrity
📝 Explanation:
Block scanning checks the checksums of stored blocks to detect corruption.
29. Which HDFS configuration parameter sets the replication factor?
✅ Correct Answer: a) dfs.replication
📝 Explanation:
The 'dfs.replication' property in hdfs-site.xml defines the default number of replicas for data blocks.
30. HDFS High Availability (HA) uses which components for failover?
✅ Correct Answer: a) Active and Standby NameNodes with JournalNodes
📝 Explanation:
HA setup includes shared storage via JournalNodes for seamless failover between NameNodes.
31. What is the purpose of the 'hdfs fsck' command?
✅ Correct Answer: a) To check file system health and find corrupt blocks
📝 Explanation:
'hdfs fsck' performs a file system check, reporting under-replicated, missing, or corrupt blocks.
32. In HDFS, data transfer between client and DataNode uses which protocol?
✅ Correct Answer: a) TCP/IP
📝 Explanation:
HDFS uses TCP sockets for reliable data streaming between clients and DataNodes.
33. What does MapReduce stand for in Big Data processing?
✅ Correct Answer: a) A programming model for parallel processing
📝 Explanation:
MapReduce is a framework that allows distributed processing of large data sets on clusters using Map and Reduce functions.
34. In MapReduce, what is the role of the Map function?
✅ Correct Answer: a) To process input data and produce key-value pairs
📝 Explanation:
The Map function takes input data, processes it in parallel, and emits intermediate key-value pairs.
35. What is the output of the Reduce function in MapReduce?
✅ Correct Answer: a) Final aggregated results
📝 Explanation:
The Reduce function receives grouped key-value pairs and produces the final output for the job.
36. In MapReduce, what is an InputSplit?
✅ Correct Answer: a) A logical division of input data for parallel processing
📝 Explanation:
InputSplits define how input data is divided among Map tasks for distributed execution.
37. Which component in MapReduce v1 is responsible for job scheduling and task management?
✅ Correct Answer: a) JobTracker
📝 Explanation:
The JobTracker oversees the entire MapReduce job lifecycle, assigning tasks to TaskTrackers.
38. What is the purpose of the Combiner in MapReduce?
✅ Correct Answer: a) To perform local aggregation before shuffle and sort
📝 Explanation:
Combiners reduce the amount of data transferred during the shuffle phase by aggregating locally on mapper nodes.
39. In MapReduce, the shuffle and sort phase occurs between which stages?
✅ Correct Answer: a) Map and Reduce
📝 Explanation:
Shuffle and sort groups and sorts the intermediate outputs from Mappers before sending them to Reducers.
40. What is the default InputFormat in MapReduce?
✅ Correct Answer: a) TextInputFormat
📝 Explanation:
TextInputFormat treats each line of input as a key-value pair, with offset as key and line as value.
41. MapReduce jobs are fault-tolerant due to which mechanism?
✅ Correct Answer: a) Task retry and speculative execution
📝 Explanation:
Failed tasks are retried, and speculative execution runs duplicates of slow tasks to ensure timely completion.
42. What is a Partitioner's role in MapReduce?
✅ Correct Answer: a) To decide which Reducer receives which key
📝 Explanation:
The Partitioner determines the mapping of intermediate keys to Reducers based on a hash function.
43. In MapReduce, what does speculative execution address?
✅ Correct Answer: a) Straggler tasks that slow down the job
📝 Explanation:
Speculative execution launches duplicate tasks for slow-running ones, using the first to complete.
44. Which class is used to define a custom Mapper in MapReduce?
✅ Correct Answer: a) Mapper
📝 Explanation:
Developers extend the Mapper class and override the map() method to implement custom logic.
45. What is the purpose of OutputFormat in MapReduce?
✅ Correct Answer: a) To control the output of Reduce tasks
📝 Explanation:
OutputFormat defines how and where the final output from Reducers is written, e.g., to HDFS.
46. MapReduce processes data in which manner?
✅ Correct Answer: a) In parallel across a cluster
📝 Explanation:
MapReduce enables parallel processing by distributing tasks across multiple nodes in a cluster.
47. What is the Counter in MapReduce used for?
✅ Correct Answer: a) To track job progress and custom metrics
📝 Explanation:
Counters collect statistics during job execution, such as bytes processed or custom application metrics.
48. In MapReduce v1, TaskTrackers run on which nodes?
✅ Correct Answer: a) Slave nodes
📝 Explanation:
TaskTrackers execute Map and Reduce tasks on worker (slave) nodes under JobTracker supervision.
49. What is the join operation in MapReduce typically implemented using?
✅ Correct Answer: a) Custom Mappers and Reducers
📝 Explanation:
Joins are achieved by emitting join keys in Map and aggregating matching records in Reduce.
50. MapReduce supports user-defined functions in which languages?
✅ Correct Answer: a) Java, Python, C++ via Hadoop Streaming
📝 Explanation:
Hadoop Streaming allows MapReduce jobs in non-Java languages using standard input/output.
51. What is the purpose of the DistributedCache in MapReduce?
✅ Correct Answer: a) To cache small files on all nodes for efficient access
📝 Explanation:
DistributedCache distributes read-only files like lookup tables to all nodes before job start.
52. In MapReduce, data locality refers to?
✅ Correct Answer: a) Processing data on the node where it is stored
📝 Explanation:
Data locality minimizes network I/O by scheduling tasks on nodes holding the data.
53. What is the default number of Reduce tasks in a MapReduce job?
✅ Correct Answer: a) 1
📝 Explanation:
By default, MapReduce sets one Reducer unless specified otherwise via job configuration.
54. Which MapReduce phase can run without the Reduce phase?
✅ Correct Answer: a) Map-only job
📝 Explanation:
Jobs can be configured with zero reducers to perform only mapping and write intermediate output.
55. What is YARN in the Hadoop ecosystem?
✅ Correct Answer: a) Yet Another Resource Negotiator
📝 Explanation:
YARN is Hadoop's resource management framework that decouples resource allocation from job execution.
56. In YARN, what is the role of the ResourceManager?
✅ Correct Answer: a) Global resource allocation and job scheduling
📝 Explanation:
The ResourceManager arbitrates resources across the cluster and schedules applications.
57. What are NodeManagers in YARN?
✅ Correct Answer: a) Per-node agents that manage containers
📝 Explanation:
NodeManagers monitor resources on their host and launch containers as directed by the ResourceManager.
58. In YARN, what is an ApplicationMaster?
✅ Correct Answer: a) Per-application manager for negotiating resources and coordinating tasks
📝 Explanation:
Each application gets its own ApplicationMaster to handle resource requests and task execution.
59. What is a Container in YARN?
✅ Correct Answer: a) An abstract unit of allocation including CPU, memory, etc.
📝 Explanation:
Containers represent allocated resources (CPU, memory, disk) for running application components.
60. YARN improves upon MapReduce v1 by?
✅ Correct Answer: a) Separating resource management from job-specific logic
📝 Explanation:
YARN generalizes the architecture to support multiple processing engines beyond MapReduce.
61. Which scheduler in YARN provides capacity guarantees?
✅ Correct Answer: a) Capacity Scheduler
📝 Explanation:
The Capacity Scheduler allows queues with guaranteed shares and supports hierarchical queues.
62. In YARN, what is the ResourceRequest used for?
✅ Correct Answer: a) ApplicationMaster requesting specific resources from ResourceManager
📝 Explanation:
ResourceRequest specifies locality preferences, priority, and resource requirements for allocation.
63. What is the default scheduler in YARN?
✅ Correct Answer: a) Capacity Scheduler
📝 Explanation:
YARN defaults to the Capacity Scheduler for multi-tenancy support in production environments.
64. YARN supports which programming models besides MapReduce?
✅ Correct Answer: a) Spark, Tez, and others
📝 Explanation:
YARN's generic interface allows diverse frameworks like Spark and Tez to run on Hadoop clusters.
65. In YARN, NodeManager heartbeats to whom?
✅ Correct Answer: a) ResourceManager
📝 Explanation:
NodeManagers send periodic heartbeats to the ResourceManager to report available resources.
66. What is the purpose of the Timeline Server in YARN?
✅ Correct Answer: a) To store and retrieve application history information
📝 Explanation:
The Timeline Server collects generic application history for monitoring and debugging.
67. YARN Federation allows?
✅ Correct Answer: a) Multiple ResourceManagers for scaling
📝 Explanation:
YARN Federation enables sub-clusters with separate ResourceManagers for large-scale deployments.
68. In YARN, what happens if an ApplicationMaster fails?
✅ Correct Answer: a) The ResourceManager restarts it
📝 Explanation:
YARN supports ApplicationMaster recovery by restarting it with the same state.
69. Which YARN component handles security via tokens?
✅ Correct Answer: b) All components
📝 Explanation:
YARN uses container launch tokens and other security tokens for authentication across components.
70. What is Opportunistic Containers in YARN?
✅ Correct Answer: a) Low-priority containers that use idle resources
📝 Explanation:
Opportunistic mode allows flexible resource usage for bursty workloads without guarantees.
71. The Fair Scheduler in YARN supports?
✅ Correct Answer: a) Hierarchical queues with fair share allocation
📝 Explanation:
It dynamically allocates resources fairly among queues and jobs within them.
72. In YARN, the 'yarn' command is used for?
✅ Correct Answer: a) Submitting applications and managing cluster
📝 Explanation:
The 'yarn' CLI submits jobs, lists applications, and kills running ones.
73. What is the default memory allocation for a container in YARN?
✅ Correct Answer: a) Configurable, default 1 GB
📝 Explanation:
yarn.scheduler.minimum-allocation-mb defaults to 1024 MB for container memory.
74. YARN's ResourceModel includes which resources?
✅ Correct Answer: a) CPU vcores and memory MB
📝 Explanation:
YARN allocates resources in terms of virtual CPU cores and memory in megabytes.
75. What is the role of the Scheduler in YARN?
✅ Correct Answer: a) To allocate resources to applications based on policies
📝 Explanation:
The Scheduler component of ResourceManager decides resource grants to ApplicationMasters.
76. In YARN, Application Timeline Service v2 provides?
✅ Correct Answer: a) Flow-level aggregation of application events
📝 Explanation:
ATS v2 enables entity-level history storage for better querying and visualization.
77. YARN supports rolling upgrades for?
✅ Correct Answer: a) Zero-downtime version updates
📝 Explanation:
Rolling upgrades allow updating nodes incrementally without stopping the cluster.
78. What is the default port for YARN ResourceManager web UI?
✅ Correct Answer: a) 8088
📝 Explanation:
The ResourceManager's web interface is accessible on port 8088 for cluster monitoring.
79. In YARN, locality preferences for ResourceRequest include?
✅ Correct Answer: a) Node, rack, and any
📝 Explanation:
Applications can request resources with preferences for specific nodes, racks, or anywhere.
80. What is the purpose of Labels in YARN?
✅ Correct Answer: a) To tag nodes for access control and scheduling
📝 Explanation:
Node labels allow partitioning the cluster logically for different workloads or users.
