Amazon EMR

The most significant advantage of Amazon EMR is its exceptional cost management capabilities. EMR's unique architecture with core nodes (24/7) and transient task nodes that automatically turn on and off is unparalleled by other providers, who typically only offer all-on-demand or all-spot auto-scaling groups. Another highly valued feature is the flexibility in choosing instance families. For example, within the R5 family, we can select from R5, R5A, R5G (graviton, which saves cost), or R5D (memory-optimized). The ability to use multiple node types within a family and assign weightage (e.g., prioritizing cost-saving Graviton instances) provides unmatched flexibility tailored to our specific use case. This also extends to time-dependent scaling, allowing us to allocate more on-demand instances during peak US hours and shift to more spot instances during India's non-peak hours, resulting in significant daily cost savings. Furthermore, EMR's seamless integration with other AWS services, such as Amazon S3, is a major benefit. Our underlying data layer is S3, and EMR provides an intuitive, built-in integration that largely eliminates the need for custom code, relying instead on simple dropdown configurations. It also integrates easily with open-source coding platforms we utilize. Finally, EMR prioritizes customer experience and ensures operational resilience. If spot machines are unavailable during peak times, EMR automatically waits for a configured period (e.g., 5 minutes) and then provisions on-demand instances to prevent customer impact. While this may temporarily increase cost, nodes automatically shut down when the load subsides. EMR also boasts built-in fault tolerance, automatically spinning up a new node if one goes down, ensuring continuous availability and preventing downtime.

hands on, configurability

Great high compute platform

My primary concern with Amazon EMR is a significant limitation in its configuration flexibility: the inability to change the instance family of a cluster once it has been created from the UI. If an incorrect or higher-cost instance family is selected during initial cluster setup, there is no direct way to modify it. To rectify such an error, one must terminate the existing cluster, create a clone, make the necessary changes, and then redeploy. This process inevitably leads to downtime or necessitates incurring additional cost by running a parallel cluster until the new one is operational. This is a major pain point, as I've observed similar feedback online from other customers. While EMR allows for editing parameters like the number of spot or on-demand nodes, the fundamental instance family cannot be altered post-implementation. Most other AWS features offer an edit option, and its absence for EMR cluster instance families is a notable drawback that, if resolved, would significantly enhance the product's usability and flexibility for many users.

finding failure is hard

not much i can think off.