Building a Serverless Data Lake architecture in AWS

As the system was overloaded and poorly managed, it was more prone to errors and failures, not always being able to handle the workload efficiently, increasing the risk of data loss or data inconsistencies.

Insufficient disk space for essential security logs or backups could impact proper monitoring and incident response efforts, making the infrastructure more susceptible to attacks. Moreover, without encryption, sensitive data being transmitted or stored within the Apache NiFi process remains vulnerable to unauthorized access or interception, further compromising the security posture of the entire system.

In times when the infrastructure was unable to cope with demand, there would be system failures, sometimes resulting in 3-4 days of service disruptions, where campaigns would not be executed.

Due to NiFi batch processing in large files pose a significant challenge for your data processing workflow. The batch processing approach employed by NiFi, where data is divided into sizable files and processed as a whole, amplifies the problem. When a failure occurs during processing, such as a system crash or component failure, the entire batch being processed becomes compromised, leading to the loss of partially processed data within the batch.