Google Cloud Professional Data Engineer Exam

Which best practice should be followed when creating repeatable data processing tasks in Cloud Composer?

1. Write each task to be responsible for one operation.

2. Use current time with the now() function for computation.

3. Update data with INSERT statements during the task run.

4. Combine multiple functionalities in a single task execution.

The correct answer is: Write each task to be responsible for one operation.

The best practice for creating repeatable data processing tasks in Cloud Composer is to write each task to be responsible for one operation. This approach aligns with the principles of modular design and the Single Responsibility Principle, which asserts that a function or module should do one thing and do it well. By designing tasks this way, you promote clarity and maintainability within your workflows. Each task can be tested, debugged, and reused independently, which simplifies the process of managing complex workflows. If a task fails or requires modification, it can be handled without impacting other tasks, leading to improved reliability and easier debugging. In contrast, the other practices recommended do not align with best practices for task design in workflows. Using the current time with the now() function is inappropriate because it may introduce variability in task execution timing, challenging repeatability. Updating data with INSERT statements during a task run can lead to issues with data consistency and integrity. Lastly, combining multiple functionalities in a single task execution adds complexity and makes it harder to identify problems within the workflow. Thus, maintaining a clear separation of concerns by assigning one operation per task is the most effective strategy.