MWAA vs Airflow

Consider the following scenario while choosing between the Amazon Managed Workflows for Apache Airflow (MWAA) vs Airflow

Accessing the Airflow’s REST API’s

You can’t access the REST API’s of Airflow directly. But AWS provide ways to access through CLI with support for only a limited functionalities. This may be inconvenient in some use cases.

Refer this URL for the supported commands: https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-cli-command-reference.html

Setting permission for bash files

You can’t set execute permission for your bash files. For e.g. if you are going to call a .sh file through bash operator, the script will not execute due to permission issue. Because the worker containers are both ephemeral and limited to user level access without executable permissions.

There is a workaround by putting the file in plugins.zip with execute permission and that permission will be preserved while deploying to MWAA

Linux packages can’t be installed

For some reason, if you want to install some package through apt-get, then its not possible. You can install only through the python packages or through the .whl file

Local storage

You can use either /tmp/ or /usr/local/airflow/ to save any files locally. You can’t create any directories in other than “/tmp” folder. Also if you want to save huge file for some processing, you have to put it in S3. This may be inconvenient in some cases. Also you can’t attach any network drive to it