Test Apache Airflow 3.0.4rc1 And TaskSDK 1.0.4rc1: Status & Help

by Pedro Alvarez 65 views

We're reaching out to the Apache Airflow community to help test the Release Candidate (RC) for Apache Airflow 3.0.4rc1! Your feedback is crucial in ensuring a smooth and stable release. This article details the current status of testing efforts and provides a list of specific pull requests (PRs) that need verification.

Why Test Release Candidates?

Testing RCs allows the community to identify and address any potential issues before the final release. By participating, you contribute directly to the quality and reliability of Apache Airflow, benefiting all users. This collaborative approach ensures that the new version is robust and meets the diverse needs of the Airflow community.

How to Participate in Testing

  1. Install the RC: You can install the Release Candidate using pip: pip install apache-airflow==3.0.4rc1
  2. Test your workflows: Run your existing Airflow workflows against the RC to identify any regressions or compatibility issues. Pay close attention to how your DAGs are performing, and check for any unexpected errors or warnings. Testing your specific use cases is invaluable, as it uncovers issues that might not appear in general testing scenarios.
  3. Review the list of PRs: The following sections detail specific pull requests included in this RC that require focused testing.
  4. Provide feedback: Share your findings by commenting on this issue or the specific pull requests. Be as detailed as possible in your feedback, including steps to reproduce any issues you encounter. Detailed reports help the developers quickly identify and fix problems, which speeds up the release process.

Key Areas to Focus On

This RC includes a variety of fixes and improvements, and we've highlighted specific areas where testing is particularly valuable. Focus your testing efforts on these key areas to maximize your impact on the release quality.

  • Core Functionality: Ensure that core Airflow components such as the scheduler, webserver, and executors are functioning as expected.
  • TaskSDK: With the growing adoption of the TaskSDK, rigorous testing is needed to confirm its stability and performance.
  • Database Interactions: Verify that database connections and operations are working correctly, especially with different database backends.
  • Provider Integrations: Test the functionality of various provider integrations (e.g., AWS, Google Cloud, Azure) to ensure they are compatible with the new release.
  • User Interface: Check the Airflow UI for any visual or functional issues, including DAG views, task logs, and user management features.

Pull Requests Requiring Testing

Below is a list of pull requests included in Apache Airflow 3.0.4rc1 that require testing and verification. Please test the relevant PRs and let us know if the issue is addressed in this RC by commenting on the respective PR or this main thread. Your input is greatly appreciated!

Core Airflow Enhancements and Fixes

Core Airflow enhancements and fixes are crucial for the overall stability and functionality of the platform. These improvements impact everything from task execution to DAG parsing, so thorough testing is essential. Let's dive into some key PRs in this area:

  • Use of debugpy to run airflow components via breeze (#51763): This pull request, contributed by @rawwar, introduces the use of debugpy to facilitate debugging of Airflow components within the Breeze development environment. Debugging is critical for identifying and resolving issues, and this enhancement could significantly improve the developer experience. Testers should verify that debugpy integration works as expected and simplifies the debugging process. This means checking that developers can easily set breakpoints, inspect variables, and step through code within Breeze.

  • [Backport v3-0-test] Add back dag parsing pre-import optimization (#50371) (#52698): This PR, with contributions from @Lzzz666 and @kaxil, reintroduces a critical optimization for DAG parsing. The pre-import optimization significantly speeds up DAG loading times, especially in environments with many DAGs. Testers need to ensure that this optimization effectively reduces DAG parsing time without introducing any side effects or compatibility issues. You should measure the time it takes to load DAGs with and without this optimization, looking for substantial improvements. Also, ensure that all DAGs are parsed correctly and that no tasks are missed or incorrectly configured.

  • [v3-0-test] Dev: Fix KeyError in release issue content generation (#52751) (#52756): @kaxil addressed a KeyError that occurred during the generation of release issue content. This fix is vital for the release process itself, ensuring that all necessary information is correctly compiled for each release. Testers should try to trigger the release issue content generation process to confirm that the KeyError is resolved and that the content is generated without errors. This involves simulating a release process and verifying the output.

  • [v3-0-test] Advanced auto classification for provider documentation (#52902) (#52925): This PR from @amoghrajesh enhances the auto-classification of provider documentation. Well-organized documentation is crucial for user adoption and ease of use. By improving auto-classification, this PR makes it easier for users to find the information they need. Testers should review the generated documentation for various providers to ensure that the classifications are accurate and helpful. Check that the documentation is logically organized and that users can easily navigate to the correct sections.

  • [v3-0-test] Workaround the need to use latest version on Lucas pre-commit's (#52931) (#52942): @potiuk implemented a workaround to address issues related to using the latest version of Lucas pre-commit hooks. Pre-commit hooks are essential for maintaining code quality and consistency, so this fix helps ensure a smooth development workflow. Testers should verify that the workaround functions as expected and does not introduce any regressions in the pre-commit process. This involves running pre-commit hooks and confirming that they execute without errors.

  • [v3-0-test] Make the update_installer_and_pre_commit script more resilient to re matches (#52960) (#52961): This pull request, also by @amoghrajesh, improves the resilience of the update_installer_and_pre_commit script when dealing with regular expression (regex) matches. A more robust script ensures that the installation and pre-commit processes are less prone to errors, even when encountering unexpected patterns. Testers should run the script with various inputs to confirm its resilience and accuracy in handling different regex scenarios. This includes testing with both valid and invalid inputs to ensure the script behaves correctly in all cases.

  • [v3-0-test] Remove upper-binding for "python-requires" (#52980) (#52984): @potiuk removed the upper binding for python-requires, providing greater flexibility in terms of Python versions supported. This change allows users to use newer Python versions without being constrained by the upper bound. Testers should verify that this change does not introduce compatibility issues with different Python versions and that Airflow functions correctly across a range of Python environments. Test installations and executions with various Python versions to ensure compatibility.

  • [v3-0-test] Remove remnants of ~= used in requires-python. (#52985) (#52987): This PR, again from @potiuk, further refines the handling of Python version requirements by removing remnants of the ~= operator. Removing these remnants ensures consistency in how Python version dependencies are managed. Testers should confirm that the change does not adversely affect dependency resolution and that all required Python packages are correctly installed. Verify that all dependencies are installed correctly and that there are no conflicts between package versions.

  • [v3-0-test] Unify connection not found exceptions between AF2 and AF3 (#52968) (#53093): This PR, with contributions from multiple developers including @Lzzz666, @github-actions[bot], @potiuk, and @amoghrajesh, unifies the exceptions raised when a connection is not found between Airflow 2 and Airflow 3. Consistent exception handling makes debugging and error handling more straightforward. Testers should try to trigger