The pandemic of Coronavirus Disease 2019 (COVID-19) has had a significant global impact, and CT scans play a crucial role in diagnosing COVID-19, where the research of deep learning-based lung infection segmentation attracts more and more attention. However, existing segmentation methods often struggle with challenges unique to COVID-19 CT images, such as scattered regions, varying numbers of infected areas, diverse sizes, intricate backgrounds, and indistinct boundaries. To address these challenges, we propose a Cascaded Aggregation Network (CANet) which incorporates a multi-level feature encoder, a cascaded decoding module and a pyramid aggregation module. Firstly, the multi-level feature encoder captures and enhances high-level semantic cues across multiple levels, allowing CANet to effectively deal with the spatial variability of infection areas. Then, the cascaded decoding module combines multi-level deep features using a hierarchical structure. It progressively fuses information from different levels through two partial decoder (PD) blocks and a dense feature aggregation (DFA) block. In this way, we can ensure detailed feature refinement and global semantic consistency. After that, the pyramid aggregation module further refines the fused features. By using three attention-aware feature integration (AFI) units, it dynamically adjusts feature importance across different spatial scales, improving the segmentation accuracy of infection regions with varying sizes and complex backgrounds. By following this approach, we can achieve the final premium segmentation results. Extensive experiments show that CANet outperforms 13 state-of-the-art methods. Specifically, CANet achieves improvements over the top-level model BS-Net in sensitivity (2.2%), F1-score (0.5%), and Hausdorff distance (10.5%), demonstrating its superior performance in segmenting COVID-19 lung infections.