Images captured in adverse weather conditions significantly impact the performance of many vision tasks. Common weather phenomenon, including but not limited to rain, snow, fog, introduce visual degradations to captured images and videos. These degradations may include partial to severe occlusions of objects, illumination changes, noise, etc. As most vision algorithms assume clear weather, with no interference of particles present under adverse weather, their performance suffers due to the domain gap.

Approaches to deal with the domain gap ranges from domain adaptation to data augmentation. Yet, domain adaptation assumes access to the target (adverse weather) data, which compared to data collected in clear weather conditions are on orders of magnitude less. Data augmentation, on the other hand, relies on generative models, e.g., image-to-image translation, and synthetic particle rendering. However, image-to-image translation may introduce artifacts into the resulting images used for augmentation. Synthetic data augmentation involves adding adverse weather particles, e.g. rain streaks and snowflakes, to the input images, which may be captured during clear or even sunny conditions; aside from the illumination mismatch, the particle models may also fail to capture diverse variations of the weather phenomenon: from size of the particles to their appearance under motion and accumulation, which is exceedingly difficult to simulate. Hence, both augmentation schemes may induce a sim2real domain gap where the input distribution of synthetic or generated images do not align with distribution of real images captured under adverse weather. Amongst the many challenges, there also lacks a large-scale dataset with annotations to train and evaluate vision algorithms under adverse conditions.

Building on the recent automated data engine, WeatherStream [1], and the all-weather (rain, fog, snow, etc.) imagery across the globe that was collected by it, we seek to address the need within the community for a large-scale adverse weather dataset with semantic annotations. In a collaboration with the authors of WeatherStream, we have annotated images within the dataset with semantic segmentation masks and will propose the WeatherStream semantic segmentation benchmark. The dataset is comprised of approximately 202K images with diverse range of scenes, camera parameters, and unique locations spanning over 94 countries across the world (see Figure 7). This benchmark will be the cornerstone of the UG2+ Challenge Track 3 for all weather semantic segmentation.

This challenge aims to promote the development of novel semantic segmentation algorithms for images captured under adverse weather conditions. The competition will feature diverse and challenging scenarios that include (i) various types of weather conditions, e.g. light and heavy degrees of rain, snow, and fog, (ii) large variety of background scenes from urban locations (i.e. buildings, streets, cityscapes) to natural scenery (i.e. forests, plains, hills), (iii) varying degrees of illumination from different times of day, and all of which are captured by (iv) cameras that cover a wide array of resolutions, noise levels, and intrinsic parameters. The goal of the challenge is to spark innovative and robust approaches for semantic segmentation under adverse conditions. To this end, the challenge will be split into three phases (training, validation, and testing), where data corresponding to each phase will be released to the participants. The submissions will be evaluated quantitatively using standard metrics like mean intersection-over-union (mIOU) and pixel accuracy (pAcc). Participants will submit their results on the validation and testing sets for evaluation, with a limited number of attempts per phase to ensure fairness amongst teams registering for the competition at different times. This limit is set at 50 total submissions for the validation phase and 25 total submissions for the testing phase at 5 submissions per day. The best results for each phase will be reported on the benchmark. The ranking of algorithms will be determined based on evaluation scores on the testing set. Participants are required to submit a fact sheet including description of their method, datasets used, hyperparameters, memory, runtime, and hardware requirements, etc. The fact sheets will be compiled into a final report post challenge to highlight trends and innovations. Participants are encouraged to submit manuscripts detailing their method to the workshop.

References:
[1] Zhang, H., Ba, Y., Yang, E., Mehra, V., Gella, B., Suzuki, A., Pfahnl, A., Chandrappa, C., Wong, A., & Kadambi, A. (2023). WeatherStream: Light Transport Automation of Single Image Deweathering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.