Suppose a player backtracks to an area within the same day. In that case, the lack of raids can make an area feel much emptier than it was a few moments ago, and this would not be remedied until a full day passes. This is unlike normal Pokemon spawns, which would refill an empty area after only a minute or two.
Raids despawning also makes it much harder to track down Legendary Pokemon raids. When a player first sees the spawn message for a Legendary raid, they aren't given any solid leads on the exact location other than the biome. From my experience, raids load in at a 100-block radius around the player, so the player would have to travel 100 blocks away in a specific direction to find the raid. If they happen to go in the wrong direction, the raid would despawn, leaving the player to search for a raid that no longer exists.