Wasps are generally more active during the hottest part of the day. They get less active at night and dusk.
So, you are more likely to be attacked by a swarm of wasps when it’s daytime and the workers are out and about and more likely to see you as a potential threat.
Please don’t attempt to remove a nest yourself even if you think it’s dead. Send me a pic if you want to be sure it’s dead and I am happy to give you some free and helpful advice.