Emperor Naruhito is concerned that holding the Games will fuel coronavirus infections in Japan, the head of the Imperial Household Agency said.