I went at 8am and it wasn't too busy; I managed to get quite a few pictures without any people in them. The queues pile up at the entrance as everyone wants to get their photo at the start, but you can skip that section and keep walking to the top of the hill... there are literally 1000 gates that go on for ages. Once you've walked back down the hill, get an amazake at Amazake Suetomi-tei for ¥500 and breakfast at the Vermillion Café.
15 hours ago