The troll is little danger, but you will not be able to cross the bridge to the Gnome's Island unless you deal with him first.
There are two ways to deal with the Troll. First, give him something valuable and he will let you cross. Or two, get the Goat and knock him off the bridge—which is the best option. For more information on dealing with the Troll, see the Gnome's Island.