I agree with Kellerhaus. The mailboat is fun too.
You mentioned Portsmouth...they have a wonderful Childrens Museum. There is always a whale watch that the kids might enjoy, although it can boring for little ones until you see a whale.
If you hike, Mt Major is a nice little hike with great views from the top. I have seen little ones up there . There are easier trails for shorter legs.
A day trip to the Cannon Mt Tram is a nice trip and you can include Clarks trading post and see the bears.
Have fun. It is a great place to visit and even better to live here
pmj