You did everything right. The public dock etiquette has been - if you are behind someone in line for waiting for a spot and you see a spot that the other boat might fit in, point out the spot and let the other boater know and let them decided if they feel they can fit. Usually if they can not they will motion you to take it. The same applies in reverse. When we are waiting for a spot and someone that is behind us in line sees a spot they think we may fit in, if we do not feel comfortable with the spot - we will motion to that boater or other boater waiting in line behind us (if he/she is in a smaller boat) to take it.
BUUUUTTTT you will always inevitably get some boater who is not as courteous and either doesn't have the same common courtesy or just blatantly ignores the unwritten rules of waiting their turn and blows right in ahead of everyone.
As you can see from everyone else's posts, it is common practice to alert the other boaters (waiting in front of you) when a spot opens up (if you feel that it is too tight a spot for you). But I think these are the boaters who respect the lake, respect other boaters, have the sense of common courtesy and more than likely are familiar with the NH boating rules and regs and are just plain safe respectful boaters. I hope I am in line waiting with them next time I am looking to use the public docks - things go so much more smoothly when everyone works together.
If all else fails - make sure you have that can of air (air horn) ready to alert the discourteous boater and let them no they are out of line. It may not stop them, but at least you get the satisfaction of letting them know they are being a jerk.