The number of visitors to the Inverness Castle Viewpoint fell by more than half in its second week, following an opening week of free access.