We went to Scotland years ago as well. Inverness didn’t impress us much either. The small fishing village Petershead was nice, some small odd places we stopped by the road also gave pleasant memories (small “castles” with no proper guide, just telling all the odd great stories).
But the highlight was the clan gathering. As my wife is of the clan Hay, it made this extra special. We were invited to lunch and dinner, but what has really stuck as a great memory was the highland games. We went to the one in Aboyne and had a good time there. I assume you’ll think it’s tourist trap, but when we went there 16 years ago, it wasn’t. Felt more like big family gatherings with contests. This might have changed during the years, and I assume the biggest game near Edinburgh(?) with the queen present will draw a crowd.
This is an old tradition only stopped by world wars and has been going on since the 11th century.
So if you’re going back in the end of July, start of August, a day at the highland games is well worth it.