There are no hard and fast rules for how to customize the css in Vanilla. You can alter the main css file if you like (it is your install, after all). Or you can create a new folder in the styles folder named whatever you want and create a whole new style - then use the administrative settings to shift the forum over to your new style.
Just in case you haven't seen it, there are a bunch of custom styles over on the docs site: