Here are two threads that touch on the topic, they are quite old, so some things may be a bit different:
http://lussumo.com/community/discussion/1300/
http://lussumo.com/community/discussion/2381/

As far as integrating Vanilla into an existing site, this is a technique I used:
1. Create a file called forum-template on your site i.e. mysite.com/forum-template.html. It should have the look and feel of your site, and be complete with title, stylesheets, etc. as needed. In the content of the file, put nothing but [Forum], this "magic word" is where Vanilla will insert it's content into that file for every Vanilla page view.
2. Install Vanilla in a folder on the site
3. Go into the themes/ folder of the Vanilla folder, and copy the head.php and page_end.php files into the themes/Vanilla/ folder
4. Go in that folder and edit the head.php file you just copied there, adding some code to the bottom like this:$BodyId = "";
if ($this->BodyId != "") $BodyId = ' id="'.$this->BodyId.'"';

// Echo the forum-template.html file, inserting the title, the meta tags, and everything up to the [Forum] tag
$template = file_get_contents($this->Context->Configuration['APPLICATION_PATH']. '..\forum-template.html');

$template = explode('[Forum]', $template);
$this->Context->PageFoot = $template[1];
$template = $template[0];
$template = preg_replace('#<title\b[^>]*>(.*?)</title>#i', '<title>'. $Title .'</title>', $template);
$template = str_replace(array('href="', 'src="'), array('href="../', 'src="../'), $template);

$template = preg_replace('#<\/head>#i', $HeadString .'
</head>', $template);

echo $template . '
<div'.$BodyId.' '.$this->Context->BodyAttributes.'>
<div id="SiteContainer">';
5. Edit the page_end.php file like this:<?php
// Note: This file is included from the library/Framework/Framework.Control.PageEnd.php class.

echo '</div>';
echo $this->Context->PageFoot;
?>