After I conceived the idea of doing a visual novel, the question was obvious, could I do it? and how? After researching the developments tools (RenPy, that was easy), I needed to assess if I could generate the type of visual assets I needed for a visual novel. Static images weren't going to be an issue, but a VN usually adds the reaction of the characters to a conversation, so I needed to find out if I was going to be able to change the faces of the characters to show expressions while preserving the character.

The changing part is easy with AI, you just make it generate a new face and add a token to match the desired expression, over the current image. Either inpainting or by using a more AI driven technique, which detects the face in the image, and automatically knows where to inpaint. I was already using ComfyUI (a node based interface for Stable Diffusion) so introducing a FaceDetailer node was very easy.

FaceDetailer is a vital part of any of my workflows, when the face is below a size, the lack of pixels cannot make a good face. So FaceDetailer or Inpainting will work at a bigger size and then shrink the image to the available space. A face in the background of any AI generated image might look like this:

At this point I hadn't started to write the story yet, so I took an image I liked, and by setting seeds to fixed and changing the prompt to add several types of expressions, and some cherrypicking, I had enough to make a small sample animation:

https://imgur.com/a/p126rmt

As you can see I had already decided that I wasn't doing background+foreground characters speaking, as in most VN, and the characters were going be changing in the background image. (In  the first act, I have used this for every scene but one), so I wasn't going to have assets for characters. This has advantages and disadvantages: the advantage, is that the clothes would change between scenes, which I wanted to do, and the disadvantage is that I would have to preserve the face between scenes. But at the moment I was just thinking about preserving the face during the same scene.

I tried with several characters and the results were ok...sometimes. It was encouraging enough to begin to work, and I would make a decision about a more complex solution at a later point.

With my computer setup, it was really all I could afford to do at the moment.

When I decided to present the my idea in public (well in 4chan, really), one thing I resented, is that the standards for a painterly western style in face consistency seemed to be much more higher than what I expected. Anime and western comic-books rely heavily in codification of characters by features that are not the face (clothing, spandex suits, hair style and color), but the same standard wasn't admissible when you try to make it more elaborate. It's easy preserving the face when there is just one face in total. In any case I knew I had room for improvement and I would more than likely need to do it.