AI beauty assistant for next-gen cosmetology services

Non-functional requirements as the driving force behind software development decisions

To help our AI team ensure proper LLM model fine-tuning, the client specified critical non-functional requirements for their AI chatbot solution:

• High performance. Answers should be provided to users within seconds.
• Reliability. The chatbot should provide only accurate, personalized answers and not overgeneralize.
• Security. The chatbot should withstand typical AI attacks including prompt attacks, training data extraction attacks, and token manipulation.
• Consistency. Answers that the chatbot provides should be in a logical and natural order.
• Continuity. The assistant should easily refer to previous conversations with the user and previously stated preferences and information.

These requirements guided our choice of optimal model training approaches, such as retrieval-augmented generation (RAG), which helped us ensure the model provides consistent, continuous, and reliable responses, avoiding overgeneralization and hallucination.

In terms of security, we primarily ensured that the model couldn’t use any data except outside the client’s knowledge base and enabled security controls for secure application access and user data storage.

To ensure high performance, we optimized the model for fast inference (text generation) by developing a custom inference pipeline.

Development of a proprietary knowledge base engine for providing the AI model with long-term memory

Our data engineering team worked closely with the client’s team of cosmetology domain experts to codify their extensive area knowledge and structure it in the most suitable way for effective LLM model training and an accurate retrieval-augmented generation process. This knowledge base contains instructions for the model on how to:

• differentiate between cosmetology procedures and beauty products that the clinic sells and offer the most suitable ones based on a user’s selected age, skin type, and lifestyle habits
• distinguish between different skin conditions based on photos and images
• explain the pros and cons of each cosmetology procedure and forecast the long-term effects of each procedure on the user’s skin
• compose a suitable skincare routine based on a user’s needs, requests, preferences, and photos
• evaluate a user’s current skincare routine
• provide advice on skincare routines with products listed in the right order of application (devised by cosmetologists)
• offer general advice on lifestyle improvements, such as recommending facial massage techniques, face yoga exercises, physical activity frequency, and food types best suited to maintaining proper skin health

The Yalantis data team built the knowledge base with the help of OpenAI GPT-4, which was used to process the data and generate lists of question-and-answer pairs based on it. These questions and answers were subsequently used to build hierarchical embeddings and fine-tune the main LLM. The knowledge base is regularly managed and updated so that the model can offer users only relevant cosmetology procedures and beauty products.

Ensuring a multimodal LLM for advanced AI assistant capabilities

To ensure the AI beauty chat assistant was capable of not only generating text responses but also processing and analyzing visual information (photos of users’ skin sent in the chat), the Yalantis AI team decided on a multimodal LLM approach to ensure the model is not only capable of inference but can also analyze visual information. Yalantis AI engineers used an existing large language and vision assistant for biomedicine (LLaV-Med) and trained it on data available in the cosmetology knowledge base to customize its responses and accomplish the following critical tasks:

• Maintain a friendly human-like conversation with application users
• Provide structured responses (for example, in the form of a bulleted or numbered list) to users’ questions about skincare routines and cosmetology procedures
• Offer skincare product cards to help users immediately analyze each product’s key characteristics without needing to follow external links
• Enable a computer vision component of the AI assistant to help cosmetology application users assess their initial skin condition and skin improvement progress

Development results: Key AI beauty assistant features

• Creation of individual skincare routines. The model can analyze a user’s information provided during app registration and when answering AI beauty assistant’s questions t, such as location, age group, skin tone, and skin condition, to compose a personalized skincare plan.
• Recommendations on suitable skincare products and procedures. The model refers to its vast knowledge base of cosmetological expertise to retrieve information on the clinic’s procedures and skincare products to recommend to users.
• General recommendations on skin health. The personal AI beauty assistant can also maintain a general conversation about skincare improvement. The tone of this conversation is friendly and approximates a consultation with an experienced human cosmetologist.
• Image analysis to check results. Users can send photos of their faces to the chat to get an initial skin assessment upon registration. They can also take before and after photos when going through a cosmetology procedure or changing their skincare routine to spot changes, track results, and change the flow if the results aren’t satisfactory.