AI beauty assistant for next-gen cosmetology services
Discover how we developed iOS and Android mobile applications with a built-in AI-powered chatbot to provide clients of cosmetology clinics with real-time personalized recommendations on skin routines and cosmetology procedures.
Period of collaboration
July 2023 – December 2023
About the client
Our client is a US-based network of cosmetology clinics aimed at delivering cutting-edge skin treatment and skin health management services.
The client was looking for a trusted software provider experienced in custom mobile app development and seamless integration of large language models (LLMs). The client’s team decided to work with Yalantis on AI chatbot development based on a recommendation from one of our previous clients.
Their development strategy for 2024–2025 involves leveraging computer vision (for initial skin condition assessment in varying lighting environments) and generative AI solutions in order to:
- promote cosmetology services, boost sales, and expand the customer base
- unburden cosmetologists while improving their productivity and reduce time spent on initial consultations
- increase customer satisfaction and loyalty with a round-the-clock AI cosmetology assistant
The client’s team composed a primary knowledge base for training the LLM that is the core of their AI beauty assistant applications. Core deliverables from Yalantis for this project included:
- a chat-based AI assistant built with LLMs and a computer vision component, leveraging our client’s proprietary knowledge base
- a knowledge base optimized for accurate question-based information retrieval by the AI model
- full-fledged iOS and Android applications
To ensure the solution delivered aligned with our client’s expectations, we followed the Scrum methodology. Scrum allows for iterative and flexible development, which gave us time to test the solution with beta users and implement a time-efficient change management process. Yalantis’ development approach involved transparent quality controls with regular demos and reports so that the client could comprehensively track project progress.
The successful release of AI-enabled mobile applications allowed the network of cosmetology clinics to:
increase the satisfaction rate among existing and new clients by 25%
grow the client base with increased referral frequency
improve cosmetologists’ productivity by 20% with applications gathering critical information about clients and their cosmetological preferences
Non-functional requirements as the driving force behind software development decisions
To help our AI team ensure proper LLM model fine-tuning, the client specified critical non-functional requirements for their AI chatbot solution:
- High performance. Answers should be provided to users within seconds.
- Reliability. The chatbot should provide only accurate, personalized answers and not overgeneralize.
- Security. The chatbot should withstand typical AI attacks including prompt attacks, training data extraction attacks, and token manipulation.
- Consistency. Answers that the chatbot provides should be in a logical and natural order.
- Continuity. The assistant should easily refer to previous conversations with the user and previously stated preferences and information.
These requirements guided our choice of optimal model training approaches, such as retrieval-augmented generation (RAG), which helped us ensure the model provides consistent, continuous, and reliable responses, avoiding overgeneralization and hallucination.
In terms of security, we primarily ensured that the model couldn’t use any data except outside the client’s knowledge base and enabled security controls for secure application access and user data storage.
To ensure high performance, we optimized the model for fast inference (text generation) by developing a custom inference pipeline.
Development of a proprietary knowledge base engine for providing the AI model with long-term memory
Our data engineering team worked closely with the client’s team of cosmetology domain experts to codify their extensive area knowledge and structure it in the most suitable way for effective LLM model training and an accurate retrieval-augmented generation process. This knowledge base contains instructions for the model on how to:
- differentiate between cosmetology procedures and beauty products that the clinic sells and offer the most suitable ones based on a user’s selected age, skin type, and lifestyle habits
- distinguish between different skin conditions based on photos and images
- explain the pros and cons of each cosmetology procedure and forecast the long-term effects of each procedure on the user’s skin
- compose a suitable skincare routine based on a user’s needs, requests, preferences, and photos
- evaluate a user’s current skincare routine
- provide advice on skincare routines with products listed in the right order of application (devised by cosmetologists)
- offer general advice on lifestyle improvements, such as recommending facial massage techniques, face yoga exercises, physical activity frequency, and food types best suited to maintaining proper skin health
The Yalantis data team built the knowledge base with the help of OpenAI GPT-4, which was used to process the data and generate lists of question-and-answer pairs based on it. These questions and answers were subsequently used to build hierarchical embeddings and fine-tune the main LLM. The knowledge base is regularly managed and updated so that the model can offer users only relevant cosmetology procedures and beauty products.
Ensuring a multimodal LLM for advanced AI assistant capabilities
To ensure the AI beauty chat assistant was capable of not only generating text responses but also processing and analyzing visual information (photos of users’ skin sent in the chat), the Yalantis AI team decided on a multimodal LLM approach to ensure the model is not only capable of inference but can also analyze visual information. Yalantis AI engineers used an existing large language and vision assistant for biomedicine (LLaV-Med) and trained it on data available in the cosmetology knowledge base to customize its responses and accomplish the following critical tasks:
- Maintain a friendly human-like conversation with application users
- Provide structured responses (for example, in the form of a bulleted or numbered list) to users’ questions about skincare routines and cosmetology procedures
- Offer skincare product cards to help users immediately analyze each product’s key characteristics without needing to follow external links
- Enable a computer vision component of the AI assistant to help cosmetology application users assess their initial skin condition and skin improvement progress
Development results: Key AI beauty assistant features
- Creation of individual skincare routines. The model can analyze a user’s information provided during app registration and when answering AI beauty assistant’s questions t, such as location, age group, skin tone, and skin condition, to compose a personalized skincare plan.
- Recommendations on suitable skincare products and procedures. The model refers to its vast knowledge base of cosmetological expertise to retrieve information on the clinic’s procedures and skincare products to recommend to users.
- General recommendations on skin health. The personal AI beauty assistant can also maintain a general conversation about skincare improvement. The tone of this conversation is friendly and approximates a consultation with an experienced human cosmetologist.
- Image analysis to check results. Users can send photos of their faces to the chat to get an initial skin assessment upon registration. They can also take before and after photos when going through a cosmetology procedure or changing their skincare routine to spot changes, track results, and change the flow if the results aren’t satisfactory.
Integrate a smart AI model into any skin management software
Increase clients’ satisfaction by providing easily accessible personalized skincare routines and improve your cosmetologists’ and dermatologists’ productivity.