How Airbnb makes use of visible attributes to boost the Visitor and Host expertise
By Yuanpei Cao, Bill Ulammandakh, Hao Wang, and Tony Hwang
On Airbnb, our hosts share distinctive listings all around the world. There are a whole lot of thousands and thousands of accompanying itemizing photographs on Airbnb. Itemizing photographs comprise essential details about type and design aesthetics which are troublesome to convey in phrases or a set checklist of facilities. Accordingly, a number of groups at Airbnb are actually leveraging laptop imaginative and prescient to extract and incorporate intangibles from our wealthy visible information to assist company simply discover listings that swimsuit their preferences.
In earlier weblog posts titled WIDeText: A Multimodal Deep Studying Framework, Categorizing Itemizing Pictures at Airbnb and Amenity Detection and Past — New Frontiers of Pc Imaginative and prescient at Airbnb, we explored how we make the most of laptop imaginative and prescient for room categorization and amenity detection to map itemizing photographs to a taxonomy of discrete ideas. This submit goes past discrete classes into how Airbnb leverages picture aesthetics and embeddings to optimize throughout numerous product surfaces together with advert content material, itemizing presentation, and itemizing suggestions.
Enticing photographs are as very important as value, critiques, and outline throughout a visitor’s Airbnb search journey. To quantify “attractiveness” of photographs, we developed a deep learning-based picture aesthetics evaluation pipeline. The underlying mannequin is a deep convolutional neural community (CNN) skilled on human-labeled picture aesthetic ranking distributions. Every photograph was rated on a scale from 1 to five by a whole lot of photographers based mostly on their private aesthetic measurements (the upper the ranking, the higher the aesthetic). Not like conventional classification duties that classify the photograph into low, medium and high-quality classes, the mannequin was constructed upon the Earth Mover’s Distance (EMD) because the loss operate to foretell photographers’ ranking distributions.
The anticipated imply ranking is very correlated with picture decision and itemizing reserving chance, in addition to high-end Airbnb itemizing photograph distribution. Ranking thresholds are set based mostly on use instances, corresponding to advert photograph suggestion on social media and photograph order suggestion within the itemizing onboarding course of.
Airbnb makes use of promoting on social media to draw new prospects and encourage our neighborhood. The social media platform chooses which advertisements to run based mostly on thousands and thousands of Airbnb-provided itemizing photographs.
Since a visually interesting Airbnb photograph can successfully appeal to customers to the platform and significantly improve the advert’s click-through charge (CTR), we utilized the picture aesthetic rating and room categorization to pick out essentially the most enticing Airbnb photographs of the lounge, bed room, kitchen, and exterior view. The criterion for “good high quality” itemizing photographs was set based mostly on the highest fiftieth percentile of the aesthetic rating and tuned based mostly on an inner guide aesthetic analysis of 1K randomly chosen itemizing cowl photographs. We carried out A/B testing for this use case and located that the advert candidates with a better aesthetic rating generated a considerably increased CTR and reserving charge.
When posting a brand new itemizing on Airbnb, hosts add quite a few photographs. Optimally arranging these photographs to spotlight a house may be time-consuming and difficult. A number may additionally be unsure concerning the superb association for his or her pictures as a result of the work requires making trade-offs between photograph attractiveness, photograph range, and content material relevance to company. Extra particularly, the primary 5 photographs are crucial for itemizing success as they’re essentially the most continuously seen and essential to forming the preliminary visitor impression. Accordingly, we developed an automatic photograph rating algorithm that selects and orders the primary 5 photographs of a house leveraging two visible indicators: residence design analysis and room categorization.
Residence design analysis estimates how nicely a house is designed from an inside design and structure perspective. The CNN-based residence design analysis mannequin is skilled on Airbnb Plus and Luxe qualification information that assess the aesthetic attraction of every photograph’s residence design. Airbnb Plus and Luxe listings have handed strict residence design analysis standards and so the information from their qualification course of is well-suited for use as coaching labels for a house design analysis mannequin. The photographs are then labeled into completely different room varieties, corresponding to front room, bed room, lavatory and so on, via the room categorization mannequin. Lastly, an algorithm makes trade-offs between photograph residence design attractiveness, photograph relevance, and photograph range to maximise the reserving chance of a house. Beneath is an instance of how a brand new photograph order is recommended. The photograph auto-rank characteristic was launched in Host’s itemizing onboarding product in 2021, resulting in important lifts in new itemizing creation and reserving success.
Unique ordering
Auto-suggested ordering
Past aesthetics, photographs additionally seize the final look and content material. To effectively signify this data, we encode and compress photographs into picture embeddings utilizing laptop imaginative and prescient fashions. Picture embeddings are compact vector representations of pictures that signify visible options. These embeddings may be in contrast towards one another with a distance metric that represents similarity in that characteristic house.
The options realized by the encoder are straight influenced by the coaching picture information distribution and coaching aims. Our labeled room sort and amenity classification information permits us to coach fashions on this information distribution to provide semantically significant embeddings for itemizing photograph similarity use instances. Nonetheless, as the amount and variety of pictures on Airbnb develop, it turns into more and more untenable to rely solely on manually labeled information and supervised coaching methods. Consequently, we’re presently exploring self-supervised contrastive coaching to enhance our picture embedding fashions. This type of coaching doesn’t require picture labels; as an alternative, it bootstraps contrastive studying with synthetically generated constructive and unfavourable pairs. Our picture embedding fashions can then be taught key visible options from itemizing photographs with out guide supervision.
It’s usually impractical to compute exhaustive pairwise embedding similarity, even inside targeted subsets of thousands and thousands of things. To assist real-time search use instances, corresponding to (close to) duplicate photograph detection and visible similarity search, we as an alternative carry out an approximate nearest neighbor (ANN) search. This performance is basically enabled by an environment friendly embedding index preprocessing and development algorithm known as Hierarchical Navigable Small World (HNSW). HNSW builds a hierarchical proximity graph construction that vastly constrains the search house at question time. We scale this horizontally with AWS OpenSearch, the place every node comprises its personal HNSW embedding graphs and Lucene-backed indices which are hydrated periodically and may be queried in parallel. So as to add real-time embedding ANN search, now we have applied the next index hydration and index search design patterns enabled by current Airbnb inner platforms.
To hydrate an embedding index on a periodic foundation, all related embeddings computed by Bighead, Airbnb’s end-to-end machine studying platform, are aggregated and persevered right into a Hive desk. The encoder fashions producing the embeddings are deployed for each on-line inference and offline batch processing. Then, the incremental embedding replace is synced to the embedding index on AWS OpenSearch via Airflow, our information pipeline orchestration service.
To carry out picture search, a consumer service will first confirm whether or not the picture’s embedding exists within the OpenSearch index cache to keep away from recomputing embeddings unnecessarily. If the embedding is already there, the OpenSearch cluster can return approximate nearest neighbor outcomes to the consumer with out additional processing. If there’s a cache miss, Bighead is named to compute the picture embedding, adopted by a request to question the OpenSearch cluster for approximate nearest neighbors.
Following this embedding search framework, we’re scaling real-time visible search in present manufacturing flows and upcoming releases.
Airbnb Categories assist our company uncover distinctive getaways. Some examples are “Wonderful views”, “Historic houses”, and “Artistic areas”. These classes don’t all the time share widespread facilities or discrete attributes, as they usually signify an inspirational idea. We’re exploring automated class growth by figuring out comparable listings based mostly on their photographs, which do seize design aesthetics.
Within the 2022 Summer season Launch, Airbnb launched rebooking help to supply company a clean expertise from Neighborhood Assist ambassadors when a Host cancels on quick discover. For the aim of recommending comparable listings all through the rebooking course of, a two-tower reservation and itemizing embedding mannequin ranks candidate listings, up to date each day. As future work, we are able to contemplate augmenting the itemizing illustration with picture embeddings and enabling real-time search.
Pictures comprise aesthetic and style-related indicators which are troublesome to precise in phrases or map to discrete attributes. Airbnb is more and more leveraging these visible attributes to assist our hosts spotlight the distinctive character of their listings and to help our company in discovering listings that match their preferences.
Keen on working at Airbnb? Take a look at our open roles.
Because of Teng Wang, Regina Wu, Nan Li, Do-kyum Kim, Tiantian Zhang, Xiaohan Zeng, Mia Zhao, Wayne Zhang, Elaine Liu, Floria Wan, David Staub, Tong Jiang, Cheng Wan, Guillaume Man, Wei Luo, Hanchen Su, Fan Wu, Pei Xiong, Aaron Yin, Jie Tang, Lifan Yang, Lu Zhang, Mihajlo Grbovic, Alejandro Virrueta, Brennan Polley, Jing Xia, Fanchen Kong, William Zhao, Caroline Leung, Meng Yu, Shijing Yao, Reid Andersen, Xianjun Zhang, Yuqi Zheng, Dapeng Li, and Juchuan Ma for the product collaborations. Additionally thanks Jenny Chen, Surashree Kulkarni, and Lauren Mackevich for enhancing.
Because of Ari Balogh, Tina Su, Andy Yasutake, Pleasure Zhang, Kelvin Xiong, Raj Rajagopal, and Zhong Ren’s management assist on constructing laptop imaginative and prescient merchandise at Airbnb.
