Case Study: Building The Augmented Reality iOS Game “Match the ‘Moji” using ARKit and SwiftUI

Match the ‘Moji is an Augmented Reality iOS game that challenges users to recreate the facial expressions on their favorite emojis.

Goals

My goal with this project was to create a game utilizing the ARFaceAnchor blend shapes features of Apple’s ARKit platform to learn more about the capabilites and challenges of developing native AR games for iOS. I set out to implement the following elements:

• Ability to identify and categorize user’s facial expressions
• Match these expressions with identified expressions of emoji images
• Create a clean and simple UI using SwiftUI
• Enhance the user experience through the use of haptics, audio, and on screen visual elements

Challenges

The biggest challenge was taking the input of the user’s face data and processing this to see if they were “matching” the emoji image on screen, and doing this in a scalable way that would be quick to apply to a large selection of images (and allow more images to be added in the future). To achieve this effect, I was able to use custom enums to boil all the facial expressions I would need into a core list (ex: for eyebrows, I could have “neutral”, “surprised”, “furrowed”, or “split / skeptical”).

enum eyebrowScale: CaseIterable {
    case neutral, surprised, furrowed, splitSkeptical
    
    var string: String {
        switch self {
        case .neutral: return "Neutral"
        case .surprised:
            return "Suprised"
        case .furrowed:
            return "Furrowed"
        case .splitSkeptical:
            return "Skeptical"
        }
    }
}

By breaking the facial expressions down to core enums, this allowed be to assign enum values quickly to the emoji images (instead of needing to assign specific data points to each image, which could get complicated and hard to manage fast).

I then was able to use ARKit to access the blend shape data from specific points on the ARFaceAnchor, and use a function to assess what enum result the user was currently returning.

 mutating func eyebrowCheck(eyebrowInnerUp: Float, eyebrowDownLeft: Float, eyebrowDownRight: Float) -> eyebrowScale {
        
        var result = eyebrowScale.neutral
        
        if eyebrowInnerUp > 0.6 && eyebrowDownLeft == 0 && eyebrowDownRight == 0 {
            result = .surprised
        } else if eyebrowInnerUp > 0.1 && ( (eyebrowDownLeft < 0.3 && eyebrowDownLeft > 0 ) || ( eyebrowDownRight < 0.3 && eyebrowDownRight > 0 )) {
            result = .splitSkeptical
        } else if eyebrowDownRight > 0.7 && eyebrowDownLeft > 0.7 {
            result = .furrowed
        }
        
        return result
    }

This made it much easier to check to see if the user’s enum result matched the enum assigned to the emoji.

Enhancing User Experience

In an effort to make the experience more engaging for users, I’ve tried to add whimsy and polish throughout the experience:

• Used CoreHaptics to add haptic feedback throughout the experience to make it clear when a user had successfully completed an action.
• Created simple custom audio effects for certain elements in Garage Band (including a success audio effect, a countdown audio effect when the timer is almost at the end, and a game complete audio effect that matches with a custom built haptic experience).
• Using Reality Composer, I created a simple green-ring animation to use when a user was successful in matching an emoji, and I used ARKit’s PersonSegmentation to place this ring behind the user for an added level of immersion.
• If the user taps on the large emoji either on the main screen or during the game, a simple animation paired with haptics will play and transition to a new emoji — this can be used to skip an emoji if it’s too challenging for you in the game!
• Using SwiftUI, I was able to build the UI for the experience with minimal code, which was great for getting started fast and being able to iterate quickly to test out what styles worked or didn’t for the simple look I wanted to achieve.

Conclusion

The development of this experience was a challenging but great learning experience that allowed me to continue to build up my Augmented Reality skills while building a game that will bring delight to users.

If you are interested in checking out the game, it can be downloaded here:

‎Match the 'Moji - AR Face Game

If you are interested in checking out the code, the GitHub repository for this project can be found here:

GitHub - coledennis/Emoji_Face_Match