You wouldn't download an AI

Extracting AI models from mobile apps

If you are drinking the Kool-Aid, Artificial Intelligence is everywhere.

And, unsurprisingly, the corporations with the deepest pockets—those throwing $,$,$$$ at hiring researchers, hoarding data, and training massive models—are leading the pack.

Many apps are starting to integrate AI models directly on the device. This approach is great for frequent inferences (like instant photo filters or real-time object detection) where speed and offline access are key.

And since everything that runs on your device can be accessed by your device, a golden window of opportunity arises for all those who are curious enough to take a look.

The target

Microsoft’s Seeing AI app, dubbed the “talking camera for the blind,” is a helpful piece of technology. You can point it at objects, documents, and even currency, and it will audibly describe what it sees, text-to-speech.

One of its standout features is currency detection, which supports over 17 different currencies and recognizes 225 unique bills. It’s a great tool for accessibility, enabling visually impaired users to independently identify money without relying on others.

But I didn’t choose this target randomly. A member of the Frida Telegram Channel reached out to me, specifically interested in the app’s currency detection feature.

Initial Recon

Android apps are distributed as APK files (Android Package Kit), which are essentially archives containing everything the app needs to run: the application binary, resources like fonts and images, and other assets.

AI models, typically being files themselves, are typically stored alongside these assets.

To dig into the contents of an APK, we can use apktool to decompile it and unpack its structure into a folder for closer inspection.

And after checking the assets folder, we find a file named currency, with a size of over 16 Megabytes, with a companion currency-labels.txt.

This seems like a pretty good file, until it doesn’t.

currency can not be detected by file, binwalk outputs gibberish, and the entropy is a constant 1. These are all pretty good indicators that the file is encrypted.

Going forward

Now one can try to reverse-engineer how the app decrypts the currency file, and developers have a wide suite of tools to practice security by obscurity. You can read the file backwards, shift everything left by 12 bits, then forward by 2, modulo every second byte with 3, subtract your birthday as a UNIX timestamp, blah blah blah.

While it certainly is fun to dig around and follow the trail to having a usable file, I enjoy actually finishing my projects much more.

In the end, the encrypted model with all of it’s weights and biases needs to be loaded into some type of Neural Inference Engine.

There are many Frameworks which you can use, the most popular one on Android is TensorFlow, and for lightweight devices in particularly, TensorFlow Lite.

And looking at the apktool dump again, we can see that TensorFlow Lite is in fact used by com.microsoft.seeingai

After a little bit of googling for documentation, one can find out that the org.tensorflow.lite.NativeInterpreterWrapper has methods which can be used to load a *.tflite model.

Time for Frida

Frida is a powerful dynamic instrumentation toolkit that can run on virtually any operating system, including QNX.

It allows you to attach to running processes and dynamically alter their behavior. With Frida, you can trace method calls, inspect their arguments and return values, and even overwrite methods to include your own logic.

checkKey(key)? Sure, return true. It is incredibly powerful.

For mobile app research, Objection is particularly handy—it’s a CLI tool that bundles a collection of Frida scripts, making it easy to perform common tasks.

Let’s trace all function calls to org.tensorflow.lite.NativeInterpreterWrapper

android hooking watch class org.tensorflow.lite.NativeInterpreterWrapper

As you can see: On startup the app calls .createModelWithBuffer(java.nio.ByteBuffer, long)

That java.nio.ByteBuffer is likely our unencrypted *.tflite model!

Long story short, I will overwrite the method to dump the model to disk.

TensorFlow Lite Dumper - Frida Codeshare

The script can take a while, especially when you are running Android in a slow emulator like me.

One adb pull later I have the currency.tflite on my device!

To verify its contents, I used Netron, a web-based neural network visualization tool. If Netron could read the currency.tflite file, it meant I had successfully dumped the actual TensorFlow Lite model, complete with weights and biases.

Bingo! We successfully extracted the currency detection model, with all weights & biases.

CTRL + C; CTRL + V

Since my script dumps models by hooking TensorFlow Lite, we can leverage this attack for any app that uses it.

Adobe Scan is a nice app that I actually use quite frequently, the best feature is scanning documents with your phone camera.

Would be a shame if someon… oh 🤭

Appendix

Ironically, I wrote the article and the script during the 38C3, whose motto this year is “Illegal Instructions.”

And everything shown in this article may or may not be an illegal instruction.

Keep in mind that AI models, like most things, are considered intellectual property. Before using or modifying any extracted models, you need the explicit permission of their owner.

PS: A lot of you folks are discussing about whether or not AI models are considered IP. I don’t know if they do, neither do I write the law.

This is the “cover-my-ass” chapter, feel free to ignore.