When you don't know where to start, start from a threat model. I think about any application in a way like the following:

Mobile application threat model (fadeevab.com(C)2020)
Mobile application threat model

A threat model can be any comfortable to you.

If to imagine any application as a black box, the imaginary boundary could be drawn around the box: the external perimeter where the data flows from/to the outer world.

Static analysis ("grepping") will give you some initial basic ideas: just use grep to find 'http://' links, passwords or any suspicious stuff. Basically, such frameworks like MobSF gives you a lot of info (but not enough) about APK on the initial step of Android application analysis. Application permissions will give you an idea what kind of personal info the application may potentially gather.

Dynamic analysis (tracing/debugging) will give you the actual picture of what happens in  the application: sessions identifier, outgoing traffic, plain data  before the encryption.

Sniff HTTPS Traffic using MITM Proxy (SSL proxy)

View  the traffic that application sends to the network via HTTPS. You may  need it to verify PII leaks or analyze protocols of application level  (e.g. RESTful API). It's a good starting point, as you analyze  application as a black-box.

  1. https://www.mitmproxy.org/
  2. Charles Proxy

Certificate Pinning and Other Certificate MITM issues

Even using SSL proxy you may see that some traffic cannot be decrypted, and it means that application uses either the default system certificate storage excluding user certificates, or certificate pinning for particular URLs.

  1. You don't want to waste your time, so try to find a TrustManager implementation in a case of Android application to see how the chain of trust is configured.
  2. Any MITM proxy doesn't work for Android applications because only system certificate storage is applied as a source of trust excluding user's MITM certificates. Modify APK to overcome this.
  3. Having Android rooted emulator you might have to modify system certificate storage there.
  4. Use certificate unpinning techniques to change the pinned certificate and analyze the traffic: use Frida to inject the certificate unpinning code snippet (and you don't need a root on Android for using Frida).
  5. Another option: reverse engineer, modify and re-pack the application putting a MITM certificate into APK instead of a pinned certificate.

iOS Application Security Testing

  1. Jailbreak iOS device for the further security research.
  2. Connect device to the same network (e.g. Wi-Fi network).
  3. Install Frida (https://frida.re), SSH daemon.
  4. Connect to device via SSH (ssh root@192.168.0.111, login: root, password: alpine). Traverse file system using bash tools; use scp to copy some files from the phone to the host.
  5. Install iOS application from App Store, extract IPA (iOS application package), test it with MobSF.
  6. Decrypt application using frida-ios-decrypt, Clutch, dumpdecrypted.dylib.
  7. Trace filesystem, network, encryption:
frida-trace -m "-[NSFileManager fileExistsAtPath:]" -U <pid/appname>
frida-trace -m "-[NSURLSession dataTaskWithRequest*]" -U <pid/appname>
frida-trace -i CCCrypt -U <pid/appname>

Android Application Security Testing

  1. Test APK with MobSF: https://github.com/MobSF/Mobile-Security-Framework-MobSF. MobSF gives much more insights about Android app than about iOS app.
  2. Unpack classes.dex from APK file, use grep looking for http links, password, key, paths.
  3. Use jadx (jadx-gui) to decompile APK (just skip d2j-dex2jar tool cause it's pretty outdated nowadays).
  4. Use Frida (https://frida.re). To avoid rooting, use frida-gadget: it's is the library which you have to insert into APK, re-pack/re-sign and then adb install -r you-self-signed-with-gadget.apk. Then run application and then frida-trace -U Gadget.
  5. Look into libs/ subdirectory of APK: if there are native shared libraries inside, then you may use strings libblabla.so | grep password and r2 libblabla.so to do some reverse engineering using Radare2 (or use IDA, but it's not free).
  6. Many of the tools above use apktool under the hood. I would recommend to install the version 2.4.1 and higher to save your time catching the errors related to the older versions.

Use OWASP MSTG to Get More Ideas

It is here: OWASP Mobile Security Testing Guide