Ah, I learn something new everyday. 
Even though I’ve enabled web view debugging (I’m guessing this is done with WebView.setWebContentsDebuggingEnabled()), I do not see any change in the XML layout returned from Inspector. All UI elements still appear as Android native elements (on Android API 21).
Granted, I never had to handle a lot of WebView work before, so I’m not sure if this is a regression in Appium or the Android and iOS automation libraries.
EDIT: The WebView inspector in Chromium works, though!