I have this same problem. On the free flight app, the graphic showing which way the controller is facing.
When my phone which is connected to the controller is pointing towards the anafi then all is good but the controller is now facing 45 degrees away from the anafi. Then as i turn myself and face the anafi with the controller now also facing the anafi the graphic now show im 45 degrees in the other direction and not lined up and showing red.
So to conclude the app only sees which way the phone device is facing and not the controller.
If there is something im doing wrong then Doh. But if its all in the design of the anafi then. Well we see.