Interesting that this discussion should start now. Just in the last week or so, I tried to test out follow me mode as I was riding my mountain bike. It failed miserably and I wasn't going very fast. It also kept loosing lock on the subject (Me). Looking back at what video I did capture, it appeared that the problem was poor contrast between me and the surroundings. I was wearing a dark shirt and was intermittently in the shade.
A day or so before though, I tried follow me on our Tahoe as we drove on the beach, and the only issue there was that if we got moving too fast, the drone couldn't keep up and would start falling back and eventually give up. I was flying in one of the cinema or photography modes, so I suspect that explains the speed issue. The main difference was, since we were on the beach, the Tahoe was in direct sunlight at all times, so there was plenty of contrast.
In your case, what would worry me about putting the phone in your pack is the amount of time it would take to regain control if something went wrong. On the mountain bike, I had the phone in front of me on a mount and I still **** near lost the drone to a tree. Had it been in a pack, I would have. As a suggestion, you might look at RAM mounts. On my motorcycles, ATV and mountain bikes, I use a 1" ball and an X mount. They come in several sizes to handle even the biggest phone and it would allow you to keep the phone in front of you to keep an eye on the drone even while shredding the dirt.