Most orthodox religious people pray "towards" something, whether it be an altar, a particular city, the four directions, some prayer beads, up or down to the sky or earth, etc. It's only idolotry if you pray "to" it as if it really was your god itself.
Since the true Noodly Appendages transcend all time-space, having a "focus item" merely allows you to concentrate your attention in the physical world to help you reach out from within yourself. This is like focusing your personal light with a microscope or telescope so you may better see what is normally beyond your reach.
PS (regarding emergencies):
If you find yourself wanting to pray because you're on a ship caught in a massive storm and you have to keep working the rigging so you don't get shipwrecked and die, then taking the time to go pray to/toward some object would be worse than idolotry -- it would be damned suicidal foolishness. Just keep doing what needs to be done and pray (silently so you can still hear orders from the captain) to Horragalles for inspiration and/or mercy. At that point, that's the only god that really matters, so any others will forgive you for it:) -- from personal experience
In case you didn't realize it, I DO have a sense of humor. How about you?
"I will not fear. Fear is the mind-killer... I will face my fear. I will let it pass over and through me, and when it has gone, only I will remain." --The Bene Gesserit
"Time is a spiral. Space is a curve. I know you get dizzy, but try not to lose your nerve." -- Neil Peart
"I'm not in the ship. I am the ship." -- River Tam
"The truth is simple. It's the lies that get complicated." -- me
"No matter where you go, there you are." --Buckaroo Banzai