Brad Williams was named as Gillespie's replacement for the triangular one-day series with India and New Zealand.
"I pulled up sore this morning following a light bowl... and I came to the decision that I shouldn't go to India," Gillespie said.
"It's not fair on anyone if I went on tour and wasn't 100 per cent fit."
Gillespie strained a side muscle in last week's first test against Zimbabwe and was ordered to miss the second Test, which started on Friday.
The injury-prone paceman was included in a 14-man squad to go to India but only on condition he passed a fitness test.
"It's fair to say that I'm very disappointed but with the right rehabilitation I'm confident that I'll be available for the Test series against India in December."
Gillespie is the third Australian player to pull of the tour, following injured paceman Glenn McGrath and middle-order batsman Darren Lehmann.
The world champions leave for India on October 22. The tournament will be held from October 23 to November 18.
Australia - Ricky Ponting (captain), Adam Gilchrist, Michael Bevan, Andy Bichel, Nathan Bracken, Michael Clarke, Ian Harvey, Matthew Hayden, Brad Hogg, Brett Lee, Jimmy Maher, Damien Martyn, Andrew Symonds, Brad Williams
