Śrīla Bhakti Vinod Ṭhākur comprehensively addresses the issue.
Deity Worship and Idolatry
Om Viṣṇupād Śrīla Bhakti Vinod Ṭhākur
Translated to English from the Bengali article
Śrī-Mūrti-sevā o Pautalikatā
published in Gauḍīya magazine.
Let us now briefly analyse what idolatry is. Those who worship objects that have no connection with the Lord’s form are idolaters. They are of five types: (1) those who, lacking proper knowledge, worship matter considering it the Lord;⁴ (2) those who deem matter insignificant and worship consciousness of its opposite, considering that to be the Lord;⁵ (3) those who have decided the Lord has no form but, because nothing can be thought about other than form, imagine a material form of the Lord to make worshipping Him easier;⁶ (4) those who imagine and meditate on a form of the Lord to purify and elevate their own consciousness;⁷ (5) those who worship the soul, considering the soul to be the Lord.⁸
Uncivilised forest tribes, fire worshippers, and the Greeks who worshipped Jupiter, Saturn, and the other planets are in the first class of idolaters. When people have no understanding of the Lord’s form yet have natural faith in the Lord, then it is seen that, in ignorance, they worship the Lord through shiny objects. This is this first class of idolatry. Considering the qualification of such persons, we should not criticise such idolatry.
When the quality of being impersonal, the opposite of all material qualities, is believed to be the Lord through logic on the basis of extensive cultivation of material knowledge, the second class of idolatry begins. Idolaters of this class believe the Lord is only formless. The quality of being impersonal can never be the form of the Lord or a quality related to the form of the Lord. But if we say impersonality is one feature amongst the Lord’s innumerable features, then it can be a quality related to the Lord’s form. The Lord’s form is different from matter, but that form is not the opposite of matter.
Those who aim ultimately at self-extinction (nirvāṇa) and imagine deities of Viṣṇu, Śiva, Prakṛti, Gaṇeś, and Sūrya to be means of reaching that do not acknowledge the eternal form of the Lord. Therefore, because they serve an imaginary deity, they are counted as the third class of idolaters. What is today called “fivefold worship” (pañcha upāsanā) is the idolatry of this class. How the opposite of a particular quality or the state of being free from that quality can be attained by immersing oneself in that quality is not intelligible.
The yogīs’ meditation on an imaginary deity of Viṣṇu is the fourth class of idolatry. Thereby, other types of attainment may be had, but the highest attainment of direct perception of the Lord’s eternal form cannot be.
Those who worship the soul considering the soul to be the Lord are the fifth class of idolaters. According to the teachings of Śrīman Mahāprabhu, there is no greater offence than this. All souls are worshippable, and if we worship them as devotees of the Lord, then we do not make the offence of considering them to be the Lord. That worshipping the forms of Śrī Rāma and Śrī Nṛsiṁha is not idolatry can be understood by reading the book Śrī Kṛṣṇa-saṁhitā that I have written.
It is not that the five types of idolaters I have mentioned only criticise the Lord’s form; they needlessly criticise one another as well. The first class of idolaters consider the sky’s quality of being all-pervading to be the foremost quality of the Lord, disregard the Lord’s form, and criticise all imaginary and appropriate forms of the gods.
Amongst those of equal qualification, co-wife-like behaviour and the quarrel that results from it are inevitable. Only idolaters criticise idolaters. Non-idolaters and self-realised devotees of the Lord have no animosity towards idolaters. They simply understand that so long as someone has not realised the Lord’s form, what can they do other than imagine it? As they go on imagining, by virtue of the association of the sādhus, they will come to disregard such imagination and understand the Lord’s form. Then, they will no longer dispute the matter.
yasyātma-buddhiḥ kuṇape tri-dhātuke
sva-dhīḥ kalatrādiṣu bhauma ijya-dhīḥ
yat-tīrtha-buddhiḥ salile na karhichij
janeṣv abhijñeṣu sa eva go-kharaḥ
(Śrīmad Bhāgavatam: 10.84.13)
“Those who think a corpse of made of mucus, bile, and air is one’s self, one’s spouse and family are one’s own, earthen images are worshippable, and bodies of water are holy places, and never associate with the wise are nothing more than donkeys who serve cows.”
antar-gataḥ svavivareṇa chakāra teṣāṁ
saṅkṣobham akṣara-juṣām api chitta-tanvoḥ
(Śrīmad Bhāgavatam: 3.15.43)
“The breeze carrying the nectar mixed with tulasī from the stamens of the lotuses that are the feet of the lotus-eyed Lord entered the nostrils of the sages meditating on Brahma and delighted their minds and bodies.”
prāduśchakartha yad idaṁ puruhūta rūpaṁ
teneśa nirvṛtim avāpur alaṁ dṛśo naḥ
tasmā idaṁ bhagavate nama id vidhema
yo ’nātmanāṁ durudayo bhagavān pratītaḥ
(Śrīmad Bhāgavatam: 3.15.50)
“O Lord who is always called upon, our eyes are delighted by seeing this form You have revealed. We offer our obeisance to You, O Lord, who have appeared before us although You are unmanifest even before those who have conquered their senses.”
kāmais tais tair hṛta-jñānāḥ prapadyante ’nya-devatāḥ
taṁ taṁ niyamam āsthāya prakṛtyā niyatāḥ svayā
antavat tu phalaṁ teṣāṁ tad bhavaty alpa-medhasām
devān deva-yajo yānti mad-bhaktā yānti mām api
(Śrīmad Bhagavad-gītā: 7.20, 23)
“The unintelligent, as a result of their respective desires, worship other gods and adhere to particular rules, being governed by their own natures. The attainments of these less intelligent persons, however, are temporary. The worshippers of the gods reach the gods, and My devotees reach Me.”
jīve ‘viṣṇu’ māni’ ei aparādha-chihna
jīve ‘viṣṇu’ buddhi kare yei brahmā-rudra-sama
nārāyaṇe māne tāre pāṣaṇḍe gaṇana
(Śrī Chaitanya-charitāmṛta: Madhya-līlā, 25.78–9)
“Considering the soul to be Viṣṇu is an offence. Those who think the soul is Viṣṇu or consider Nārāyaṇ equal to Brahmā and Śiva are wicked.”