The technique of art is to make objects 'unfamiliar', to make forms difficult, to increase the difficulty and length of perception, because the process of perception is an aesthetic end in itself and must be prolonged. - Shklovsky, "Art as Technique", 1917.
Why are some poems 'difficult'? There are many reasons, lack of control or concern by the writer amongst them. Here I'd like to concentrate on some of the more legitimate reasons for obscurity, borrowing some ideas from maths and the visual arts.
Though poetry may aspire to music, it shares many inherent problems with painting. The discovery of perspective in the middle ages provided a rigorous method of transforming 3 dimensions into 2. We have little difficulty in restoring the lost dimension ourselves. In poetry the loss is more extreme and the restoration more problematic. A poem is, after all, just a long line of characters concertina'd to fit onto the page, a one dimensional thread that began as an idea. in time and space. Whenever dimensions are collapsed there's a risk that information will be lost or distorted. In fact, there's a topological theorem (The "Flower Pressing Theorem") which states that 2 effects are likely
These traits (discontinuity and ambiguity) become more pronounced the more dimensions that are compressed, and modern poetry in particular is susceptible. How does it compensate?
Each kind of representation engenders new artforms and new ways of thinking. Flat patches of paint have interactions missing from the objects they represent. When thought is put into language, the characteristics of the medium, (rhyme, meter, etc) are emphasised by some wordsmiths, just as Matisse, for example, flattened the picture space to emphasise shape and colour until representationalism was all but lost. Look at Picasso's Les Damoiselles d'Avignon or Guernica and you will see flatness and depth on the very same canvas, making the observer's mind change its "focal depth" as it roams. With poetry the shifts aren't so clearly signposted. In The Waste Land attention switches between the symbolic, the declamatory, the prosaic and the oracular so quickly that anyone with conceptual astigmatism is unlikely to get the most from the poem. Some words, like some paintings, aren't meant to be "seen through"; trying will end in bewilderment. Language is not a window upon reality, but stained glass at best, with a glory and interest of its own. Indeed, philosophers like Barthes go further, saying that the lucidity of language is a delusion borne of familiarity.
Cartographers have made a science of compressing dimensions. There's not much of a problem if only a small area is being mapped; one can translate the height dimension into colour or contour. The trouble begins when a "big" subject like the globe is being tackled. Mercator's projection preserves compass directions but not areas, so that the scale varies wildly across a map; 'Equivalent projections' preserve relative size but only by distorting directions. Most projections are compromises. Cartographers are lucky though; at least they can resort to building a model globe. Poets cannot simply "scale down" what they experience. They can only ever make squashed representations.
Even if obscurity and ambiguity were avoidable, they are still useful to the poet. Music is hardly ever representational. Poetry can struggle against it by exploiting obscurity - the more difficult the poem, the less likely is it to be read linearly; the reader is forced to backtrace, to keep the whole poem in mind at once. Discontinuity and sudden switches dislocate the flow, forcing the reader to find other relationships between the parts than linearity, adding depth so that the shapes formed in the mind by the poem aren't restricted by the order the components are presented in. We can roam over the text as one might a painting.
However, the conventions (because that is all they are) that help us restore solidity to a painting can be played upon. Escher's etchings frustrate our expectations, unsettling us with flat images of worlds that cannot exist in just 3 dimensions. Similarly some poems resist interpretation as representations of our familiar surroundings, making new shapes inside us that could not be produced other than by words, and the elbow room to subvert the analogy between the world and words is greater than painters have. As a final example from the visual arts, look at any stereogram. It contains an image if you focus beyond the page. Striking when you see it (and, like some poetry, infuriating if you can't and others can), but the depth can only exist if the surface remains "obscure".
Obscurity may sometimes shield the charlatan but without it we might lose a valuable opportunity to transcend words, a loss we can't afford.