The idea that in a masterpiece everything contributes to the whole can be traced back through Hegel's principle of 'organic unity' to Aristotle's theory of tragedy. For Coleridge, "organic unity" was the manifestation of a poet's powers of synthesising. It was Baumgarten (1714-62, his Aesthetica was written in Latin) who coined the term 'perfectible beauty' according to which the beauty of an artwork or natural object corresponds to the degree of its organization or integration. In the ideal case no elements of an object should appear arbitrary, accidental, or irrational - it's a microcosm of the greater 'totality'.
With order and harmony comes unity and a way to distinguish what "belongs" from what doesn't. Workshop criticism of short stories and poetry is often harsh on the apparently inconsequential - anything not contributing to the main theme is easily dismissed as not belonging. But the need for 'perfectible beauty' is open to challenge, just as erstwhile requirements for symmetry or circles were. Order brings unity but also predictability and claustrophobia; items appearing in the first paragraph of a short story doomed to reappear sooner or later.
It was from such predictability that surrealism reacted. When editing "Le Chien Andalou", Dali and Bunuel removed anything that resembled a logical connection. Dada went further, introducing randomness. Since then, people have experimented with paper cut-ups and computer generated work, and the Oulipo school have invented arbitrary rules to generate new texts from old.
Others have escaped by foregrounding performance or improvisation. Without going as far as randomness, they have adopted the more relaxed, inclusive and baggy approach of Laurence Sterne or the New York school, producing mosaics of interesting fragments and loose ends. Beginning their work from life they retain some of its untidiness in dialog and plot, leaving the work open; a transcript of a process rather than unified and closed project.
Order can be maintained on a large scale while allowing local variation and surprise. The idea of establishing a norm from which one can profitably deviate also goes back to the Greeks. Expectation can be thwarted by: delaying the expected feature (Ogden Nash's long lines delaying the final rhyme); omitting it entirely (calling a non-sonnet a sonnet); or providing an alternative to the expected feature (a pararhyme). It can be a large or small scale phenomenon: a phonetically or semantically unexpected word; deviations from meter; deviations from the expectation of the genre - (killing the hero in scene one; a whodunit that's never solved). As in jazz, the norm can be established in a brief introduction or, if the norm is sufficiently well known, the title alone can be a sufficient allusion, the variations played against an absent melody as a parody might counterpoint the original absent text.
Once the author has signalled the form, the knowing reader will notice (and seek purpose in) variations. Writers may want to exploit the dynamics of anticipation/surprise and tension/relaxation. Some norms have a built-in expectation of variation. If the author exceeds the expected bounds, the reader may accept the work as an extension of the form but more likely the reader, upset by the authors breaking of the reader/author contract, will reject the work.
Martindale suggests that once a literary style has become established writers must work harder to create novelty that will attract and hold the attention of readers; they do so by shifting increasingly to the use of primary process thought in their writing. The stylistic climate for writers and readers may differ in important respects, so writer and reader perceptions of novelty may not coincide.
The readers' expectations will modify as they read. For example, they may initially expect a happy ending. If later a happy ending is too strongly prefigured, the reader may expect a twist in the tail. If finally there is no closure they may feel tricked rather than surprised. Too many failed predictions may result in the reader not trying to anticipate. To discourage this passivity, the writer may adopt a strong form (a whodunit) or use plot crises and line-breaks to force premature interpretations. Adumbration can be used to lull the reader into a false sense of security.