Sentence length in English writing in early 1900s vs. English writing now

I am reading a book called The Best American Essays of the Century. One thing I keep noticing is a lot of the sentences in this book are very long — 5 to 6 lines. I read a lot of articles on the web, and the difference is obvious to me. In my opinion, contemporary writers keep their sentences short. Obviously there is not one way to write, but what I learned in college was to not make sentences too long and keep varying the length of sentences — not just too short in particular.

Now coming back to the question. What was the reason for this paradigm shift? I understand languages, including English, evolve over time. What was the main reason(s) behind the change?

According to a study of the Corpus of Historical American English, the phenomenon the OP notes is real and hold across all genres. Nailing down one exact cause is unlikely, but contributing factors that have been explored for this change in the relevant literature include:

• the informalization and colloquialization of written language

• a trend toward less explicitness

• a rise in use of compressed and complex noun phrases

• fewer relative clauses in scientific writing

• the growth of mass readership and competition for readers

• changes in punctuation conventions

• etc.

The following from Karolina Rudnicka's 2018 Variation of sentence length across time and genre.

The results of the comprehensive analysis of sentence length across time (1810 – 2000) and genre show a decrease in sentence length for all the investigated genres. The degree of the decrease is slightly different for each genre, namely, in fiction sentences are approximately (on average) 6.5 words shorter now than they were in the beginning of the nineteenth century, for magazine the decrease amounts to 10.2 words, for newspaper 4.9, for non-fiction 8 and for movie & play script 8.3 words. The initial sentence lengths were, however, different for each genre, with movie & play script starting from the lowest sentence length and finishing at the lowest sentence length from all genres and non-fiction starting and staying the genre with longest sentences.

Various explanations have been offered to account for the observed decrease in sentence length across time and genre. Among the processes at work which might influence sentence length there are informalization of the language of the media (Leech et al. 2009: 239), colloquialization (Mair 1998: 153), a trend towards less explicitness in writing (noted by e.g. Biber & Gray 2010), trends accounting for changes in the scientific discourse such as the increase in the use of compressed and complex noun phrases (Gross et al. 2002: 171; Dorgeloh 2005: 92; Hund et al. 2012: 236) and a decrease in the relative clause frequency (Gross et al. 2002: 171; Hundt et al. 2012: 225). In the non-fiction genre, the decrease happening between 1990 – 2009 might, at least partially, be attributed also to the addition of COCA contents (mostly texts from peer-reviewed academic journals).

The more marked decrease in sentence length observed around the beginning of the twentieth century might have to do with the development of mass readership (Schneider 2002, quoted in Fries 2010: 23) and competition for the potential reader (Westin 2002: 161) and a change in punctuation conventions (noted by e.g. Fahnestock 2011: 256 and Biber & Conrad 2009: 153) which as such might actually be the resultant of the two previous phenomena. The above mentioned processes and developments might account for the decrease in sentence length across particular written genres but they do not explain the observed decrease in the proxy for the spoken language – movie & play script. The question whether the observed decrease would also be visible in the spoken language and which processes could possibly account for it remains open.