Sampling words from Darwin’s Origin of Species

Load libraries

library(mosaic)
library(dplyr)
library(googledrive)
library(googlesheets4)
library(flextable) 
set_flextable_defaults(fonts_ignore=TRUE)

Read in new data:

darwin <- read_sheet("https://docs.google.com/spreadsheets/d/1M8QjSDMP-gw8RqFW-O_RvFQY8HkaOVxH7oOSkJ-uUQM/edit?usp=sharing")
darwin<-darwin[-1,] # Git rid of my test response

Rename some of the variables to make them easier to work with

names(darwin)[3]<-"mean.char"  
names(darwin)[2]<-"Sampling.method"  

Read in the passage from Darwin so we can calculate the true mean number of characters

oas<-read.csv("data/Darwin.csv")

True population mean - this is what we are trying to estimate!

(mean.nchar<-mean(~nchar, data=oas))

[1] 4.936652

Distribution of sample means from students’ samples

gf_histogram(~mean.char, data=darwin, xlab="Mean Number of Words") %>% 
  gf_vline(xintercept=~4.93, col="red") %>% 
  gf_vline(xintercept=~mean(~mean.char, data=darwin, na.rm = TRUE),
           col = "blue")

`stat_bin()` using `bins = 30`. Pick better value `binwidth`.

Mean (of the means) for the class

mean(~mean.char, data=darwin, na.rm = TRUE) 

[1] 5.977778

We see that, on average, the mean number of words in the sample taken by students is higher than the population mean (4.93 words) Bias = the difference between the blue and red lines

(bias<-mean(~mean.char, data=darwin, na.rm = TRUE) - 4.93)

[1] 1.047778

# Methods used by students
methods <- darwin %>% select(Sampling.method) %>% data.frame() 
flextable::flextable(methods, cwidth=60)%>%
  width(width=6)

Sampling.method
I tried to pick words that appeared several times within the text. Words like; individuals, variability, have, probability all appear several times in the text. I chose this since they show up the most and the goal was to figure out the average length of the words that Darwin uses. I also chose words like cultivation that only show up once, but the past tense of the word shows up like cultivation and culivated and cultivate. So I came up with the words: individuals, variability, have, probability, cultivated, generations, our, which, varied, and still.
I chose mostly connecting words because they are the most commonly used, I also chose words that were used frequently.
I chose them based on how frequently they appeared within the text and with some of the words with the same amount of letters (especially the 2 or 3 letter words), I just picked one at random to account for them all.
I randomly selected them by closing my eyes and noting the first word that I saw
I wanted to be as unbiased as possible, so I picked a random word in the first line and went down in a vertical line and took the average of those words in that line.
Tried to pick some small, some large, and more in the middle.
I chose a random sentence in the middle of the paragraph.
picked the first 10 words
I just randomly started at a sentence in the passage
I chose lots of small connecting words, mixed in with longer words since there are several longer words.
I chose them completely based on feeling.
the longest words
randomly put my finger on a starting word in the paragraph
I tried to pick a mix of long, short, and medium-length words to get a somewhat more accurate average
Picked every 4th word
words that I feel like contribute to random things within natural resources, that were meaningful and not filler. I know that is biased
I chose my ten words after reading the passage and writing the words that I saw appear more than once in the passage
whatever I looked up and immediately saw
I would pick the word I see first, and would exit the tab to write it down and then go back and write down the word I see first again.
I looked for words that I associate with Darwin and the theory of evolution.
beings exposed during several generations cause great amount of variation
I scanned over the entire passage and whatever word my eyes landed on I chose for the sample.
I used the words on the left hand side of all ten lines as the passage appears on the canvas page. These had a mix of words at the beginning, middle, and end of sentences which felt
I chose my 10 words by picking random words in each line, one line having two words.
I chose my 10 words through just randomly seeing a word that caught my eye.
I counted every 11th word until I reached 10 samples
I closed my eyes and moved my cursor to a random part of the page. I did this until I hovered over 10 different words.
counted every 11th word until I hit a sample of 10
Pasted the text into a word-counter, then took the average amount of characters in the ten most common words
Sub-variety, cultivated, generally, diversity, domestic, productions, conditions, variability, organism, and generations. I chose them because they're all around 10 letters and the same comprehensibility level.
I randomly chose 2 words from each sentence of the text.
I chose main topic words such as plants and animals and then generations to give a solid background and then I started picking more action words to show what is happening. Like varying, different, conditions. Then lastly with my few remaining words I chose things such as generations and cultivation to kind of tie everything together.
I notice that there are exactly 10 rows of this paragraph on my screen. SO I just randomly chose 10 words from each paragraphs.
Every 22nd word (because there were 221 words total)
close my eyes and randomly draw circles on the screeshot of that piece. count the word that's closest to where the circle begins as my sample.
I estimated the total number of words and then used a random number generator to choose 10 numbers

Add on a “year” column to this year’s data

darwin <- darwin %>% mutate(year = "2026")

Compare to data from 2020, 2022, 2023…

Read in data from past years

darwinold<-read.csv("data/Darwin2025.csv")

For some reason, the columns are arranged differently in the old and new data sets. We can rearrange the columns using the select function

darwin <- darwin %>% select(Timestamp, mean.char, Sampling.method, year)
darwinold <- darwinold %>% select(Timestamp, mean.char, Sampling.method, year)
darwinall <- rbind(darwin, darwinold)

Create a multi-panel plot to illustrate the results from each year

gf_histogram(~mean.char | year, data=darwinall, xlab="Mean Number of Words") %>% 
  gf_vline(xintercept=~4.93)

`stat_bin()` using `bins = 30`. Pick better value `binwidth`.

Warning: Removed 2 rows containing non-finite outside the scale range
(`stat_bin()`).

mean(~mean.char | year, data=darwinall, na.rm = TRUE)

    2020     2022     2023     2024     2025     2026 
5.854107 5.607442 5.995349 6.246154 5.990244 5.977778 

(mean.nchar<-mean(~nchar, data=oas))

[1] 4.936652

Random sampling

With random sampling, however, we should get estimates that are, on average, equal to the population mean. Lets explore this by taking 10,000 random samples and computing the mean number of words in each sample!

randomsamps<-do(10000)*{
  samp.char<-sample(oas, 10)
  mean(~nchar, data=samp.char)
  # Alternative way to accomplish the same thing in 1 line of code
  #mean(~nchar, data=sample(oas, 10))
}   

gf_dhistogram(~result, data=randomsamps, xlab="Mean Number of Words",
          main="Random Sampling") %>% gf_vline(xintercept=~4.93)

`stat_bin()` using `bins = 30`. Pick better value `binwidth`.

mean(~result, data=randomsamps)

[1] 4.94321

## Larger samples -> less variable estimates!

Lets look at what sorts of estimates we would have gotten if we had sampled 50 words instead of 10. We find that our estimates would again be centered around the population mean, but there would be less variability in our estiamtes from sample to sample.

randomsamps<-do(10000)*{
  samp.char<-sample(oas, 50)
  mean(~nchar, data=samp.char)
  # Alternative way to accomplish the same thing in 1 line of code
  # mean(~nchar, data=sample(oas, 10))
}

gf_dhistogram(~result, data=randomsamps, xlab="Mean Number of Words",
              main="Random Sampling") %>% gf_vline(xintercept=~4.93)

`stat_bin()` using `bins = 30`. Pick better value `binwidth`.

mean(~result, data=randomsamps)

[1] 4.932152

Write out the data for a future lab

write.csv(darwinall, file="data/Darwin2026.csv", row.names = FALSE)