- Byron Bunch, a worker at a mill in Jefferson, recalls the day three years prior when Joe Christmas, a strange, silent and rude man drifted into town and began working at the mill.
- The men at the mill take turns guessing about the stranger, and many assume that he's a foreigner because of his strange name.
- Byron Bunch also remembers how just six months ago another stranger named Joe Brown began working at the mine. Brown has a white scar on his face and laughs, gambles, and drinks far more than he works.
- Byron admits that he knows a lot more about them now than he did back then. Now he knows that Christmas wasn't only working at the mill – he and Joe Brown were selling whiskey illegally (this is during Prohibition so alcohol is illegal) in a cabin owned by Miss Burden.
- Miss Burden is a local recluse whose ancestors were Northerners that settled in Jefferson after Reconstruction in order to advocate for African-American rights.
- Christmas and Brown live on her property and make a good deal of money selling whiskey. The money allows Christmas to buy a new car and quit his job at the mill.
- Brown quits the mill job soon after.
- The men in the town wonder if Miss Burden knows exactly what Brown and Christmas are doing on her property.
- Some say that Miss Burden has had sex with black men in town.
- On a Saturday at the mill, Byron Bunch is working alone. He works on Saturdays in order to stay out of trouble.
- Only Reverend Hightower knows that, on Saturday nights, Bunch leads a choir at a church thirty miles away. The two men hang out and talk two to three nights a week, in Hightower's study.
- One Saturday while Bunch is at the mill, Lena Grove comes into the loading shed looking for a Lucas Burch. Bunch tells her that she must have heard the name wrong since there's no one there named "Burch," only himself, Byron Bunch.
- As Lena shares her story, Bunch falls in love with her. He tells her about the fire going on in town at Miss Burden's place, and about the two odd men named Joe who live in her cabin.
- Lena gets intrigued by Bunch's description of Brown so she asks if he has a white scar on his face. Bunch admits that Brown does, and Lena realizes that she's found her baby's father in Jefferson after all.