The act of talking through the past in search of understanding often begins with unexpected conversations like one I once had with an African American repairman in my home. As he worked on the project at hand, we talked about many things, including my research. He told me that he had a Southern white ancestor who had married not one, not two, but three African American women, and he had a white wife.
When I first heard this story, it seemed a bit far-fetched. Three black wives? The repairman also said some of his relatives still resided in the house once owned by this white man. They even have his papers and photographs. It is a big house “down a really long dirt road in the middle of nowhere,” he said. I laughed when he said this because he seemed to want nothing to do with this property, although he admitted that his male kinfolk made a workshop of a building that once served as slave quarters.
Though his skin was as dark as mine, the curl in his hair affirmed that he, like most African Americans (and many white ones, too), was of mixed race. As he walked out the door, he said, as if he did not entirely believe it, “Not all white people are bad.”
After he departed, I reflected on what he said about his white male ancestor and wondered whether those four marriages had been short ones because of public pressure or whether this man had just outlived the women he had married. The researcher in me longed to know more. It would have been good to have other accounts. While those in this book are better documented than most, they are still based on slim evidence. They do, however, give voice to many small details surrounding black-white intimacies of the past, among them, that Rice Ballard’s ancestry can be traced to the English county in which Shakespeare was born, though the four children he freed in Cincinnati doubtless never knew this trivial fact.
As the repairman’s truck left the driveway, I took comfort in having been made aware of another relationship that appears to resonate with my subject. While completing this book, I learned about a former coworker’s ancestor who ran off with a freedwoman before the Civil War. This coworker mentioned as much in a memoir capturing his own life as the white husband of a black woman and father of an adopted African child.1 As society’s obsession with genealogy continues, other newly found records will doubtless reveal more individuals whose fears, self-interests, and triumphs tell us more about the “messiness” of black-white encounters.2 Many of them will begin with stories like the one the repairman heard about his ancestors or the one my former coworker shared, stories researchers should take to heart especially because the general public readily sees traces of them in imagined works.
No matter our take on them, interracial relationships are a known legacy of American history. That they are might help explain the success of certain television shows and motion pictures.3 Many such works are more willing to take up the subject of intimacy between African Americans and whites than are scholarly ones.4 There are reasons for this, including the obvious lack of evidence. But the arrival of more and more evidence such as the letters, legal documents, and other sources presented here provide opportunities for scholars boldly to reimagine people of the past, even though stories have been told in certain ways for so long that it is hard to change the narratives.
I initially only wanted to explore the power of fancy girls after reading about them in Deborah Gray White’s now-classic Ar’n’t I a Woman: Female Slaves in the Plantation South.5 While learning about these expensive women and girls, I also delved into the issue of kinship between them and their children, if they were mothers. After stepping back, I realized something difficult had gone unsaid: the degree to which Southern white men figured into these individuals’ lives before and after they were freed. I did not know how to address this topic with the obvious horrors of slavery always in view. I learned how by teaching a class.
It is interesting to see that so much of what is learned goes back to the classroom, even for instructors. It was only after teaching the course African Americans in the City at the University of Alabama while on a graduate fellowship that I saw how changes in nineteenth-century American society and culture, including how people scrutinized one another and how they demonstrated their awareness of what it means to be an individual in a modern and increasingly wealthy and urban society, factored into this project. These related developments helped me craft my central argument. Even if Southern white men were only a loan away from being penniless, they were capable of freeing instead of selling black women and children for whom they cared.
All of a sudden I could see a generational portrait of a particular kind of man—one who was married, had not married, or would never marry, or one who was aging or divorced—who might invest himself emotionally and financially in “relationships” with women and children of color. I struggled with how to name such investments. After looking at five letters from Avenia White to Rice Ballard, her former master, the most appropriate word seemed to be intimacy. This book explores one huge outcome for enslaved women like her, and their children: manumission, specifically, manumission that resulted in the relocation of African Americans who received some measure of white men’s favor to, as I would soon discover, a particular place, Cincinnati.
Such favor existed in my own family. Years before I was born, my fairskinned maternal great-grandmother was the mistress of a white man named Mister Ray. My mother told me she and her sister could charge whatever they wanted at the local general store in their Mississippi Delta town because Mister Ray paid the bill. And whenever my mother and my auntie passed the local dress shop, they were summoned by Miss Diamond, a Jewish shopkeeper, who would say, “Hello, girls! Tell Louella I got some pretty dresses. Mister Ray said y’all can have any you want. Just come on by.” But my mom and auntie had been given strict orders to never enter this or any store without my great-grandma’s permission. In their town, Jim Crow “whites only” signs served as a constant reminder that though my family had access to the white man’s money, they were not white.6
Though I did not realize it consciously, my interest in fancy girls was a way to explore the particular pain of fair-skinned African Americans, including women like my great-grandmother.7 She and the former enslaved individuals presented here and elsewhere, the women in particular, seemed to stand in a very complicated space. Being of mixed race could lead to ostracism by other blacks if it was believed they received material or financial advantages as a result of their fair complexion. Such ostracism occurred even though many fair-skinned blacks were themselves poor.
African Americans, even those in the scholarly community, often do not want to acknowledge, much less confront, such intragroup conflict.8 Certainly the peculiar institution generated tensions even between people in the African American community. Such tensions manifests in “color struck” attitudes of those who buy into white Americans’ normative standards of beauty and the accompanying myths that fair-skinned people are not only more beautiful (and have “better” hair) but are smarter, too. It was not simply the privileged access that some blacks had to powerful white men that other blacks found off-putting: it was the way in which such pairings, coerced or not, produced fair-skinned individuals who reinforced the caste system. Illustrative of this attitude is the disposition of Cora Gillam, a freedwoman from Arkansas. Gillam’s half sister was sent to Ohio to attend Oberlin and her half brother to a school in Cincinnati. Gillam, whose white father was an overseer and presumably of more modest means than the fathers of her half siblings, attended neither. But as if to align herself with them in any manner she knew how, Gillam reportedly announced, “My father was not a slave. Can’t you tell by me that he was white?”9
It should be emphasized that some black men certainly resented African American women’s involvement with white men, some even blaming them for inviting white men’s overtures. Robert Smalls, a former slave from South Carolina who became a boat pilot for the U.S. government and later a congressman, reportedly once charged that African American women were inherently immoral. He bemoaned that they seemed to want “principally” “white men with whom they would rather have intercourse than with their own color.”10 At least one woman of color similarly took a jaundiced view of women of color engaging in interracial sex, whether they had a choice or not. Said Sarah Fitzpatrick, a former bondswoman interviewed in Alabama, “The reason our race is so mixed up,” she stated, “is by fooling with these white men.”11 All this as white Americans often do not want to be confronted with how the oppressive ideas of their ancestors were foundational to such conflict in the first place. Each act of favor black women and children received from white men seems to point to the hypocrisy in American life. Certainly after losing the Civil War, as a part of reconciling itself to the nation, white Southerners reasserted control over their region by means of Jim Crow laws and lynchings but also storytelling about black-white relationships. Annette Gordon-Reed argues that such rewriting was a necessary part of their healing process.12 The imposition of white domination required that whites remain silent about “ubiquitous” interracial sex and illegitimate children with women of color.
Nowadays such conversations are especially difficult in this so-called postracial world.13 With so much unspoken, it is hard to see the many little-discussed costs some fair-skinned African American women paid while sometimes accruing a bit of power from their intimate encounters with white men. Certainly my great-grandmother paid such costs while maintaining her ties to Mister Ray. It may have been worth it for she was driven by poverty. The gains were not just dresses and other material items, but really important items like food. Mister Ray provided things that she and other family members needed, even as the segregationist practices around them never permitted him fully to flaunt his African American woman. Their relationship occurred in an environment in which interracial unions were acknowledged at the community level, as they had been since the colonial era, but they were never sanctioned either socially or legally owing to the rules that elite white men themselves imposed on women, people of color, children, and poor whites.14
I believe that historical actors’ ways of speaking and understanding their lives and tactics shift across time and must be contextualized and delineated according to the norms of the moment, but I also believe that in some ways my great-grandmother’s life had parallels with the experiences of untold numbers of other women, even former enslaved women, with whom white men developed some form of intimacy. I carried on with my research, knowing the experiences of such women and their children are an understandably taboo subject for those who have conflicted feelings about the era of slavery.
The women to whom Smalls pointed forever carried with them traces of their particular past, isolated not only by whites, but other African Americans. Frederick Law Olmsted, the renowned landscape architect who traveled throughout the South to document the evils of slavery, once saw a coffle containing a woman who did not wear “the usual plantation apparel” worn by other slaves. This woman, he observed, also took “no part in the light chat” of the others. He noted, too, that she did not help them make a fire, nor did she later stand with them around it. Instead, she stood alone, like a statue, “bowed and gazing” into the flames.15 This bondswoman’s past is unknowable, but from Olmsted’s description she appears to have been a female slave in whom whites, not just white men, had invested more time and interest than they had in others.
A growing number of scholars have urged those in the academic community to take seriously the African American oral history tradition, which “has long realized the significance of white men as fathers of mixed-race enslaved children providing a foundation for a pre-Civil War black elite even though mainstream scholars are unwilling to do the same.”16 Again, that such African Americans have remained members of the black elite may have had some impact on how their lives are presented in academic writings. The black elite, some may have decided, needs little help.
To be sure, the favor some women of color had was manifested in the New World. Just months before the start of the Civil War, Mary Seacole, a Jamaican woman of mixed race, attempted to help a young man enlist in the Royal Navy. Having had much contact with British society because of her services as a nurse during the Crimean War, she sought an appointment with Alexander Milne, a senior naval officer, upon his arrival in Port Royal, Jamaica. What makes their meeting especially meaningful is that Milne took the time to record this event in a gossipy letter to one of his colleagues. “Mrs Seacole is on board in high Crinoline,” he wrote. “She wants me to Enter a Boy but he is all head and no body.”17
As did others of his day, Milne objectified Seacole. He noted her crinoline dress, which was standard attire for middle- and upper-class white British women. He probably did so because the dress was at odds with the naval uniforms on the men around him and possibly because it went against the grain of what was expected of a woman of color. In noting her attire, Milne was suggesting that she was no ordinary “Negro.” She had a respectable occupation. She was of mixed race. She had good breeding. These three facts were intricately connected. Because she was all of these things, she had Milne’s attention, even being allowed on his quarterdeck or into his office, both of them male, professional spaces.18
This meeting, one about which we have a few details, makes visible a level of intimacy between two people of very different social statuses in a private space. Although Milne refused to recruit the young boy for whom Seacole had lobbied, he granted her an audience, something both he and she understood as being a reasonable gesture despite his racial identity given her privileged position in society and her service. His failure to comply with her request was almost certainly based on sound professional reasons, not on racial or gender prejudices.
It has been far easier to explore the blatant oppression experienced by people of African descent than such intimacy, especially the kind that concerns fair-skinned women and girls who, in imagined works and surviving records, often appear larger than life. While some were people of real significance to white men, others were bodies exploited by lonely and sometimes vile individuals. Getting at both extremes and all of the grayer areas in the middle requires investigation. It is through such scrutiny that we can probe how black women emerged as both victors and victims, immoral and upright, enslaved and indeed free with white men’s help. As if addressing the hurdles of such a task, a white woman, this one from the upper South, once told me, “It’s easier for both blacks and whites to say it was only rape, because to say it was love or something approaching that is too difficult. We’d then have to explain how this happened alongside of everything else.”19