Pay Attention to Seeds in Your Cassandra Ring

At work, we've got a some 0.8.5 cassandra clusters running stably for nearly a year now. This is fairly behind the stable cassandra release, at this point, but we got things to a stable point and didn't want to mess with it further, so 0.8.5 it is.

So, a node died. A volume went dead, and that was that. Fortunately, the documentation on how to handle node failure are pretty clear, particularly when using the docs from DataStax.

We went for option two, bringing up a new node at the adjacent token. Unfortunately, the new node saw the dead node it was meant to replace as up, and thus in bootstrapping attempted to stream data from the dead node. Shockingly, that doesn't work.

Why did the new node thing the dead node was up? Every other node in the ring saw the dead node as down.

My suspicion, not truly confirmed by a proper review of the code but conveniently strengthened by observed behavior, is because of seeds.

When we built our cassandra clusters, we used chef. And not just chef, but "cluster chef". As the link shows, there are several generations of cluster chef. When we built our clusters, version 2 was under development, so we based things on version one (if memory serves). That's so old and decrepit that they don't even list it.

Unfortunately, while the version made rolling out a new cluster relatively easy, it was not so great about maintaining existing clusters, and it made some unwise choices when selecting seeds for your cassandra ring. Specifically:

• Every node known to cluster chef was used as an entry in the seed list
• Including each node's own IP

(Note: it is entirely possible that subsequent releases of cluster chef make better choices. Of course, it's similarly possible they do not.)

(Note further: I was aware of this stupid choice, but frankly, it was working, and development efforts are focused on totally separate parts of the system. So I left it alone, noting but not comprehending the magnitude of this folly. It is thus not unreasonable to conclude that somebody should shoot me in the face).

Populating the seed list this way made cluster maintenance a real nuisance, because a new node would always come up thinking of itself as a seed, and seeds won't auto-bootstrap. So we would carefully pick out initial tokens, make sure the node attribute's were set to bring it up with auto_bootstrap, and of course, it wouldn't, and we'd be stuck scrambling to repair this node that joined the ring without ever receiving any of its range data. Our reliance on QUORUM operations saved us in this case; a sub-QUORUM operation could have lead to fairly unpleasant levels of inconsistency (which for us would've been unacceptable but of course varies according to use).

Anyway, about seeds. Take a look at the gossip architecture. Note that seeds are special.

I read up more on seeds and general practices within the community, and found that folks in the know seemed to prefer a smaller, actively managed seed list. They kept it consistent across the cluster (or at least across a "data center"), and kept it up to date.

At this point, I:

• Stopped Cassandra on the new node that was stuck in its never-ending-'cause-it's-never-starting streaming state
• Picked two of the servers with marginally lighter load as seeds, and updated the configuration for all other nodes in the ring. (The dead node was explicitly excluded from the possible seeds, naturally.)
• Performed rolling restart of all other nodes in the ring
• Verified that all the nodes listed the old dead node as, well, dead (down).
• Updated the seed configuration to be consistent on the new node
• Brought up the new node, and verified that it saw the dead node as down, and that it was streaming data from one of the live neighbors instead of the dead one.
• Cheered. Hurray!

Some lessons I take from this, that I'll at some point dig into the Cassandra source to verify/understand:

• Don't accept the choices of a tool you don't know well; pick the seeds yourself
• Actively manage the seeds so they are known to you and reflect a reasoned choice.
• If a seed dies, get it out of the seed list and propagate that change throughout the ring before you try to replace the dead node.

Wonderful North Carolina

For the sake of convenience, I've gathered all my offensive comments about North Carolina in one place. Enjoy!

1. This state apparently doesn't learn from its history on the ills of denying the humanity of an entire group of people.

2. Why are "states' rights" used again and again to trump "individuals' rights"?

3. Perhaps the real reason NCers disapprove of gay marriage is the ambiguity; without traditional gender roles, it's impossible to determine which party owns the other.

4. In a country with relatively poor voter turnout, it's good to know people will haul their asses to the polls for the opportunity to express their bigotry.

Enabling block compression in Hive's sequence files

If you're like me, you like Hive, and you like storing some of your data in sequence files. You also like compressing your data, so that your data is snappy and delicious.

If you're like me, you may have taken these recommendations from the Hive wiki for enabling block compression in your sequence files.

Alas, we found that block compression wasn't actually happening. In looking at the Hadoop 0.20.203.0 source, the logic associated with the "io.seqfile.compressiong.type" setting is marked as deprecated.

We found it necessary to use the newer "mapred.output.compression.type" setting instead.

Program Notes for Pinkham's Wedding Cantata: Draft the First

This is the first draft. I expect to need to prune it down. Comments welcome.

Daniel Pinkham's Wedding Cantata, published in 1956 and scored for mixed chorus and piano, celebrates the joy and profundity of love as expressed through biblical texts drawn from the Song of Solomon. By turns exuberant and poignant, the long, leaping lines and sparkling harmonies magnify the spirit of the texts, while clever formal construction and common musical material meld the four movements into a unified whole. Pinkham accomplishes this with his characteristic brevity.

The cantata opens with Solomon verses 2:10-12 ("Rise up, my love, my fair one..."), set to a joyous, dancing 6/4 rhythm. The piano accompaniment features bright, closely voiced harmonies, generally arranged with added seventh or ninth tones; the harmonies sound familiar, but shine with a sparkling, mystical character that pervades much of the piece. The chorus' opening unison melody, in rising by a fourth, then fifth (appropriately given the text), states a powerful melodic theme that appears in various forms throughout the first and third movements. The second section of the opening movement builds upon the earlier melodic material, this time employing mixed meter, to Solomon verses 6:1-3 ("whither is my beloved gone..."). The text continues the common themes of joy in the beloved, with pastoral imagery.

The musical and textual themes of the first movement return in the third movement, to verse 4:16 (“Awake, O north wind...”); while the third movement features canon and generally simpler accompaniment, it shares an unmistakable similarity in melodic shape and rhythmic character, and opens its themes with the same ascending intervals of fourth and fifth. The conclusion of the third movement brings a dramatic resolution to this set of themes, which the first movement denies us.

In contrast, movements two and four draw from verses 8:7 and 8:6, respectively, and meditate on the profundity of love. Both feature thicker choral textures, with a heightened use of counterpoint, and intense legato over steady duple rhythms. Both employ dissonance in a somewhat more traditional way, though Pinkham refrains from the full resolutions we might expect (of particular interest is the ringing dissonance used in movement two between tenor and bass, shortly before the soprano entrance). The homorhythmic chorale setting of number four (“Set me as a seal...”), and the plaintive insistence of number two’s canon theme (“Many waters...”), paint their shared subject with passionate intensity.

While the pairings of odd and even movements stand out, other aspects unify the piece as a whole. The inner movements both employ canonic singing, providing a sense of overall cohesion. In contrast, the outer movements both contain extended sections of homorhythmic chorale singing, and share some striking harmonic passages; the harmonic progression that closes the whole piece first appears in the conclusion of the first section of the first movement. Most strikingly, and appropriately, the celebration of love unifies the whole.

Large Scale Symmetry in Pinkham's Wedding Cantata

The movements can be paired in various ways, giving a general symmetry overall and a sense of formal and stylistic cohesiveness.

The outer movements can be paired for a couple reasons:

• Extended sections of either unison or chorale textures (in the chorus)
• Common harmonic material (as noted earlier, mm. 28-31 of movement one has same harmonic motion as mm. 15-22 of movement four, which concludes the piece); it's a fairly striking harmonic passage.

The inner movements can be paired in that they're both feature canonic singing.

The odd movements are a more obvious pairing in that they have similar themes/imagery in text, some common melodic material, dancing compound meter, syncopation.

The even movements are additionally an obvious pairing in their focus on love's profundity, in their thicker choral textures, in their meditative or even plaintive qualities rather than jubilation. Use of dissonance between voices is somewhat more traditional in these movements, though there are of course occasions in which traditional resolution does not take place.

Much of the piece uses a common harmonic language, with traditional tertian harmonies expanded with added sevenths, sixths, ninths, etc. Such extended chords are often spelled in a way that adds a crystalline or spiky quality to the sound (as one often observes in Ravel, Satie, or even Stravinsky). In contrast, some harmonic passages utilize standard triads, but moving in unconventional ways; motion by thirds between major triads (requiring chromatic movement in one voice) is especially prominent.

ETHAN: The use of canon in mvt. 2 helps to paint the text. "Many waters cannot quench love." The text conveys an idea of a feverish, burning love that cannot be tamed, a fervent passion that cannot be calmed. The music reflects that idea by cycling through the main melody in canon, each voice pursuing the others, not stopping for rest, endlessly alight with motion. The final, forte unison D at the end represents the coming together of hearts and souls into unity.

Text sources for Pinkham's Wedding Cantata.

I. Rise up, my love

King James Bible, Cambridge Ed.; Song of Solomon 2:10-2:12

Rise up, my love, my fair one, and come away.

For lo, the winter is past, the rain is over and gone.

The flowers appear on the eart; the time of the singing of birds is come, and the voice of the turtle is heard in our land.

Song of Solomon 6:1-6:3

Whither is thy beloved gone, O thou fairest among women? whither is thy beloved turned aside? That we may seek him with thee.

My beloved is gone down into his garden, to the beds of spices, to feed in the gardens, and to gather lillies.

I am my beloved's, and my beloved is mine; he feedeth among the lillies.

II. Many waters

Song of Solomon 8:7

Many waters cannot quench love; neither can the floods drown it: if a man would give all the substance of his house for love, it would utterly be contemned.

III. Awake, O north wind

Song of Solomon 4:16

Awake, O north wind; and come, thou south; blow upon my garden; that the spices thereof may flow out. Let my beloved come into his garden, and eat his pleasant fruit.

IV. Epilogue: Set me as a seal

Song of Solomon 8:6

Set me as a seal upon thine heart, as a seal upon thine arm: for love is strong as death; jealousy is cruel as the grave: the coals thereof are coals of fire, which hath a most vehement flame.

Observations

Solomon 2:10-13, and 8:6-7, appear to be pretty popular biblical readings for weddings. Solomon 4:16 may be less common, but has obvious symmetry with the themes of 2:10-2:13.

The music appropriately focuses on the differing characters of these sections; the wind/spice-related settings of movements one and three share a dancing, jubilant rhythmic character, and seem to revel in the joy of love and the wedding occasion. In contrast, the text of 8:6-7 focuses on the profundity of love, and as such the settings of movements two and four reflect that profundity, using thicker choral textures, with lines suggesting reverence, in contrast to the dancing joy of the others.

Prep notes on Pinkham's Wedding Cantata

First obvious formal thing: the clear relationship between first and third movements. Both have a primary theme in a fast 6/8, with similar piano accompaniment at opening (repeated chord patterns with prominent parallel fourths). Both themes open with the same melodic figure (both in terms of intervals and rhythm). The melodies differ in their particulars but have a similar overall shape. And the text are clearly related, with the first having the male wondering what his beloved's up to in the garden, and the third having the female asking the wind to blow the spice fragrance to the man that he'll be inclined to come to the garden and eat his pleasant fruit.

What husband doesn't enjoy him some pleasant fruit?

The harmonic sequence at the end of the first section of the first movement is the same as the final of the piece.

Published with Blogger-droid v2.0.4

Regarding Difficulty

As mammals with linguistic tools handy for the structuring of thought, we are by nature inclined to analyze and categorize and tokenize, so that we might apply labels to things and speak about those things in a rough group consensus of shared meaning/experience.

Modern culture owes a great deal to this tendency, as it allows us to accumulate knowledge across generations, in volumes far surpassing the capacity of an oral transmission medium.

It seems when we're learning, however, that the tendency may get in the way. Consider the case of the word "hard":

"I am learning this new thing, and it is hard."

People who dabble in creative activities, particularly temporal arts like music or dance, would find this statement familiar. Certainly as a software engineer the statement seems like an archetypal expression of experiences encountered every few months.

What does it actually mean? The word "hard" provides a rough label encompassing all sorts of situations.

• "I am working with a distributed database that employs eventual consistency to allow for improved availability and scalability; it is difficult to adapt my thinking to an eventually consistent model."
• "I am learning a new piece of music that freely mixes meter and does not use functional harmony; it is challenging to get the details right because they do not move in ways I can predict."
• "I am trying to understand how collateralized debt obligations wreaked such havoc in our financial system, and I can't make heads or tails of it because everything I read is in such financial insider speak."
• "I am learning a new piece of music that has a lot of melismatic passages and awkward phrasing; it's difficult to keep my tone consistent and to find spaces to breathe without disrupting the line."
• "I trying to make some configuration changes in my distributed software system and it's challenging because there are so many performance-impacting details to consider."

Of the list above, I think only the last two are truly aligned with our original statement ("I am learning this new thing, and it is hard"). These last two are not about familiarity with the problem space; they are about actual details of the problems themselves. The others are better summarized:

"I am learning this new thing and it falls outside my realm of experience."

Or better yet:

"I am learning this new thing and it is not yet familiar to me."

If a thing is outside one's experience, then one cannot authoritatively say whether that thing is intrinsically "hard" or not. Here's an accurate statement about what is hard:

"I am learning and it is hard."

Isn't that more to the point? In learning, we program ourselves. We force ourselves think differently (in the literal sense, as opposed to the Apple marketing sense). We tackle problems that we previously avoided, ignored, or were entirely ignorant of. We experience the world from a broader place. In short, we change ourselves, and that is very hard work indeed.

Take something you tackled some time in the past. A piece of music, for instance. When you first looked at it, perhaps you thought it looked hard. Perhaps those notes just wouldn't come to your ear. Perhaps you just couldn't get those rhythms. Whatever.

After you transcended the challenges of the piece, did it still feel hard to do? Did you even stop and consider it? Or did you integrate the piece into your experience, so it stopped being about hard versus easy and simply became about doing it well every time?

I cannot think of an instance in my musician life or engineer life in which classifying an unlearned thing as "hard" was useful, except as a means of identifying things that would require more effort than usual. For any given learning challenge, there's a pile of stuff you don't know well, but there's a pile of stuff you do know. In most situations, the pile of stuff you do know is larger than the other. Is it fair, then, to classify an unlearned thing as "hard" if in fact you are intimately familiar with the bulk of its parts?

It's more helpful, in my opinion, to think and speak accurately about the problem:

• "The technical challenges of this piece are difficult and require practice."
• "There are a lot of variables in this system and finding the optimal configuration will require careful benchmarking and analysis."
• "I am unfamiliar with the harmonic language of this piece and need to spend more time than usual getting the harmonies in my ear."
• "I am unfamiliar with the problem space of distributed databases and need to read some of the seminal literature on the subject to better understand the implications of this system's design."
• "I have trouble following the metric switches throughout this piece; I need to be better about counting and pay more attention to the word stresses."

That is, shed the vague notion of "hard", isolate the real problem(s), and identify concrete solutions to each. Then the notion of difficulty changes:

• I have not achieved familiarity with the harmonic language of this piece yet.
• I do not have the necessary background knowledge to work effectively with this system yet.
• I am not able to consistently execute these melismatic passages yet.

But I will. That's what's fun about challenges.

Thanksgiving Prayer

On that day, God created Carbon;

He created Carbon, and sprinkled it both within and without the face of the Earth;

and He created Gravity, because it is not Just a Theory, to hold the Carbon and the Oxygen and the gases of the Firmament about the Earth;

and He gave Man dominion over every living creature on the face of the Earth;

and He said unto Man: "Go forth and multiply, and create Markets, and let not ye regulate the Markets".

And Man created Markets, and Man incorporated, and called it Man, and Man gave Man dominion over every Carbon on the face of the Earth.

On that November the ninth of 2011, God saw the Markets, and the way of Man with the Carbon, and the Earth that looked like the Earth of September the fifteenth, and God said it was Good.

On Normalcy

My children are not normal. Not surprising; their parents aren't, either.

Normalcy never struck me as an especially desirable state for a human being, within a certain range of tolerance. The "normal" of standard decorum -- excrete in the allotted area, speak at an appropriate volume for your environment, wear pants to work, take turns speaking when engaged in conversation, etc. -- makes sense and provides a reasonable framework for productive discourse. But the "normal" of social interaction with one's peers, taking the foundational stuff as a given, is another thing entirely. That "normal" feels biased towards a relatively narrow range of interests and behaviors, none of which appear to offer any unique merit apart from their popularity. The former gives us rules for the mechanics of conversation, the latter for the content. I happen to think the mechanics are fine and the content basically sucks.

So I don't offer the assessment of my daughters' standing in this respect with woe or despair. More like amusement at best, and pride at worst.

We recently enjoyed a hurricane-filled weekend with my parents, elder brother, and his family. He observed during the visit that while his son sings occasionally throughout the day, my girls seem to sing pretty much continuously between the two of them.

And well they should: their parents both sing pretty seriously for pleasure and to a lesser degree work. Both work on so-called classical singing on a regular basis. Their mother sings kiddie tunes at various pre-school events. Their father sings damn near constantly, while working at the computer, driving to the grocery store, walking through the grocery store, cooking dinner, taking a shower, or excreting in the allotted area. For these girls, singing is normal.

Lots of parents, and lots of children, sing. The near ubiquity of it in our home, and the primary stylistic focus, are probably outside the norm, but gently so. A little eccentric, perhaps, but an eccentricity that people can appreciate.

Eccentricity: when we listen to "Music Together" albums in the car, my daughters' parents frequently improvise harmonies. Not all that unusual. Except that daddy improvises chromatic counterpoint, or turns every harmony into a seventh, ninth, or sharp eleventh.

Eccentricity: the father's endless singing often eschews identifiable "songs", instead playing out a part in some new piece in his head that will never be written down, never heard by anyone else. It might be the Coltranesque improvisatory horn line, the jazz/funk bass line to the latest James Brown noncreation, or a chant melody on nonsense syllables sounding two parts DuFay and one part Hindustani raga.

(The mother's loving acceptance of the father's oddities in this respect teach us the meaning of grace.)

For my daughters, singing is a constant companion, rather than a mere activity. You don't fit it into your schedule like gymnastics classes and play dates; you carry it with you everywhere you go. Sometimes you go to the grocery store.

Young children naturally emulate the adults around them, before learning the hipness of rejection. The ways in which the described parental behaviors manifest in the children are easy to see. But the degree to which this shifts the girls' sense of normalcy is only recently coming out.

The two year-old now sings around the house with a frequency rivaling that of the almost-five year-old. While she colors, looks at books, etc. The almost-five year-old breaks into song in the aisles of the grocery store, mid-conversation, after dinner at Stone Heath Pizza, whenever and wherever. She looks you in the eye, as if she regards this as a standard part of discourse.

Additionally, after asking her mother about her parents' disdain for Ariel's singing in "The Little Mermaid," and their appreciation of Aurora's singing in "Sleeping Beauty", the almost-five year-old will consciously shift between her regular voice and an affected nasal style. We didn't tell her that one is better than the other, and she continues to enjoy stuff that we pretty much despise. As she should. She simply integrated the information into her awareness and takes note of the different approaches as she hears them.

Every little kid has his or her developmental points of interest. I don't pretend that the girls' behavior is in some way exceptional (in the overloaded sense of exceptional that conflates "highly unusual" with "excellent"). Nor do I maintain lofty aspirations for them or a desire to push them towards a living in music. The degree to which music matters to me, and the place it occupies in my interior life, falls outside the range of normal. My only hope for the kids in this regard is that they might have something occupying a similar position in their lives. It gives tremendous satisfaction to see that they already do. We didn't need to teach them or give lessons. We needed only to let them see us being our own weird selves.

Published with Blogger-droid v1.7.4