Pay Attention to Seeds in Your Cassandra Ring

At work, we've got a some 0.8.5 cassandra clusters running stably for nearly a year now. This is fairly behind the stable cassandra release, at this point, but we got things to a stable point and didn't want to mess with it further, so 0.8.5 it is.

So, a node died. A volume went dead, and that was that. Fortunately, the documentation on how to handle node failure are pretty clear, particularly when using the docs from DataStax.

We went for option two, bringing up a new node at the adjacent token. Unfortunately, the new node saw the dead node it was meant to replace as up, and thus in bootstrapping attempted to stream data from the dead node. Shockingly, that doesn't work.

Why did the new node thing the dead node was up? Every other node in the ring saw the dead node as down.

My suspicion, not truly confirmed by a proper review of the code but conveniently strengthened by observed behavior, is because of seeds.

When we built our cassandra clusters, we used chef. And not just chef, but "cluster chef". As the link shows, there are several generations of cluster chef. When we built our clusters, version 2 was under development, so we based things on version one (if memory serves). That's so old and decrepit that they don't even list it.

Unfortunately, while the version made rolling out a new cluster relatively easy, it was not so great about maintaining existing clusters, and it made some unwise choices when selecting seeds for your cassandra ring. Specifically:

• Every node known to cluster chef was used as an entry in the seed list
• Including each node's own IP

(Note: it is entirely possible that subsequent releases of cluster chef make better choices. Of course, it's similarly possible they do not.)

(Note further: I was aware of this stupid choice, but frankly, it was working, and development efforts are focused on totally separate parts of the system. So I left it alone, noting but not comprehending the magnitude of this folly. It is thus not unreasonable to conclude that somebody should shoot me in the face).

Populating the seed list this way made cluster maintenance a real nuisance, because a new node would always come up thinking of itself as a seed, and seeds won't auto-bootstrap. So we would carefully pick out initial tokens, make sure the node attribute's were set to bring it up with auto_bootstrap, and of course, it wouldn't, and we'd be stuck scrambling to repair this node that joined the ring without ever receiving any of its range data. Our reliance on QUORUM operations saved us in this case; a sub-QUORUM operation could have lead to fairly unpleasant levels of inconsistency (which for us would've been unacceptable but of course varies according to use).

Anyway, about seeds. Take a look at the gossip architecture. Note that seeds are special.

I read up more on seeds and general practices within the community, and found that folks in the know seemed to prefer a smaller, actively managed seed list. They kept it consistent across the cluster (or at least across a "data center"), and kept it up to date.

At this point, I:

• Stopped Cassandra on the new node that was stuck in its never-ending-'cause-it's-never-starting streaming state
• Picked two of the servers with marginally lighter load as seeds, and updated the configuration for all other nodes in the ring. (The dead node was explicitly excluded from the possible seeds, naturally.)
• Performed rolling restart of all other nodes in the ring
• Verified that all the nodes listed the old dead node as, well, dead (down).
• Updated the seed configuration to be consistent on the new node
• Brought up the new node, and verified that it saw the dead node as down, and that it was streaming data from one of the live neighbors instead of the dead one.
• Cheered. Hurray!

Some lessons I take from this, that I'll at some point dig into the Cassandra source to verify/understand:

• Don't accept the choices of a tool you don't know well; pick the seeds yourself
• Actively manage the seeds so they are known to you and reflect a reasoned choice.
• If a seed dies, get it out of the seed list and propagate that change throughout the ring before you try to replace the dead node.

Wonderful North Carolina

For the sake of convenience, I've gathered all my offensive comments about North Carolina in one place. Enjoy!

1. This state apparently doesn't learn from its history on the ills of denying the humanity of an entire group of people.

2. Why are "states' rights" used again and again to trump "individuals' rights"?

3. Perhaps the real reason NCers disapprove of gay marriage is the ambiguity; without traditional gender roles, it's impossible to determine which party owns the other.

4. In a country with relatively poor voter turnout, it's good to know people will haul their asses to the polls for the opportunity to express their bigotry.

Enabling block compression in Hive's sequence files

If you're like me, you like Hive, and you like storing some of your data in sequence files. You also like compressing your data, so that your data is snappy and delicious.

If you're like me, you may have taken these recommendations from the Hive wiki for enabling block compression in your sequence files.

Alas, we found that block compression wasn't actually happening. In looking at the Hadoop 0.20.203.0 source, the logic associated with the "io.seqfile.compressiong.type" setting is marked as deprecated.

We found it necessary to use the newer "mapred.output.compression.type" setting instead.

Program Notes for Pinkham's Wedding Cantata: Draft the First

This is the first draft. I expect to need to prune it down. Comments welcome.

Daniel Pinkham's Wedding Cantata, published in 1956 and scored for mixed chorus and piano, celebrates the joy and profundity of love as expressed through biblical texts drawn from the Song of Solomon. By turns exuberant and poignant, the long, leaping lines and sparkling harmonies magnify the spirit of the texts, while clever formal construction and common musical material meld the four movements into a unified whole. Pinkham accomplishes this with his characteristic brevity.

The cantata opens with Solomon verses 2:10-12 ("Rise up, my love, my fair one..."), set to a joyous, dancing 6/4 rhythm. The piano accompaniment features bright, closely voiced harmonies, generally arranged with added seventh or ninth tones; the harmonies sound familiar, but shine with a sparkling, mystical character that pervades much of the piece. The chorus' opening unison melody, in rising by a fourth, then fifth (appropriately given the text), states a powerful melodic theme that appears in various forms throughout the first and third movements. The second section of the opening movement builds upon the earlier melodic material, this time employing mixed meter, to Solomon verses 6:1-3 ("whither is my beloved gone..."). The text continues the common themes of joy in the beloved, with pastoral imagery.

The musical and textual themes of the first movement return in the third movement, to verse 4:16 (“Awake, O north wind...”); while the third movement features canon and generally simpler accompaniment, it shares an unmistakable similarity in melodic shape and rhythmic character, and opens its themes with the same ascending intervals of fourth and fifth. The conclusion of the third movement brings a dramatic resolution to this set of themes, which the first movement denies us.

In contrast, movements two and four draw from verses 8:7 and 8:6, respectively, and meditate on the profundity of love. Both feature thicker choral textures, with a heightened use of counterpoint, and intense legato over steady duple rhythms. Both employ dissonance in a somewhat more traditional way, though Pinkham refrains from the full resolutions we might expect (of particular interest is the ringing dissonance used in movement two between tenor and bass, shortly before the soprano entrance). The homorhythmic chorale setting of number four (“Set me as a seal...”), and the plaintive insistence of number two’s canon theme (“Many waters...”), paint their shared subject with passionate intensity.

While the pairings of odd and even movements stand out, other aspects unify the piece as a whole. The inner movements both employ canonic singing, providing a sense of overall cohesion. In contrast, the outer movements both contain extended sections of homorhythmic chorale singing, and share some striking harmonic passages; the harmonic progression that closes the whole piece first appears in the conclusion of the first section of the first movement. Most strikingly, and appropriately, the celebration of love unifies the whole.

Large Scale Symmetry in Pinkham's Wedding Cantata

The movements can be paired in various ways, giving a general symmetry overall and a sense of formal and stylistic cohesiveness.

The outer movements can be paired for a couple reasons:

• Extended sections of either unison or chorale textures (in the chorus)
• Common harmonic material (as noted earlier, mm. 28-31 of movement one has same harmonic motion as mm. 15-22 of movement four, which concludes the piece); it's a fairly striking harmonic passage.

The inner movements can be paired in that they're both feature canonic singing.

The odd movements are a more obvious pairing in that they have similar themes/imagery in text, some common melodic material, dancing compound meter, syncopation.

The even movements are additionally an obvious pairing in their focus on love's profundity, in their thicker choral textures, in their meditative or even plaintive qualities rather than jubilation. Use of dissonance between voices is somewhat more traditional in these movements, though there are of course occasions in which traditional resolution does not take place.

Much of the piece uses a common harmonic language, with traditional tertian harmonies expanded with added sevenths, sixths, ninths, etc. Such extended chords are often spelled in a way that adds a crystalline or spiky quality to the sound (as one often observes in Ravel, Satie, or even Stravinsky). In contrast, some harmonic passages utilize standard triads, but moving in unconventional ways; motion by thirds between major triads (requiring chromatic movement in one voice) is especially prominent.

ETHAN: The use of canon in mvt. 2 helps to paint the text. "Many waters cannot quench love." The text conveys an idea of a feverish, burning love that cannot be tamed, a fervent passion that cannot be calmed. The music reflects that idea by cycling through the main melody in canon, each voice pursuing the others, not stopping for rest, endlessly alight with motion. The final, forte unison D at the end represents the coming together of hearts and souls into unity.

Text sources for Pinkham's Wedding Cantata.

I. Rise up, my love

King James Bible, Cambridge Ed.; Song of Solomon 2:10-2:12

Rise up, my love, my fair one, and come away.

For lo, the winter is past, the rain is over and gone.

The flowers appear on the eart; the time of the singing of birds is come, and the voice of the turtle is heard in our land.

Song of Solomon 6:1-6:3

Whither is thy beloved gone, O thou fairest among women? whither is thy beloved turned aside? That we may seek him with thee.

My beloved is gone down into his garden, to the beds of spices, to feed in the gardens, and to gather lillies.

I am my beloved's, and my beloved is mine; he feedeth among the lillies.

II. Many waters

Song of Solomon 8:7

Many waters cannot quench love; neither can the floods drown it: if a man would give all the substance of his house for love, it would utterly be contemned.

III. Awake, O north wind

Song of Solomon 4:16

Awake, O north wind; and come, thou south; blow upon my garden; that the spices thereof may flow out. Let my beloved come into his garden, and eat his pleasant fruit.

IV. Epilogue: Set me as a seal

Song of Solomon 8:6

Set me as a seal upon thine heart, as a seal upon thine arm: for love is strong as death; jealousy is cruel as the grave: the coals thereof are coals of fire, which hath a most vehement flame.

Observations

Solomon 2:10-13, and 8:6-7, appear to be pretty popular biblical readings for weddings. Solomon 4:16 may be less common, but has obvious symmetry with the themes of 2:10-2:13.

The music appropriately focuses on the differing characters of these sections; the wind/spice-related settings of movements one and three share a dancing, jubilant rhythmic character, and seem to revel in the joy of love and the wedding occasion. In contrast, the text of 8:6-7 focuses on the profundity of love, and as such the settings of movements two and four reflect that profundity, using thicker choral textures, with lines suggesting reverence, in contrast to the dancing joy of the others.

Prep notes on Pinkham's Wedding Cantata

First obvious formal thing: the clear relationship between first and third movements. Both have a primary theme in a fast 6/8, with similar piano accompaniment at opening (repeated chord patterns with prominent parallel fourths). Both themes open with the same melodic figure (both in terms of intervals and rhythm). The melodies differ in their particulars but have a similar overall shape. And the text are clearly related, with the first having the male wondering what his beloved's up to in the garden, and the third having the female asking the wind to blow the spice fragrance to the man that he'll be inclined to come to the garden and eat his pleasant fruit.

What husband doesn't enjoy him some pleasant fruit?

The harmonic sequence at the end of the first section of the first movement is the same as the final of the piece.

Published with Blogger-droid v2.0.4