Posts Tagged MongoDB
Beware of the dreaded passive arbiter!
The setup seemed simple enough: Two application servers running ASP.NET MVC, and a MongoDB replica set with two standard nodes and an arbiter.
If you’re familiar with MongoDB replica sets, feel free to skip the following paragraph:
When using MongoDB in a productive environment, two servers and an arbiter is much of the absolute minimum requirement. You cannot seriously run on a single server; MongoDB is not designed to provide good durability on just one machine. A two-node replica set provides replication of changes to a second machine. It also provides a failover server should one server go down. A replica set works much like classic master-slave replication, but the master is elected automatically: if the node that currently holds the master role (the “primary”) does not respond to a heartbeat, another node (a “secondary”) becomes the new primary. Which secondary is promoted is quite an interesting ceremony. An important aspect is that for a primary to be elected, it needs to “see” the majority of nodes in the replica set – otherwise, a broken network link could split the replica set in the middle, and there’d be two primarys (see a popular piece of fantasy literature on what happens in such a situation). A result of these rules is that a replica set with just two nodes cannot work – when one server goes down, the other won’t see a majority; even when the secondary node fails, the primary demotes itself because it finds itself alone.
This is where an arbiter comes in. An arbiter is a MongoDB instance that does not hald any data. It does not take part in replication. Its sole purpose is to provide a majority for the election of a new primary. In a two-node setup with an arbiter, one server can fail and the other still has a majority of two thirds of the replica set.
And then there’s priority. Priority is a MongoDB server setting that determines the order in which nodes are elected primary – a node with a higher priority wins. The most useful setting for priority however is 0. Setting a node’s priority to 0 makes it a “passive” node – it will never become elected the primary. This is useful if a machine is e.g. used just for backups, or if it’s in a remote data center. Otherwise, a passive node acts like a normal member of the replica set – data is replicated to it, and it can be used for running queries against it.
Alright. So we’ve got passive nodes, and arbiters. This is mutual exclusive – an arbiter cannot be a passive node. Passive nodes can be used for queries, arbiters can’t, they don’t hold data.
To query the status of a node in a replica set, you can run the command db.isMaster() against it. It returns a document that describes the shape of the replica set, and the role the queried server plays within it. Two fields that are returned are if the node is a passive, and if it’s an arbiter.
The C# driver for MongoDB (and probably other drivers as well) use the isMaster command to determine which servers are available, which is the primary, and qhich ones can be used for queries. If a query is allowed to be executed against a read-only replica of the database by setting the slaveOk setting, the driver looks for secondary and passive nodes. Which is correct. But this is where things start to go wrong – using this sequence of in itself harmless details.
MongoDB up to and including the current version 1.81 allow setting the priority on an arbiter node.
If you set the priority to 0, a node will report itself to be a passive. Yes, this includes arbiters. Ouch.
The driver looks for secondaries and passives to run queries. It does not exclude arbiters, because these normally aren’t secondaries or passives. Except when the server claims to be one, because someone set the priority to 0. Second ouch.
If you run a query against an arbiter, it will return an error that it’s neither primary nor secondary, and cannot be read from: { "$err" : "not master or secondary, can't read", "code" : 13436 } .Luckily, the driver recognizes the connection as faulty and removes it from the connection pool.
So what kept me busy for about ten working hours was this: an application server was reporting a MongoDB error once every time the IIS application pool had restarted. It turned out it was connecting to an arbiter, and the reason for that was that a DBA had set the priority on the sorry arbiter to 0, which was a leftover of its configuration before it became an arbiter.
First conclusion: don’t set the arbiter’s priority to 0. Just don’t. Arbiters don’t need a priority setting, or in fact, mostly no settings at all.
Second conclusion: I really love MongoDB for its simplicity, the data modelling possibilities, and its performance. But things like the dreaded passive arbiter really raise a question mark on its maturity – the problem would have been easily avoidable on either the server or the driver (or both, for better). What also needs to be admitted that 10gen is extremely responsive and helpful. They promised to fix the C# driver within a week(!) in release 1.2, and scheduled a fix for the server for version 1.92 – all within one day reaction time.
MongoDB: when documents aren’t perfect, cope with it
MongoDB is a NoSQL database that organizes data as JSON documents in collections instead as rows in tables. One major difference between a document in a collection and a row in a table is that while all rows in a table share the same schema, each document in a collection can be completely different. With all the huge advantages this approach brings, there is a friction when strongly typed domain objects are deserialized from a collection. A document needs to match the shape of the object that it should be serialized to.
The “official” MongoDB driver for .NET – that is the one offered and supported by 10Gen, the company behind MongoDB – offers a serializer that can be told quite exactly how tolerant it needs to be with the deserialization of JSON documents in to objects:
- When a JSON document defines extra elements that are not present in the class that it is deserialized into, the serializer can either ignore them, stash them into a “catch-all” property, or throw an exception.
- When properties on the class are missing in the JSON document, the serializer can set the property to null, it can set a default value, or it can throw an exception.
These options are well described in the official documentation.
Handling deserialization errors in collections
The Mongo C# driver supports operations to retrieve single elements from the database as well as collections. As we’ve seen, there is a lot of flexibility in defining a level of tolerance that matches your application’s requirements for the deserialization of single objects, the policy for collections is very simple: if any element fails, the whole collection is not loaded. While this is the right behaviour for scenarios that rely on data integrity, applications that focus on availability are threatened by complete loss of functionality when just a single document in a collection does not match the deserialization requirements.
- A scenario that relies on data integrity is a form in a line of business application that edits a list of closely related data like line items of an invoice. The line items all need to provide the same set of information, otherwise the invoice cannot reliably be calculated.
- A scenario that focuses on availability is a search in a product catalogue in an online shop. If a single product is not correctly formatted in the database, it’s better to just not show that single product than letting the whole search page become unavailable and basically close the shop until the ill-formatted product is identified and corrected.
In the case of the search page, a better behaviour would be to exclude any document that cannot be serialized, and provide a callback that identifies the problematic documents. The approach I’ve taken is to copy the default implementation of the collection serializer, and augment the bit that iterates over the elements of the collection with an event handler that keeps the iteration running:
while (bsonReader.ReadBsonType() != BsonType.EndOfDocument)
{
var elementType = discriminatorConvention.GetActualType(bsonReader, typeof(T));
var serializer = BsonSerializer.LookupSerializer(elementType);
T element;
try
{
element = (T)serializer.Deserialize(bsonReader, typeof(T), elementType, null);
list.Add(element);
}
catch (FileFormatException exc)
{
// Pass the exception to the provided callback
if (HandleDeserializationError != null)
{
HandleDeserializationError(exc);
}
// Move the cursor to the next element after the faulted one
while (bsonReader.State != BsonReaderState.EndOfDocument)
{
if (bsonReader.State == BsonReaderState.Value) bsonReader.SkipValue();
if (bsonReader.State == BsonReaderState.Type) bsonReader.ReadBsonType();
}
bsonReader.ReadEndDocument();
}
}
bsonReader.ReadEndArray();
You can download the full source from Bitbucket. Please note that it’s not production quality – it has not been used in production, but that could change over the next few weeks, and then it’s going to get an update with the real thing.
Authors
-
