Limit MongoDB memory use on Windows without Virtualization

http://captaincodeman.com/2011/02/27/limit-mongodb-memory-use-windows/

Limit MongoDB memory use on Windows without Virtualization

Posted on February
27, 2011



I’ve seen the question of how to control MongoDB’s memory usage on Windows come up several times and the stock answer always seemed to be “you can’t – it uses memory-mapped files and if you want to limit resources you need to use some form of virtualization
to do it (HyperV, VMWare, Virtuozzo etc…)”.



If you are using MongoDB on a dedicated server then you generally want it to use all the memory it can but if
you want to use it on a server shared with other processes (e.g. an IIS website using MongoDB for storage, maybe with SQL Server as well) then you will want
to put a cap on how much it uses to ensure memory is kept available for the other processes.

So is it possible if you are not on a virtualized environment? Yes (otherwise this would be a very short blog post!) and we’ll explore how …

The standard behaviour described above is actually a result of the default resource manager used by Windows but both Windows 2003 and Windows 2008 have a separate installable option called the “Windows System Resource Manager” (WSRM) that allows greater control
over the CPU and Memory available to a process.

First of all, lets look at what we’re trying to solve. Here we have a low-memory server (only 2Gb) running MongoDB on Windows 2008 R2 x64. There are a few databases of a few Gb each so the mongod.exe process quickly starts consuming as much memory as it can
(rightly so) to keep as much of it’s indexes in memory for the fast performance we know and love:









What we’d like to do is save some memory for other processes by limiting the mongod.exe process to 1Gb in this case (I know this is ridiculously low but the only thing that will change for you are the actual limits you want to use).

To do this we first need to install Windows System Resource Manager which on Windows 2008 is available under the Features section of the Server Manager.





Once that’s installed fire it up and you’ll see the default resource management policies. By default the standard Windows “memory is given to whoever shouts loudest” policy is used but other pre-configured alternatives are available. WSRM also provides a calendar
/ event system where the policy can be changed at certain times (a typical scenario is giving critical business apps priority during the day but then batch processes greater priority overnight). We’re not going to go into the calendar features here but it’s
interesting to know about.

Let’s create a new policy to control the resources that MongoDB can consume. To do this, right click on the “Resource Allocation Policies” container and chose “New Resource Allocation Policy …”. This will present us with the New Resource Allocation Policy dialog
below:





First of all, we need to add a new resource allocation entry so click the ‘Add…’ button and we get to another “Add or Edit Resource Allocation” dialog:





We don’t have a Process matching criteria for MongoDB yet so choose <New…> to get … yes, you guessed – another dialog, this time “New Process Matching Criteria”. We’ll call it “mongod_process” and click the Add… button to get another death-by-dialog to define
it.

There are a few ways to do this – if MongoDB is installed as a service then you can choose “Registered Service” in the drop-down, click “Select” and choose it from the list or you can select from a list of running processes or you can just enter the full path
and filename to mongod.exe. Here is the entry after selecting an installed MongoDB Windows Service:





After clicking OK we get back to the Process Matching Criteria dialog showing our new rule:





After clicking OK we’re now back at the Resource Allocation dialog with the new “mongod_process” Process matching criteria selected and can now decide what resources we want to allocate to the process. Lets limit the CPU to 50% (not that MongoDB seems to consume
much CPU):





The Memory tab allows us to limit the memory and here there are two options. The maximum committed memory limit is more to control apps that may have a memory leak and can be setup to stop or alert someone when the process goes above the configured limit. We
don’t want this one … instead we’ll set a maximum working set limit which will control how much memory is allocated to MongoDB. In this case, we’ll set the limit to 1Gb but the actual value to use will depend on your circumstances:





After clicking OK we should then be at the Resource Allocation Policy dialog with our process matching criteria, CPU and memory limits shown. We could include more limits in the policy but we’ll leave it as it is for now – any remaining resources will be allocated
to other processes as normal after the limits have been imposed.





The final piece is to make this policy active which is done by clicking on the “Selected Policy” link on the main ‘page’ or right-clicking on the new entry under the “Resource Allocation Policies” and choosing “Set as Managing Policy’”. You can also right-click
on the “Windows System Resource Manager (local)” entry and choose “Properties …” to display the dialog below which allows you to select the Current resource allocation policy:





So, we’ve created a new policy that has a criteria to match the mongod.exe process which will limit the CPU usage to 50% and memory to 1Gb … does it work? Here’s the result after it’s enabled showing the memory used immediately dropping:





… and the MongoDB / mongod.exe process using the 1Gb limit we specified (1Gb = 1024Mb = 1,048,576Kb).





So, we’ve successfully limited the CPU and memory that MongoDB can consume without havign to resort to any form of server-virtualization and while MongoDB will probably not run as fast as it did when it had free-reign to consume as much as it wanted (or rather,
when the default windows resource manager gave it what it asked for) we will probably have a faster overall system as our other processes are allocated the memory and CPU that they need for a better balanced system.

Please let me know what you think of the above technique and if you find it useful.

About
these ads

Share this:

Twitter5

Facebook

Like this:

This entry was posted in Uncategorized and
tagged mongodb, windows byCaptain
Codeman. Bookmark the permalink.

19 THOUGHTS ON “LIMIT MONGODB MEMORY USE ON WINDOWS WITHOUT VIRTUALIZATION”

Chris
Fulstow on February
28, 2011 at 6:47 pm said:

Great post Simon! I’ve been wondering if this was possible for a while, but never got around to trying it out. It’s really useful to now have the option of running MongoDB on the same machine as IIS and other apps, instead of needing to give it an entire box
or VM to itself.

Reply ↓


aaron on March
1, 2011 at 6:06 pm said:

thanks a lot, this is excellent to know

Reply ↓


chris on March
2, 2011 at 4:55 am said:

It would be great if there would be a tutorial for Linux as well


Reply ↓

Pingback: ehcache.net


Ed
Rooth on April
18, 2011 at 1:34 am said:

Awesome! Exactly what I was looking for. I was surprised nobody in the mongo forums or anyone from 10gen were able to answer this.

Reply ↓


Ash on April
18, 2011 at 1:40 am said:

Life saver! Due to resource constraints we’ve got MongoDB running alongside SQL Server and a few other things, and MongoDB eats all the pies even if it’s not hungry and there’s no pie left for anyone else. It’s DB gluttony.

Mmmm pies.

Reply ↓


lwolf on April
21, 2011 at 6:37 am said:

Hi! Thanks for great tutorial.

I faced an interesting situation: when I’m setting limit to 1gb it works fine, but when I’m setting 4 or 8gb it doesn’t work. mongo eating all free ram(15.2 from 16gb).

Any ideas what I’m doing wrong?

Reply ↓


Magnuson August
1, 2012 at 6:58 am said:

I have the same issue. Setting it to sup 1GB it all works fine. Any number above it and the policy wont kick in.

Not sure if it’s a bug in Resource Manager, but it’s not working.

Reply ↓


Mark on June
27, 2011 at 6:38 am said:

Very good post! Thanks a lot.

Reply ↓


dwight on October
1, 2011 at 9:02 am said:

It’s not clear to me this is necessary. The memory usage by mongo is primary just space in the file system cache. The OS should yield that space to other processes as they need it.

Reply ↓


Magnuson August
1, 2012 at 7:00 am said:

If you’ve ever used Mongo with 10GB< data on Windows you'd know that Mongo's RAM strategy just isn't working.

Once Mongo's "borrowed" all the memory the machine will be extremely slow.

Reply ↓


Ronon June
26, 2013 at 1:37 pm said:

Unfortunately not, it will use all memory and bring the entire Windows system to a crawl where it is unusable.

Reply ↓


Nathan
Thiya on March
26, 2012 at 11:47 am said:

Thanks my friend. This is exactly what I was looking for…

Reply ↓


Bryan on April
13, 2012 at 12:09 pm said:

Is it possible to install Windows System Resource Manager on Windows 7? I am doing some R&D work, and don’t have Windows 2008 handy.

Reply ↓

Pingback: Limit
Mongo CPU and Memory usage on Windows 2008 R2 | Avivo Tech Blog


mabead on August
9, 2012 at 7:14 pm said:

AS Iwolf said, WSRM can’t limit the memory so something bigger than 2047 Mb. Here is the issue on Microsoft connect.

https://connect.microsoft.com/WindowsServer/feedback/details/757646/windows-system-resource-manager-cant-limit-process-memory-usage-above-2047-mb

Reply ↓


Ron on June
26, 2013 at 12:36 am said:

This works until I specify more than 2GB of memory to mongo, then it quits working. 2GB works, 3GB does not work, it will grow and use all memory. Very strange, Trying a couple of values in between to see where it breaks. Any ideas?

Reply ↓


Ronon June
26, 2013 at 1:45 pm said:

Now see post just above mine regarding this issue. Unfortunately, the connect.microsoft.com link is no longer valid.

Reply ↓

Pingback: Windows
System Resource Manager working set memory limit not working above 2GB | BlogoSfera