Child pages
  • VM Troubleshooting
Skip to end of metadata
Go to start of metadata

Overview

This document covers the steps necessary to reproduce the apparent memory issue with our CentOS VMs, in particular the wiki-dev VM.

What Happens

To reproduce this problem, do the following:

  1. Reboot server
  2. Look at memory usage: about 5GB free, and going down
  3. Watch Confluence start up in about 2 minutes
  4. Stop Confluence (it may not stop, so you might have to kill the process, but give it a few minutes to try to exit cleanly)
  5. Look at memory usage: on a server with 6GB of RAM, only 1.5GB are free
  6. Start Confluence – it will now take about 10 minutes to start, and there could be considerable errors
  7. Look at memory usage: probably about 500MB free
  8. Stop Confluence – will probably have to kill the process this time
  9. Look at memory usage: still 1.5 GB free

Clearing the Filesystem Cache

Linux caches pages and inodes in memory, but you are supposed to be able to clear these. To do this, type the following:

free -m && sync ; echo 3 > /proc/sys/vm/drop_caches ; free -m

The free command is given so that you can see how much is being used before and after the sync; echo... is run.

sync ; echo... on wiki-dev

             total       used       free     shared    buffers     cached
Mem:          5963       4021       1942          0          4         32
-/+ buffers/cache:       3983       1979
Swap:         8927          0       8927
             total       used       free     shared    buffers     cached
Mem:          5963       3998       1965          0          0         13
-/+ buffers/cache:       3983       1979
Swap:         8927          0       8927

As you can see, not much is freed.

sync ; echo... on Sven clone

[root@lor-dev ~]# free -m
            total       used       free     shared    buffers     cached
Mem:          7983        569       7414          0          4        280
-/+ buffers/cache:        284       7699
Swap:          767          0        767
            total       used       free     shared    buffers     cached
Mem:          7983        345       7638          0          0         62
-/+ buffers/cache:        282       7701
Swap:          767          0        767

A lot more is freed here. Of course, the Sven clone has 2GB more of RAM, but the numbers are telling: 4GB used on wiki-dev with no Java running, and less than 400MB on the Sven clone with no Java running.

Below is another snapshot of freeing memory on the Sven clone:

            total       used       free     shared    buffers     cached
Mem:          7983       1525       6458          0        196        965
-/+ buffers/cache:        363       7620
Swap:          767          0        767
            total       used       free     shared    buffers     cached
Mem:          7983        405       7577          0          0         62
-/+ buffers/cache:        343       7640
Swap:          767          0        767

Again, as you can see in this run, the result is 400MB, not 4GB, as is the case with wiki-dev.

Conclusion

The above is how the filesystem cache clearing should work. But, for some reason, it does not work this wan on wiki-dev.

After trying lots of things on wiki-dev, I can not get it to have more than 2GB of free memory, and yet, the Sven clone has almost all of its memory free. This test should be reproduced on Stefan's previous Alfresco VM to see if it behaves the same.

References

  • No labels

1 Comment

  1. Since the inexplicable portfolio performance issues we've had have revolved around memory, it seems probable that it has a similar cause to this.