Large Memory Optimization --- Huge Pages
=====================================================

Pages in Memory
---------------
Linux allocates memory to processes by dividing the physical memory into pages, and then mapping those physical pages to the virtual memory needed by a process. It does this in conjunction with the Memory Management Unit (MMU) in the CPU. Typically a page will represent 4KB of physical memory. Statistics and flags are kept about each page to tell Linux the status of that chunk of memory.

Huge Pages in Red Hat Enterprise Linux 3, 4 and 5 are very useful for large Oracle SGA sizes and in general for systems with large amount of physical memory. It optimizes the use of Translation Lookaside Buffers (TLB), locks these larger pages in RAM, and the system has less bookkeeping work to do for that part of virtual memory due to larger page sizes.

Physical memory is partitioned into pages which are the basic unit of memory management.
When a Linux process accesses a virtual address, the CPU must translate it into a physical address.
Therefore, for each Linux process the kernel maintains a page table which is used by the CPU to translate virtual addresses into physical addresses.
But before the CPU can do the translation it has to perform several physical memory reads to retrieve page table information.
To speed up this translation process for future references to the same virtual address, the CPU saves information for recently accessed virtual addresses in its Translation Lookaside Buffers (TLB) which is a small but very fast cache in the CPU.
The use of this cache makes virtual memory access very fast.
Since TLB misses are expensive, TLB hits can be improved by mapping large contiguous physical memory regions by a small number of pages. So fewer TLB entries are required to cover larger virtual address ranges. A reduced page table size also means a reduction in memory management overhead.
To use larger page sizes for shared memory, Big Pages (Red Hat Enterprise Linux 2.1) or Huge Pages (Red Hat Enterprise Linux 3, 4 and 5) must be enabled which also locks these pages in physical memory.

=> . The Huge Pages feature in Red Hat Enterprise Linux 3 or 4 allows you to dynamically allocate large memory pages without a reboot.

 The advantages of Big Pages and Huge Pages for database performance are:

 =>    increased performance through increased Translation Lookaside Buffer (TLB) hits.
 =>    pages are locked in memory and are never swapped out which guarantees that shared memory such as SGA remains in RAM.
 =>    contiguous pages are pre-allocated and cannot be used for anything else but for System V shared memory, for example SGA.
 =>    less bookkeeping work for the kernel in that part of virtual memory due to larger page sizes.
 
 -->> Huge Pages in Red Hat Enterprise Linux 3, 4 and 5 need to be requested explicitly by the application by using the SHM_HUGETLB flag when invoking the shmget() system call. This ensures that shared memory segments are allocated out of the Huge Pages pool.
 
 
-->> With the Big Pages and Huge Pages feature you specify how many physically contiguous large memory pages should be allocated and pinned in RAM for shared memory like Oracle SGA.
For example, if you have three Oracle instances running on a single system with 2 GB SGA each, then at least 6 GB of large pages should be allocated. This will ensure that all three SGAs use large pages and remain in main physical memory. Furthermore, if you use ASM on the same system, then it is prudent to add an additional 200MB. I have seen ASM instances creating between 70 MB and 150 MB shared memory segments. And there might be other non-Oracle processes that allocate shared memory segments as well.

-->> It is, however, not recommended to allocate too many Big or Huge Pages. These pre-allocated pages can only be used for shared memory. This means that unused Big or Huge Pages will not be available for other use than for shared memory allocations even if the system runs out of memory and starts swapping.

-->> IMPORTANT

It is very important to always check the shared memory segments before starting an instance. An abandoned shared memory segment, from an instance crash for example, is not removed, it will remain allocated in the Big Pages or Huge Pages pool. This could mean that new allocated shared memory segments for the new instance SGA will not fit into the Big Pages or Huge Pages pool.

 

===>>>> Configuring Huge Pages in Red Hat Enterprise Linux 4 or 5

In Red Hat Enterprise Linux 4 or 5 the size of the Huge Pages pool is specified by the desired number of Huge Pages.
To calculate the number of Huge Pages you first need to know the Huge Page size. To obtain the size of Huge Pages, execute the following command:

[root@ace2oracledb ~]# grep Hugepagesize /proc/meminfo
Hugepagesize:       2048 kB
[root@ace2oracledb ~]#

The output shows that the size of a Huge Page on this system is 2MB.This means if a 1GB Huge Pages pool should be allocated, then 512 Huge Pages need to be allocated.

Where as Normal page_size would be 4KB and this is difference .

The number of Huge Pages can be configured and activated by setting nr_hugepages in the proc file system. For example, to allocate 512 Huge Pages, execute:

[root@ace2oracledb vm]# cat /proc/sys/vm/nr_hugepages
0
[root@ace2oracledb vm]#

# echo 512 > /proc/sys/vm/nr_hugepages

 

 To make the change permanent, add the following line to the file /etc/sysctl.conf. This file is used during the boot process. The Huge Pages pool is usually guaranteed if requested at boot time:

# echo "vm.nr_hugepages=512" >> /etc/sysctl.conf

------->>>>
To verify whether the kernel was able to allocate the requested number of Huge Pages, run:

$ grep HugePages_Total /proc/meminfo
HugePages_Total:   512
$

The output shows that 512 Huge Pages have been allocated. Since the size of Huge Pages is 2048 KB, a Huge Page pool of 1GB has been allocated and pinned in physical memory.

==========>>>

Note
In order that an Oracle database can use Huge Pages in Red Hat Enterprise Linux 4 or 5, you also need to increase the ulimit parameter "memlock" for the oracle user in /etc/security/limits.conf if "max locked memory" is not unlimited or too small, see ulimit -a or ulimit -l. An example can be seen below.

oracle           soft    memlock         1048576
oracle           hard    memlock         1048576

The memlock parameter specifies how much memory the oracle user can lock into its address space. Note that Huge Pages are locked in physical memory. The memlock setting is specified in KB and must match the memory size of the number of Huge Pages that Oracle should be able to allocate. So if the Oracle database should be able to use 512 Huge Pages, then memlock must be set to at least 512 * Hugepagesize, which on this system would be 1048576 KB (512*1024*2). If memlock is too small, then no single Huge Page will be allocated when the Oracle database starts.

========================================DONE FROM NOTE1===========================================================
Without HugePages, the memory of the SGA is divided into 4K pages, which have to be managed by the Linux kernel. Using HugePages, the page size is increased to 2MB (configurable to 1G if supported by the hardware), thereby reducing the total number of pages to be managed by the kernel and therefore reducing the amount of memory required to hold the page table in memory.

 Automatic Memory Management (AMM) is not compatible with Linux HugePages, so apart from ASM instances and small unimportant databases, you will probably have no need for AMM on a real database running on Linux. Instead, Automatic Shared Memory Management and Automatic PGA Management should be used as they are compatible with HugePages.
 
 REASON -> In addition to these changes, the memory associated with HugePages can not be swapped out, which forces the SGA to stay memory resident.
 
--> Check the MEMORY_TARGET parameters are not set for the database and SGA_TARGET and PGA_AGGREGATE_TARGET parameters are being used instead.

===>>> >USE_LARGE_PAGE Parameter in Oracle database

Sizing the number of HugePages correctly is important because prior to 11.2.0.3, if the whole SGA doesn't fit into the available HugePages, the instance will start up without using any.
From 11.2.0.3 onward, the SGA can run partly in HugePages and partly not, so the impact of this issue is not so great.
Incorrect sizing may not be obvious to spot. Later releases of the database display a "Large Pages Information" section in the alert log during startup.

USE_LARGE_PAGES --> True| Only | false

USE_LARGE_PAGES is used to manage the database's use of large pages for SGA memory. This parameter does not affect process-private memory allocations. This parameter is applicable only on the Linux operating system.

Values:

TRUE

    Specifies that the instance can use large pages if large pages are configured on the system.

    In Oracle Database 11g Release 2 (11.2.0.3), Oracle allocates as much of the SGA as it can in large pages, and if it runs out, it will allocate the rest of the SGA using regular sized pages. This can cause the instance to create additional shared memory segments for the SGA, but the total SGA size will be unchanged. In this supported mixed page mode allocation, the database will exhaust the available large pages before switching to regular sized pages.

    FALSE

    Specifies that the instance will not use large pages. This setting is not recommended because it can cause severe performance degradation for the instance.

    ONLY

    Specifies that the instance will fail to start if large pages cannot be used for the entire SGA memory.

=====
=====
********** About Transparent Hugepages ************

What is the difference between HugePages and transparent huge pages?

Transparent Hugepages are similar to standard HugePages. However, while standard HugePages allocate memory at startup, Transparent Hugepages memory uses the khugepaged thread in the kernel to allocate memory dynamically during runtime, using swappable HugePages.

“Transparent Hugepages” is a Linux kernel feature intended to improve performance by making more efficient use of your processor’s memory-mapping hardware. It is enabled (“enabled=always”) by default in most Linux distributions.

Transparent Hugepages (“THP” for short), as the name suggests, intended to bring hugepage support automatically to applications, without requiring custom configuration. Transparent hugepage support works by scanning memory mappings in the background (via the “khugepaged” kernel thread), attempting to find or create (by moving memory around) contiguous 2MB ranges of 4KB mappings, that can be replaced with a single hugepage.

Issue with THP is memory leak,CPU eater

Transparent Huge Pages (THP) are enabled by default in RHEL 6 for all applications. The kernel attempts to allocate hugepages whenever possible and any Linux process will receive 2MB pages if the mmap region is 2MB naturally aligned. The main kernel address space itself is mapped with hugepages, reducing TLB pressure from kernel code. For general information on Hugepages

The kernel will always attempt to satisfy a memory allocation using hugepages. If no hugepages are available (due to non availability of physically continuous memory for example) the kernel will fall back to the regular 4KB pages. THP are also swappable (unlike hugetlbfs). This is achieved by breaking the huge page to smaller 4KB pages, which are then swapped out normally.

Check system-wide THP usage

Run the following command to check system-wide THP usage:

[root@ace2oracledb vm]# grep AnonHugePages /proc/meminfo
AnonHugePages:         0 kB
[root@ace2oracledb vm]#

IMPORTANT ========>>>>

To disable THP at boot time
-----------------------------
Append the following to the kernel command line in grub.conf:

transparent_hugepage=never

----------------------------
Oracle always insist to not use THP and it disable the same while running 19c-prerequisite setup .

Setting boot params - Checking for blscfg
old boot params: "crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet numa=off"
new boot params: "crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet numa=off transparent_hugepage=never"

========================================FROM ORACLE OTN================================================================

--->>> A.7 Overview of HugePages

A.7.5 Restrictions for HugePages Configurations

HugePages has the following limitations:

    You must unset both the MEMORY_TARGET and MEMORY_MAX_TARGET initialization parameters. For example, to unset the parameters for the database instance, use the command ALTER SYSTEM RESET.

    Automatic Memory Management (AMM) and HugePages are not compatible. When you use AMM, the entire SGA memory is allocated by creating files under /dev/shm. When Oracle Database allocates SGA with AMM, HugePages are not reserved. To use HugePages on Oracle Database 19c, You must disable AMM.

    If you are using VLM in a 32-bit environment, then you cannot use HugePages for the Database Buffer cache. You can use HugePages for other parts of the SGA, such as shared_pool, large_pool, and so on. Memory allocation for VLM (buffer cache) is done using shared memory file systems (ramfs/tmpfs/shmfs). Memory file systems do not reserve or use HugePages.

    HugePages are not subject to allocation or release after system startup, unless a system administrator changes the HugePages configuration, either by modifying the number of pages available, or by modifying the pool size. If the space required is not reserved in memory during system startup, then HugePages allocation fails.

    Ensure that HugePages is configured properly as the system may run out of memory if excess HugePages is not used by the application.

    If there is insufficient HugePages when an instance starts and the initialization parameter use_large_pages is set to only, then the database fails to start and an alert log message provides the necessary information on Hugepages.

A.7.6 Disabling Transparent HugePages

Oracle recommends that you disable Transparent HugePages before you start installation.

Transparent HugePages memory differs from standard HugePages memory because the kernel khugepaged thread allocates memory dynamically during runtime. Standard HugePages memory is pre-allocated at startup, and does not change during runtime.