The memtrace Task

Type

Source

Output

memtrace

memtrace is a tracing task that produces memory events containing process information, event type, and memory sizes. The data is collected through various memory-related tracepoints to ensure comprehensive visibility into memory operations.

Quick Intro into Virtual vs Physical Memory Events

The memtrace task monitors both virtual memory allocations (mmap/munmap) and physical memory allocations (Page Allocation/Free):

Virtual Memory (mmap/munmap):

  • Creates a mapping in the process’s address space without immediately allocating physical memory

  • Commonly used when loading files, creating shared memory, or reserving memory regions

  • Physical pages are only allocated when the virtual memory is actually accessed (demand paging)

Physical Memory (Page Allocation/Free):

  • Represents actual physical memory pages being allocated or released by the kernel

  • Occurs when virtual memory is accessed for the first time or when the kernel needs memory

  • Directly corresponds to RAM usage

For example, a call to malloc will only result in a call to mmap if the process needs more memory. The mmap call will simply allocate virtual memory and not allocate physical memory immediately. Due to on demand paging, physical memory is only allocated once the user access the new memory.

Memory Events Generated by Memtrace

During its execution, the task monitors four types of memory events:

  • Page Allocation (pageAllocation): When physical new memory pages are allocated by the kernel

  • Page Free (pageFree): When physical memory pages are released back to the kernel

  • Mmap (memoryMap): When virtual memory mappings are created (e.g., loading files, allocating memory regions)

  • Munmap (memoryUnmap): When virtual memory mappings are removed

  • Page Faults: Occurs when a process attempts to access a memory page that is not currently mapped into virtual memory.

Sample Command Lines

Tawon conditions can be set to filter processes based on various attributes. For example, to target all processes in a specific container and monitor their memory operations:

tawonctl d sub -c "container.name:myapp" -t "memtrace"

The example below targets pods in the default namespace with the label 'prod':

tawonctl d sub -c "pod.namespace:default AND pod.label:prod" -t "memtrace"

To monitor memory events for a specific process:

tawonctl d sub -c "process.name:nginx" -t "memtrace"

Conditions

memtrace will not produce data without a Condition.

memtrace requires one of the following Conditions to be defined:

process.pid

Filtering memory events by Process PID.

process.name

Filtering memory events by Process Name.

process.cmd

Filtering memory events by Process Command.

process.exe

Filtering memory events by Process Executable.

process.capinh

Filtering memory events by Process' Inherited capabilities.

process.capprm

Filtering memory events by Process' Permitted capabilities.

process.capeff

Filtering memory events by Process' Effective capabilities.

process.capbnd

Filtering memory events by Process' Bound capabilities.

process.capamb

Filtering memory events by Process' Ambient capabilities.

container.id

Filtering memory events by Container ID/UID.

container.name

Filtering memory events by Container Name.

container.label

Filtering memory events by Container Label value or key=value.

pod.id

Filtering memory events by Pod ID/UID.

pod.name

Filtering memory events by Pod Name.

pod.namespace

Filtering memory events by Pod K8s Namespace.

pod.label

Filtering memory events by Pod Label value or key=value.

Output

memtrace

map Holds the memory event output.

memtrace.evt_type

string Type of memory event (pageAllocate, pageFree, memoryMap, memoryUnmap, pageFault).

memtrace.size

uint64 Size of memory operation in bytes.

memtrace.pid

uint64 Process ID of the event source.

memtrace.process

map Holds the Process information.

memtrace.process.pid

uint32 Process PID.

memtrace.process.ppid

uint32 Process Parent PID.

memtrace.process.name

string Process name.

memtrace.process.cmd

string Command line string.

memtrace.process.exe

string Executable name.

memtrace.process.ns

string Namespace in the format of {device ID}-{inode number}.

memtrace.process.startedAt

string Time the process started.

memtrace.process.container

map Container information for process (if it is a container).

memtrace.process.container.ID

string ID of Container.

memtrace.process.container.Name

string Name of Container.

Sample output:

"data": [
    {
        "data": {
            "evt_type": "pageAllocate",
            "size": 4096,
            "pid": 1234,
            "process": {
                "cmd": "/usr/bin/myapp",
                "container": {
                    "id": "780f159f59c75b09da4103566b39da1a3793beb20a11db48e1056601036472fe",
                    "name": "/webapp"
                },
                "exe": "/usr/bin/myapp",
                "name": "myapp",
                "ns": "4-4026533309",
                "pid": 1234,
                "ppid": 1200,
                "startedAt": "2023-12-11T21:04:00Z"
            }
        },
        "type": "memtrace"
    }
],
    "src": {
        "directiveID": "clroeh9eo0msn9golhkg",
        "hostname": "8e77aad68513",
        "instanceID": "clroefpeo0msnck6dus0",
        "name": "memtrace",
        "tawonID": "cfr7vtqm49rliqd1vs0g"
    },
    "timestamp": "2023-12-11T22:03:59.277299703Z"
}