GPU memory

Overview The big picture

Two kinds of resource

Everything the GPU reads or writes lives in one of two kinds of object:

• A buffer is a plain run of bytes: vertex data, triangle indices, a block of per-frame settings such as light directions.
• An image is structured pixel data: a texture, a depth map, a render target. Images have a size, a pixel format and a sample count, and are usually stored in a GPU-specific tiled arrangement that makes nearby pixels fast to fetch together. The CPU can't easily write that arrangement directly.

Creating either one gives you only a description. It has no memory yet. In Vulkan, you allocate memory separately and attach it, and that's where the choices come in.

Memory types

A GPU offers several memory types with different trade-offs. The two properties that matter most are:

• Device-local: fastest for the GPU. On a PC graphics card this is the card's own memory, which the CPU often can't see at all.
• Host-visible: the CPU can map it, getting an ordinary pointer to write through. It can be system memory or CPU-accessible device-local memory; performance depends on which memory type the driver exposes.

These are independent properties: a memory type can be both device-local and host-visible, even on a discrete graphics card.

On Apple Silicon the CPU and GPU share the same physical memory (unified memory), so the distinction is far smaller. Vulkan code is still written to handle both, which is why the code states a preference for each allocation rather than assuming.

Allocating Vulkan memory directly is laborious. You must search the memory types yourself, and drivers limit how many separate allocations you may make (to about four thousand on many drivers), so real programs carve many resources out of a few big blocks. RustyVulkan uses VMA, AMD's widely used Vulkan Memory Allocator, which does both jobs. You say "a buffer this big, used like this, preferably CPU-writable", and VMA picks the memory type and sub-allocates it.

Two ways to upload

This renderer gets data onto the GPU in one of two ways, depending on what the data is.

Write it in place. Vertex and index buffers, and the per-frame settings, are put in host-visible memory. The CPU maps it, copies the bytes in, and the GPU reads from there. That's simple, and on unified memory it costs little.

Stage it. Textures (and the grass's large instance buffers, Chapter 14) go into device-local memory. Textures in particular are images with an optimal tiled layout, which the CPU can't write. So the pixels are first copied into a temporary host-visible staging buffer, and then the GPU itself is told to copy from that buffer into the image, rearranging the pixels as it goes. The staging buffer has to stay alive until the GPU has actually done that copy.

The bottom row introduces an idea that recurs throughout Vulkan. An image's layout is the arrangement its data is in, and each kind of use expects a particular one. Moving an image from one layout to another is an explicit step, recorded with a barrier, which also makes sure the previous step has finished. Chapter 10 covers barriers properly.

Skip to next overview →

The allocator

VMA is created once, straight after the device, in VulkanBase::new:

// Create a Vulkan Memory Allocator (VMA) instance.
let allocator_info = vk_mem::AllocatorCreateInfo::new(&instance, &device, physical_device);
let allocator = unsafe { Allocator::new(allocator_info)? };

VMA estimates the memory budget by default. Its EXT_MEMORY_BUDGET flag would also require enabling VK_EXT_memory_budget on the device, so it is left unset here. The allocator is stored as an Option so it can be destroyed at a chosen moment during teardown, before the Vulkan device. The rest of the code borrows it through vb.allocator.as_ref().

The shared uniform and push-constant structures live in src/graphics/gpu_data.rs. Like the vertex below, they use repr(C) and Pod. Padding fields are explicit where needed. Compile-time assertions check the Rust sizes and offsets expected by the GLSL declarations; changes to that interface must update both languages together.

A vertex, as the GPU sees it

Before uploading vertices, the code needs a Rust type whose memory layout the GPU can read:

/// A single vertex: 3D position + normal, color, uv, tangent, bitangent.
#[repr(C)]
#[derive(Default, Clone, Copy, Pod, Zeroable)]
pub struct Vertex {
    pub pos: [f32; 3],
    pub normal: [f32; 3],
    pub color: [f32; 3],
    pub uv: [f32; 2],
    pub tangent: [f32; 3],
    pub bitangent: [f32; 3],
}

The same file also describes this layout to Vulkan, field by field, with offset_of!(Vertex, normal) giving each field's byte offset. That description is part of the pipeline, so it's covered in Chapter 7.

Vertex and index buffers: write in place

pub fn new<T: Pod>(allocator: &Allocator, data: &[T]) -> Result<Self, Box<dyn Error>> {
    let size = std::mem::size_of_val(data) as vk::DeviceSize;

    let buffer_info = vk::BufferCreateInfo {
        size,
        usage: vk::BufferUsageFlags::VERTEX_BUFFER,
        sharing_mode: vk::SharingMode::EXCLUSIVE,
        ..Default::default()
    };
    let alloc_info = vk_mem::AllocationCreateInfo {
        usage: MemoryUsage::AutoPreferHost,
        flags: vk_mem::AllocationCreateFlags::HOST_ACCESS_SEQUENTIAL_WRITE
            | vk_mem::AllocationCreateFlags::MAPPED,
        ..Default::default()
    };
    let (buffer, mut allocation) =
        unsafe { allocator.create_buffer(&buffer_info, &alloc_info)? };

    unsafe {
        // SAFETY: MAPPED keeps this allocation mapped; it holds the whole data slice.
        let ptr = allocator.get_allocation_info(&allocation).mapped_data as *mut T;
        std::ptr::copy_nonoverlapping(data.as_ptr(), ptr, data.len());
        if let Err(error) = allocator.flush_allocation(&allocation, 0, size) {
            allocator.destroy_buffer(buffer, &mut allocation);
            return Err(error.into());
        }
    }

    Ok(VertexBuffer { buffer, allocation })
}

Step by step:

• The buffer is described: its size in bytes, and that it'll be used as a vertex buffer.
• VMA is asked for memory: AutoPreferHost picks CPU-visible memory where possible, and HOST_ACCESS_SEQUENTIAL_WRITE promises that the CPU will only write it, in order. That lets VMA choose memory that's fast to write but slow to read back. create_buffer creates the buffer, allocates memory and binds them together, returning both.
• MAPPED keeps the allocation mapped. The data is copied with copy_nonoverlapping (Rust's memcpy, except that it counts elements of type T, not bytes), then flushed and kept mapped for its lifetime. No extra map/unmap pair is needed.

The flush matters. Some host-visible memory is non-coherent: the CPU's writes may sit in its caches, invisible to the GPU, until explicitly flushed. VMA's flush does nothing on coherent memory, so calling it unconditionally is the portable choice. (Apple Silicon's memory is coherent, so here the flush is effectively free.) DEVELOPMENT.md makes it a rule: flush non-coherent host writes before submission.

IndexBuffer::new is the same with u32 indices and INDEX_BUFFER usage. LoadedMesh::load creates one of each for a mesh, keeping the mesh's bounding box for placing it on the terrain later.

Uniform data: write in place, every frame

The per-frame block of settings (sun direction, shadow matrices, lights) also lives in host-visible buffers, one per swapchain image, created with VMA's MAPPED flag so they stay mapped for their whole life. Updating one is just a copy and a flush:

pub fn update_global_ubo(&mut self, image_index: usize, ubo: &GlobalUbo) {
    let allocator = self.allocator.as_ref().expect("allocator");
    let allocation = &self.ubo_allocations[image_index];
    unsafe {
        // These allocations were created with MAPPED, so VMA keeps the pointer valid
        // for their lifetime and the per-frame path does not need map/unmap calls.
        let ptr = allocator.get_allocation_info(allocation).mapped_data as *mut u8;
        debug_assert!(!ptr.is_null());
        std::ptr::copy_nonoverlapping(
            bytemuck::bytes_of(ubo).as_ptr(),
            ptr,
            std::mem::size_of::<GlobalUbo>(),
        );
        allocator
            .flush_allocation(allocation, 0, std::mem::size_of::<GlobalUbo>() as u64)
            .expect("flush UBO");
    }
}

There's one buffer per swapchain image so that the CPU never overwrites data the GPU is still reading for an earlier frame. Chapter 9 shows how the frame loop guarantees that, and Chapter 8 how shaders find these buffers.

Textures: stage, copy, transition

Texture uploads are batched. While the scene is being built, every texture's copy commands are recorded into one command buffer, which is submitted once at the end. Here is TextureUploadBatch::upload, first decoding the file and filling a staging buffer:

// Decode on the CPU first so every source format becomes RGBA8.
let img = image::open(image_path)?.to_rgba8();
let (width, height) = img.dimensions();
let pixels = img.into_raw();

// Stage the pixels in host-visible memory; the final image stays device-local.
let buffer_info = vk::BufferCreateInfo {
    size: pixels.len() as vk::DeviceSize,
    usage: vk::BufferUsageFlags::TRANSFER_SRC,
    sharing_mode: vk::SharingMode::EXCLUSIVE,
    ..Default::default()
};
let alloc_info = vk_mem::AllocationCreateInfo {
    usage: MemoryUsage::AutoPreferHost,
    flags: vk_mem::AllocationCreateFlags::HOST_ACCESS_SEQUENTIAL_WRITE,
    ..Default::default()
};
let (staging_buffer, mut staging_allocation) =
    unsafe { allocator.create_buffer(&buffer_info, &alloc_info)? };
let write_result = (|| -> Result<(), vk::Result> {
    unsafe {
        let data_ptr = allocator.map_memory(&mut staging_allocation)?;
        std::ptr::copy_nonoverlapping(pixels.as_ptr(), data_ptr, pixels.len());
        let result =
            allocator.flush_allocation(&staging_allocation, 0, pixels.len() as u64);
        allocator.unmap_memory(&mut staging_allocation);
        result
    }
})();
if let Err(error) = write_result {
    unsafe {
        allocator.destroy_buffer(staging_buffer, &mut staging_allocation);
    }
    return Err(error.into());
}

The image crate decodes PNG or JPEG and converts it to 8-bit RGBA. The staging buffer's usage is TRANSFER_SRC: it will only ever be copied from. Next, the real image:

let image_info = vk::ImageCreateInfo {
    image_type: vk::ImageType::TYPE_2D,
    format: vk::Format::R8G8B8A8_UNORM,
    extent: vk::Extent3D {
        width,
        height,
        depth: 1,
    },
    mip_levels: 1,
    array_layers: 1,
    samples: vk::SampleCountFlags::TYPE_1,
    tiling: vk::ImageTiling::OPTIMAL,
    usage: vk::ImageUsageFlags::TRANSFER_DST | vk::ImageUsageFlags::SAMPLED,
    sharing_mode: vk::SharingMode::EXCLUSIVE,
    initial_layout: vk::ImageLayout::UNDEFINED,
    ..Default::default()
};
let image_alloc_info = vk_mem::AllocationCreateInfo {
    usage: MemoryUsage::AutoPreferDevice,
    ..Default::default()
};
let (image, mut allocation) =
    unsafe { allocator.create_image(&image_info, &image_alloc_info)? };

OPTIMAL tiling is the GPU's own arrangement. The usage flags say it will be copied into (TRANSFER_DST) and read by shaders (SAMPLED). AutoPreferDevice asks for device-local memory. It has one mip level only: this renderer doesn't generate smaller mipmap versions of its textures. And it's R8G8B8A8_UNORM, the format behind Chapter 4's note about gamma.

Then three commands are recorded: a barrier to move the image into the transfer layout, the copy, and a barrier to make it shader-readable. The two barrier structs are built just above (elided):

unsafe {
    // undefined -> transfer destination -> shader-readable
    device.cmd_pipeline_barrier2(
        self.command_buffer,
        &vk::DependencyInfo::default().image_memory_barriers(&[to_transfer]),
    );
    device.cmd_copy_buffer_to_image(
        self.command_buffer,
        staging_buffer,
        image,
        vk::ImageLayout::TRANSFER_DST_OPTIMAL,
        &[region],
    );
    device.cmd_pipeline_barrier2(
        self.command_buffer,
        &vk::DependencyInfo::default().image_memory_barriers(&[to_shader_read]),
    );
}

Each barrier carries its own pair of pipeline stages: "work at the source stage must finish before work at the destination stage may start". The first has no source stage, since nothing has touched the new image yet, but the copy may not start until the layout change is done. The second makes fragment shaders wait for the copy to complete. These are synchronization2 barriers, explained in Chapter 10.

Nothing has actually happened yet. These are only recorded commands. So the staging buffer can't be freed here; the batch took ownership before image creation:

// The batch owns staging before any subsequent step can fail.
self.staging_buffers.push(StagingBuffer {
    buffer: staging_buffer,
    allocation: staging_allocation,
});

When scene construction is done, TextureCache::flush calls finish:

unsafe {
    device.end_command_buffer(self.command_buffer)?;
    self.fence = device.create_fence(&vk::FenceCreateInfo::default(), None)?;
    let command_buffers =
        [vk::CommandBufferSubmitInfo::default().command_buffer(self.command_buffer)];
    let submit_info = vk::SubmitInfo2::default().command_buffer_infos(&command_buffers);
    device.queue_submit2(queue, &[submit_info], self.fence)?;
    // On failure the cache retains the batch, fence, and staging allocations.
    // App teardown waits for the device before cache cleanup.
    device.wait_for_fences(&[self.fence], true, u64::MAX)?;
}

Submit, then wait on a fence (a CPU-visible "done" signal) until the GPU finishes, and only then flush releases the completed batch through discard, destroying its staging buffers and fence. Stalling like this is fine at start-up, once. The frame loop, in Chapter 9, uses the same fence idea but avoids waiting.