Photo Triage in Zig

What is this project about and how did it start?

Recently I went to Stanford to attend my high school friend’s graduation. Over the weekend, their parents took hundreds of pictures of the “kids” (us). I now had several gigabytes worth of .jpg image on my hard drive.

I tried lossless optimization, but wasn’t satisfied, leading me to do my JPG -> PNG -> WEBP conversion trick using imagemagick’s mogrify (this was very slow).

As expected, I now had ballooned the folder size from ~5 Gb to around 25 Gb, with 3x the number of images. For a .jpg that was about 5-6 Mb, with each .png being anywhere from 14 Mb to 23 Mb.

images/
  photo-1.jpg (5 Mb)
  photo-1.png (14 Mb)
  photo-1.webp (800 Kb)
  photo-2.jpg (4.5 Mb)
  photo-2.png (20 Mb)
  photo-2.webp (737 Kb)
  photo-3.jpg (6.1 Mb)
  photo-3.png (16 Mb)
  photo-3.webp (500 Kb)
  ...

Alright, I now have over 1700 files, and I have to delete 2/3 rds of them, specifically, all but the smallest version of an image.

The file strucure above should look like this:

images/
  photo-1.webp (800 Kb)
  photo-2.webp (737 Kb)
  photo-3.webp (500 Kb)
  ...

This should be easy right? Simply rm *.png *.jpg!

However, there are two problems with the approach above:

1. What if the smallest image isn’t the .webp? (Check the table for the outlier).
2. This is incredibly boring!

Solution

I therefore decided to write some code in Zig. I am still unsure why I decided to do Zig, however I did not want to write any JavaScript or Python, as I know these languages (not a challenge), deletion would have been slow, and I didn’t feel like using Node/Conda (Mamba).

I could have chosen to use Rust but I never had fun with that language.

Let’s jump into Zig!

(Thank you to the person who answered my StackOverflow question)

Problem description

Scan a folder full of files (dosn’t have to be images), and group them by name. Delete all but the smallest file of each name.

First attempt & LLMs

At first, I wanted to see if ChatGPT or Claude could do it, so I ask both to code something up.

Although both got the strcuture right, it seems like their training data does not include much Zig (which is not surprising considering it is at version 0.13.0 at time of writing).

Anyway, it didn’t work, time to figure it out myself.

My Code

I will upload my script(s) to github at some point. If I forget, please send me a DM on twitter. (Code is also on StackOverflow)

I’m going to only post the interesting parts here.

I had much trouble getting this to work, and unfortunately I stuck with the LLM approach for my first version, which meant using an ArrayList. Of course a Hashmap makes more sense, but here is what I came up with.

Initialization

const fileStruct = struct { baseName: []const u8, files: [][]const u8 };

var gpa = std.heap.GeneralPurposeAllocator(.{}){};
defer std.debug.assert(gpa.deinit() == .ok);
const allocator = gpa.allocator();

var groupedFiles = std.ArrayList(fileStruct).init(allocator);
defer {
  for (groupedFiles.items) |group| {
    for (group.files) |file| {
      allocator.free(file);
    }
    allocator.free(group.files);
    allocator.free(group.baseName);
  }
  groupedFiles.deinit();
}

Here we can see I’m basically making a HashMap without its benefits.

Iterating through the files and building the ArrayList

This all takes place in a while loop iterating through the files in the target directory.

Command line arguments

const args = try std.process.argsAlloc(allocator);
defer std.process.argsFree(allocator, args);

if (args.len < 2) {
  std.debug.print("Usage: {s} <directory>\n", .{args[0]});
  return;
}

const dir_path: []const u8 = args[1];
std.debug.print("Lets open the {s} directory\n", .{dir_path});

var target_dir = try fs.cwd().openDir(dir_path, .{});
defer target_dir.close();

Interesting that args has its own method to have its memory freed.

Case 1: the basename already exists in the array list

var foundGroup: bool = false;

for (groupedFiles.items) |*group| {
  if (mem.eql(u8, group.baseName, baseName)) {
    const new_files = try allocator.alloc([]const u8, group.files.len + 1);
    for (group.files, 0..) |file, i| {
      new_files[i] = file;
    }
    new_files[group.files.len] = try allocator.dupe(u8, entry.name);
    allocator.free(group.files);
    group.files = new_files;
    foundGroup = true;
    break;
  }
}

At first I struggled to understand why I need to use *group, and allocator.dupe(u8, entry.name);.

Case 2: the basename doesn’t exist in the array list

if (!foundGroup) {
  const new_files = try allocator.alloc([]const u8, 1);
  new_files[0] = try allocator.dupe(u8, entry.name);
  const newGroup = fileStruct{ .baseName = try allocator.dupe(u8, baseName), .files = new_files };
  try groupedFiles.append(newGroup);
}

Simple stuff! (we will NOT be discussing how long this took to get right, specifically why I have to .dupe)

Deleting the unwanted files

Now, we have to reiterate through our arraylist.

for (groupedFiles.items) |item| {
  if (item.files.len > 1) {
    var smallestSize: u64 = 0;
    var smallestIndex: usize = 0;

    for (item.files, 0..) |currentFile, idx| {
      const fullPath = try path.join(allocator, &[_][]const u8{ dir_path, currentFile });
      defer allocator.free(fullPath);
      var path_buffer: [std.fs.MAX_PATH_BYTES]u8 = undefined;
      const realPath = try fs.realpath(fullPath, &path_buffer);
      const file = try fs.openFileAbsolute(realPath, .{});
      defer file.close();
      const info = try file.stat();
      const size = info.size;
      std.debug.print("{any} -> {s}\n", .{ size, realPath });

      if (smallestSize == 0 or size < smallestSize) {
        smallestSize = size;
        smallestIndex = idx;
      }
    }

    std.debug.print("Smallest: {any} -> {any}\n", .{ smallestSize, smallestIndex });

    for (item.files, 0..) |file, idx| {
      if (idx != smallestIndex) {
        const fullPath = try path.join(allocator, &[_][]const u8{ dir_path, file });
        defer allocator.free(fullPath);
        var path_buffer: [std.fs.MAX_PATH_BYTES]u8 = undefined;
        const realPath = try fs.realpath(fullPath, &path_buffer);
        try fs.deleteFileAbsolute(realPath);
        std.debug.print("Deleted {s}\n", .{fullPath});
      }
    }
  }
}

I do NOT like this code. Not one bit. I am frsutrated that I have to loop multiple times throughout the program, but I haven’t tried any optimization (yet) and getting to this point was time consuming.

What I dislike the most is getting the file and its absolute path:

const fullPath = try path.join(allocator, &[_][]const u8{ dir_path, file });
defer allocator.free(fullPath);
var path_buffer: [std.fs.MAX_PATH_BYTES]u8 = undefined;
const realPath = try fs.realpath(fullPath, &path_buffer);

I’m going to study why this is all necessary (path_buffer, and &[_][]const u8{ dir_path, file}), but if someone is willing to explain please contact me.

What I enjoyed about Zig

The mystery. Documentation is there alas it is incomplete, especially for anything with fs. Having to directly inspect package code was insightful.

Memory managment (with defer) requires thought and makes me appreciate my code and higher-level languages more.

Downsides of using Zig

The online fs documentation and its lack of examples.

Conclusion

Overall, I’m very pleased with this language. This code was also very fast, doing “the job” of deleting ~24 Gb of data in a fraction of a second.

I’m excited to use Zig more, as I like the control I have over hardware/memory. This does in turn put my brain in duress, but learning is satisfying.

This was the first time I have used Zig, and I expect to come back to this problem with a better solution in the near future. In the meantime, I’m going to figure out what else I can do with this language.