Workflow Tip: How to Keep Files in Two Places at Once

Recently, I wanted to back up my UDK 3 project files to my Dropbox, but UDK requires project files to be in the same directory as the (4GB) installation itself—and 4GB is far too big for my Dropbox! In this article, you'll learn how you can solve this problem by making files appear to be in two places at once, using hard links, symbolic links, and junctions.

Preview image icon by P. J. Onori.

The Basic Concept

When you defragment your hard disk drive and the information is moved around, none of your saved files cease to work just because the information is stored in a different place. This is because your operating system has an abstraction layer that manages all of this; essentially, what you click to open an image or executable is just a link to the desired file or data.

Expanding on this, you can deduct that you could have more than one link to the same item; foo.exe and bar.exe could open the exact same program!

Let's get a little more technical...

Hard Links

Hard links are similar to file shortcuts; they enable you to have access to the same file in different locations. The main difference is that this implemented on the file system itself, so, when you create an hard link to a file, you are just creating a new name for it, and it will be the exact same file, with both names pointing to the exact same location in the hard drive.

If you make any changes to the file via any of the "names" (that is, if you open the original file or one of the hard links and modify it), the changes will be propagated to all the other places that the file exists, in regards to the file attributes and the file itself.

Each file will exist until all of its hard links are deleted; when that happens, the space used by the file is declared as free. (You can use a file restore tool to get your file back, but that's not guaranteed to work.)

Junctions (or Junction Points)

Hard links are to files what junctions are to directories, partitions or volumes, on (and only on) the same system. Both are interpreted at the operating system level and, as a result, are transparent to programs and users.

Symbolic Links

Unlike the above, symbolic links can be used for both files and folders, thus, in essence and perception, they are really similar to shortcuts.

The difference is, a symbolic link points to the original file path, rather than to the file's location on your hard drive. This means that if you change the contents of the file, a symbolic link will still point to it (and will reflect the changes), but if you change the name of the original file, the symbolic link will no longer point to it.

Practical Applications

I have used this to solve a few practical problems.

UDK 3 Project Management Nightmare

Anyone that has worked with UDK 3 knows that, although the engine is great, the project management workflow is cumbersome, for instance, it's not trivial to deal with having multiple projects in the same installation, so developers usually have multiple installations for multiple projects. It gets worse if you start thinking about backing up your project, as I mentioned above.

For my private projects, I always use two backup solutions: source control and cloud backup, with BitBucket and Dropbox respectively. Setting up source control of your code in UDK is trivial, but, what if you want to back up your custom project with a cloud backup solution like Dropbox that requires all of the files to be in a specific system folder?

Here's my solution:

• In your cloud backup folder, recreate the folder structure for your custom project files, so that it's possible to just copy and paste in another UDK installation. (This is an added bonus and helps if you want to take your project on an USB pen to an offline device.)
• Confirm that your project files (including UDK configuration files that set up your custom project) are correctly placed.
• Create junctions of the custom project folder to the UDK installation, and hard links for the isolated files outside those folders.

A Well-Organized Music Library

I've always had problems with the way I organized my music library folder; since I was never a fan of using apps like iTunes, and I wanted my hard drive clean and organized. After spending years trying different approaches, the one I liked most was having one folder per artist and, inside, the respective artist singles and multiple folders per album.

That's all good, until you start coming across singles of one artist featuring another twenty. What are you going to do about it, copy the same file to every artist's folder? Nope; use hard links!

• As there's always one artist that takes the ownership of the song, place the song in that artist's folder.
• Create hard links of the same song to every featured artist's folder.

Hands On!

Here's how to set these links up on the three main operating systems.

Windows

In Windows you can do this using the Command Line prompt without installing anything, or through the Explorer GUI after the installation of a small tool called Link Shell Extension.

If using the command line, then open an elevated Command Line prompt (that is, one with administrative privileges), input mklink, and press Enter. You'll see the help information explaining how to create hard links, junctions and directory symbolic links. For instance, you can create an hard link between two text files by calling mklink /H c:\file1.txt c:\file2.txt.

To do this via Explorer, first download and install Link Shell Extension. (This is my preferred solution as it has extended and useful functionality like volume points and overlay icons.) The workflow is really straightforward; you can see it in action and learn how to use it on the utility's web page.

Linux

In my search I didn't find a total GUI solution for Linux; if you know of one, please leave it in the comments.

If using the Terminal, you can use the ln command to create any hard links and symbolic links. For instance, try ln fooOriginal barLink. Wikibooks has a simple guide to ln, or you can read the full specification.

There is a partial GUI solution: in Ubuntu (and possibly other Linux variants), you can create symbolic links by dragging a file with the middle mouse button and choosing Link Here.

Unfortunately, I have not found any GUI application that enables hard link creation.

Mac OS X

OS X is a special case, as directory hard links are known to create all sorts of problems. Plus, there's another notion, similar the others but more advanced: an alias.

Aliases are similar to symbolic links, but are implemented in a higher level and more robust structure. They can be linked to files and folders anywhere, including networked storages, using an unique ID. One of the best features of aliases is that they keep tabs on file location changes; that is, they don't break when the original file is moved to another place. Unfortunately, some applications can have a problem with following them to the target.

On the other hand, symbolic links (or symlinks) are low level system files that contains the path of the target; they break when you move the original file, but any application that uses it will arrive at the desired place. Symlinks are great to use with predefined system or application folders, or when you want to move your user Documents or Email cache to another drive.

In the Terminal, you can use the ln command just like in Linux.

In the GUI, you can use the context menu to create an alias by right-mouse-clicking on a file or folder. Alternatively, you can create symbolic links with the app SymbolicLinker.

A third option is to write your own code or compile the code from this GitHub repository

Other Use Cases

I'm drifting further and further away from game development here, but I think it's important to show the variety of problems you can solve with this concept, so that you're well prepared to use it when it's next appropriate in your gamedev workflow!

Moving Settings Files or Key Directories

Most applications are not flexible regarding the user's Settings path, so you can use links to overcome this problem in case you want to back up the information or keep it separate and organized elsewhere.

In Linux, you could also use symbolic links to back up your home folder to a different drive:

• Copy your home folder to a different drive or partition,
• Format your system partition with a fresh installation.
• Create a symbolic link to new system home folder. (If you know bash, you could create a script that does this for you, making it easy to have fresh installations of new versions!)

In Windows, especially if you have a rather small SSD, you could use links to separate your Windows installation from your User and Program Files folders:

Or, if you dual-boot Linux and Windows, you could use links to keep the same personal files on both operating systems. This is particularly useful in scenarios where you need Email or Dropbox access in both OS but, of course, don't want your files duplicated across each.

Programming

You could create one big program with multiple functionality that relies on the application name to determine which function it will execute. (That is, the same application has multiple hard links that differ only in name.) This was actually done in old Unix systems, where mv, cp and rm ran the same executable file. If you think about it, moving a file in a worst case scenario can be a cp and rm operation and, with this method, you could share most of the code and keep assemblies storage requirements to a minimum.

While I was researching this I tried it out in C# and Windows 8.1; in my research I've found that that are many ways to get the filename of a program, but only some of those methods can retrieve the filename of a hard link.

The above image shows a test program I made to demonstrate this problem. As you can see, the filename of the first window is OriginalFileName.exe and that of the second is HardlinkedFileName.exe. The code shown is the code needed for the results shown, respectively. You can get the code for this application at this GitHub repo.

Finally, as a web developer, you might find you run into the same sort of problems as I outlined for UDK 3 development, especially if you are using LAMP, WAMP, or similar. You could code a script to create hard links and symbolic links to your site files, so that the files can be anywhere on your storage devices, but you don't have to change a thing in order to test the site out.

Conclusion

Don't expect to use this on a daily basis, but expect to feel blessed when you have a problem that is easily solved with this. It's one of those things to keep in the back of your mind because some day it will be useful to know.