Photoshop: Masking by Brush and Invert

A technique I’ve developed for masking pictures in Photoshop.

Posted on 2005-03-14 23:30 by Jørn Støylen [permalink]

I’m going to teach you a technique I’ve developed for masking pictures in Photoshop. I’m sure many others are doing something similar, but so far I haven’t seen it described anywhere online. Also, along the way, I present a few complementary tips and points of view.

This technique is what I use when all the other, more convenient ways to mask fail to produce a satisfactory result: The magic wand, the background removal tool, the magnetic lasso etc. are all good for masking objects with sharp edges and/or clean backgrounds.

1. The Task at Hand

Consider this image:

How would you go about masking that girl to produce something like this? (It's not a masterpiece, I know, but it serves nicely to illustrate my point.)

The technique I’m going to describe here is ideal for this sort of image. It works best for masking pictures containing:

I’ve always detested clipping paths, and after converting from QuarkXPress to Indesign, thankfully I will never have to use them again. They create ugly, unnaturally sharp edges in the final, printed product. They’re especially bad for hair and other fluffy things.

However, there are ways to use clipping paths that other people have made. Check back soon for an upcoming article on that.

Also, for working with very complex objects, there are a few other advantages:

So, without further ado, let’s get to business.

2. Cropping the Image

We probably want to be nice to our computer and crop the image we’re working with to the desired/required size. The smaller the image, the faster the computer will be able to work with it.

Sharpening, if used, should ideally be the last step of any basic image processing. Adjust colors, crop, retouch and do anything else first, and then, just before you save your completed image, sharpen.

Masking, however, is best done after sharpening, as you want to know exactly how those edges look. If you mask first and then sharpen, much of your masking might need refinement.

After cropping, we apply an appropriate amount of sharpening, as resampling (changing an image’s pixel dimensions) always introduces a little blur.

3. Adding the Layer Mask

To add a layer mask, we need to unlock the background layer. From Photoshop Help:

When you create a new image with a white background or a colored background, the bottommost image in the Layers palette is Background. An image can have only one background. You cannot change the stacking order of a background, its blending mode, or its opacity. However, you can convert a background to a regular layer.

The easiest way to do this is to doubleclick it in the Layers palette. Also from Photoshop Help (my screenshots):

Nontransparent background layers are locked. To change the stacking order, color, blending mode, or opacity of a locked background, you must unlock it by converting it to a standard layer.

4. Make the Brush

Photoshop has a default brush spacing value of 25%, which I find much too coarse for most of my uses, and especially for masking. As you paint with the brush, you have to move a certain distance (25% of the brush size, if I understand it correctly) before a new dab of color is added. And for the sizes we’ll use here, there’s less control than we like.

Here’s an example of 40% vs. 5% spacing:

This effect is more apparent with big brush sizes. For example, a brush size of 4 pixels will appear to move smoothly, since 25% of 4 pixels is 1 pixel, which is the minimum distance that anything pixel-based can move anyway.

So we’ll bring out the Brushes palette and set up a brush like so:

Depending on the nature and size of your image, you might want to set a different size. We’ll adjust the hardness along the way to match the hardness of the object we’re masking.

Also, since we’re going to paint with black on the initially white mask, press D to set the default white foreground and black background colors (this is the opposite of the normal painting default) and X to switch them.

Now we’re ready to …

5. Start Masking!

I find that it’s easier to get good results and less strenuous on both eyes and hand to work at a zoom level around 200 or 300 percent. So we zoom in and start painting with the brush along the edge of the object.

For straight edges, you can simply click at one spot, then press shift and click further along the edge, and Photoshop will draw a straight brush stroke between the points you clicked.

And now for the real trick of this technique! See how the outlines of her two legs meet? You might wonder how we’re going to manage to mask an inverse corner like that with the brush we’re using. What to do, what to do?

First we paint a bit further, all the way along the edge of her leg:

And then we hit Cmd-I (Ctrl-I for Windows users) to invert the mask:

Now we can see the portion of her leg that we masked. We can bring this back by painting over it while the mask is inverted:

Now we invert again Cmd-I (Ctrl-I), and see the result:

Continue masking:

Do you see the convenience of using this method? You don’t have to change your tool, brush size or anything else. You just invert the mask to unmask details you’ve masked.

Admittedly, for just this portion of this image, the lasso tool would do the job just as well. But as we go further along the outline of the girl, we get to the end of her scarf:

This is where we right-click to bring up the Brushes context menu and lower the hardness to, say, 75%. Then we mask, invert, unmask, invert, mask, invert, unmask, invert … and in just a few seconds we have not only masked the end of her scarf, but we have done so with an appropriate level of fluffiness to it, as opposed to what we’d get with any lasso tool: the same amount of edge hardness all around. And with this kind of background and outline, we’d get nowhere at all with the magic wand.

Another advantage of this method is that if we make a mistake with just parts of our brush stroke, instead of undoing, we can invert and unmask just the wrong part of it.

6. Masking Hair

It’s part of the nature of hair to let light through it, and placing this masked girl’s hair on top of a color background should let the background show through. For this kind of thing, you really need partial transparency.

For masking hair in general, a brush hardness of 50-75% should do the trick, depending on the hair in the image and the brush size, of course. Sometimes people’s hair can have a hard outline here and there, other times people have really fluffy hair that’s a nightmare to mask whichever method you use. I’ve found that you can use a soft brush and mask away a little more than you think you can, and you’ll still get results that look natural.

For this specific image, it would be a shame to just mask away the locks of hair that hang above the book, since they do so much for how the girl looks, and also does a good job of directing the eyes of the viewer from her head down to the book, accentuating what’s happening in the picture.

This requires a lot of manual work, but it doesn’t take too much time considering the rewarding result. We employ the same technique I’ve already explained (mask, invert, unmask, invert etc.), but reduce the brush size to a few pixels, and also set the opacity of the brush to something like 50-75%.

It's also possible to change the color of the transparency grid, and there's even a reasonably quick way to do it:

Hit Cmd-K/Ctrl-K to open Preferences and then hit Cmd-4/Ctrl-4 to switch to the Transparency & Gamut pane.

Sadly, you have to choose the grid colors with the mouse, or else this tip would've been SUPER FAST.

It's also a good idea to make an extra background layer in Photoshop, either containing the final background color, or just solid black, for example, to see the effects of your masking more clearly. Here's a side-by-side comparison of how the girl's hair looks on the transparency grid vs. solid black:

7. The Finished Image

So here’s the final, masked image, as viewed in Photoshop.

Comments closed

Commenting is closed for this article.