Categories

Jul 18, 2013

Amazing Trick For Refining Masks In Photoshop

The Brush Tool Gets Better


Photoshop has an impressive arsenal of tools when it comes to creating and refining masks. But there’s one little tool that’s so powerful, so almost-magical in its application that you’ll wonder why you never knew about it before. It’s the brush tool, with one small quirk: it’s the brush tool with the blend mode set to Overlay. And this one little change lets you dramatically and easily clean up your masks, even in areas with complex edges, like hair or tree branches. In this tutorial we’re going to use this technique, as well as a few other nifty ones, to combine two photos to create the image at right.

For my base image I’m using a photo I took in the incomparably beautiful Ansel Adams Wilderness in California. This is a dawn shot of Mt. Ritter and Banner Peak reflected in the still waters of Garnet Lake, and while the landscape has a certain majesty to it, the sky could benefit from a little more oomph. So let’s add some dramatic clouds which I photographed in Death Valley. It’s important to note that in both photos the light is coming from the same direction (behind the camera, and fairly low down), and that the focal length of each photo is approximately the same.

Since the sky in the Banner Peak photo is a little tight, let’s first add some empty space to it by using the crop tool to expand the canvas, then paste the clouds as a new layer on top.

Creating Your Layer Mask

Now it’s layer mask time. We could use something like the quick selection brush to select the blue sky to create a basic layer mask for our cloud layer, but you’ll notice right away that we’ll run into a problem with that method: the sky in the mountain photo, even though it’s boring blue, is actually very complex. Those trees stick up into the sky and create a lot of complexity which the quick selection tool will never handle.

Josh Cripps Refining Masks in Photoshop

Now, I hear some of you in the back saying “use the color select tool!” And we could, but I don’t really like it because it’s not a very clean selection method and tends to create blockiness and posterization, especially when you’re editing 8-bit jpegs. Instead we’re going to use Channels to make an initial selection. The trick to using Channels is to click through each channel (Red, Green, Blue) in the Channels palette and find the one that shows the greatest contrast between the thing you want to select (the blue sky in this case), and everything else (like the mountains and trees and such). In this case, the Blue Channel provides the most contrast between the sky and the rest of the landscape.

Josh Cripps Refining Masks in Photoshop

Making Selections From Channels

When you turn a channel into a selection (via the Load Channel as Selection button at the bottom of the Channels Palette) it works like this: everything that’s pure white becomes 100% selected, everything that’s pure black is 0% selected, and all the gray stuff in between is proportionately selected based on how bright it is. So if I load this blue channel as is as a selection, the sky is going to be mostly selected, and the trees aren’t going to be selected much at all, which is pretty much what we want. But the mountains are fairly light gray, which means that they’ll be somewhat selected, and that is no bueno, because that means when we make our mask for the clouds, they’re going to bleed over into the mountains.

What we need is a way to further refine the contrast in this channel to make a really good selection. Well it turns out that you can apply curves adjustments to channels, which means we can make the blue channel even more contrasty. But rather than adjust the blue channel directly (which would affect the overall color balance of the image), we first need to make a copy of the blue channel by dragging it down to the Create New Channel button at the bottom of the palette.

This will create an alpha channel titled Blue copy. We can then hit Ctrl / Cmd + m to bring up a curves adjustment. By cranking the brights up and the shadows down we can greatly enhance the contrast of this alpha channel, which means we’re one step closer toward replacing our sky.

Josh Cripps Refining Masks in Photoshop

Now even though this is a much better starting point for our mask, it’s still problematic because you can see that some tones in the mountains and in the foreground are still gray, which means they’re going to be partially selected. But let’s go ahead and create a mask for our sky anyway so we can see what we’re working with. First we want to load our Blue copy channel as a selection via the Load Channel as Selection button at the bottom of the palette. Then, head back to the layers palette and making sure the cloud layer is selected, hit the Add Layer Mask button.

Josh Cripps Refining Masks in Photoshop
Josh Cripps Refining Masks in Photoshop
Josh Cripps Refining Masks in Photoshop

Now because parts of our mountains and foreground were partially selected via our Blue copy channel, we should expect to see a funky result when we create this mask.

Refining The Mask

Yup, our image is looking a little weird. We replaced the blue sky alright, but there are bits of clouds showing through the mountains and the foreground reflection. So let’s clean up our mask a little to fix this.

First, Alt / Option click on the layer mask thumbnail so that we can see what the layer mask actually looks like. Now we’re going to use a brute-force method to clean up most of the mask. Grab the lasso tool and make a selection covering the entire bottom half of the image, and roughly paralleling the tops of the mountains, as seen in the figure below.

Josh Cripps Refining Masks in Photoshop

Now you can use Shift + F5 or Edit ? Fill to fill this selection with black.

Josh Cripps Refining Masks in Photoshop

Pretty darn good, but we’ve still got to clean up the edges of those mountains. And this, my dear friends, is where our wonderful friend Overlay mode comes in. Grab your paint brush, make sure black is your foreground color, and start painting!

Josh Cripps Refining Masks in Photoshop

Oh shoot, what did we do? Right, our paint brush is still in Normal mode. So of course it made a normal splash of black. Let’s undo that. Then in the tool bar click the drop down and select Overlay.

Josh Cripps Refining Masks in Photoshop

Now with our brush in Overlay mode, let’s try that brush stoke again.

Josh Cripps Refining Masks in Photoshop

And holy crap, did you see that? Our brush stroke now only affected the mountains, and had zero effect on the sky. ZERO. See, I told you it was like magic. So what is going on here? In Overlay mode, when you paint with something brighter than middle gray, it brightens the underlying pixels; when you paint with something darker than middle gray it darkens the underlying pixels. But the meat and potatoes of Overlay mode is this: no matter whether you’re darkening or lightening, pure white and pure black pixels are unaffected.

Since we increased the contrast of our Blue copy channel, we made all the sky pixels pure white in the mask, which means that we can paint over them as much as we want in Overlay mode and they will never be affected. Pretty amazing. Knowing this, we can now take our brush tool and paint over the rest of the mountain to create a final mask that looks like this:

Josh Cripps Refining Masks in Photoshop

Now is that a mask or what?! Look at the beauty, the complexity, the detail! All those nooks and crannies have been preserved in the mountains and even in the branches of the trees. Most excellent. If we now Alt / Option click on the layer mask thumbnail again we should see a much cleaner sky.

Josh Cripps Refining Masks in Photoshop

Now the cloud mask is razor sharp, but the color of the clouds isn’t quite right, so let’s add a curves adjustment layer to increase the red in the highlights. Then we can clip it (right click on the curves layer and select Create Clipping Mask from the drop down) to the cloud layer to make sure it affects only that layer.

Josh Cripps Refining Masks in Photoshop

Fixing the Foreground

This looks pretty great, except for one big, glaring issue: the foreground is still reflecting that lovely blue sky from the original photo. In order to fix that we’re going to take almost the exact same steps as before.

First I’m going to duplicate the cloud layer (and the clipped curves layer as well in order to preserve the color matching). Then I’ll flip the clouds vertically, pull them down to the bottom of the image, and then add a little bit of perspective distortion to match the wide-angle distortion of the original photo.

Josh Cripps Refining Masks in Photoshop
Josh Cripps Refining Masks in Photoshop

With the cloud reflection in place, I’m going to add a layer mask to it and fill it with black (this can be done automagically by holding Alt / Option and clicking on the Add Layer Mask button in the Layers palette) in order to completely conceal the clouds.

Working With Channels

With our clouds masked out completely we need to paint them back in selectively into the reflection. Again I’m going to use my Channels to create a selection, and again I’m looking for a channel where the contrast between the blue sky reflection and everything else is the greatest. And in this case, the Blue copy channel we made before is an outstanding candidate because of its high contrast. But what makes this channel really excellent is that the rocks in the foreground fade from black to white. Which means that when we create a mask using this channel, the clouds will fade out gradually into the rocks, which will help preserve the transparency of the water.

Josh Cripps Refining Masks in Photoshop

This time we’re going to do things a little differently, just to show you another technique. Instead of creating a mask from this channel, then refining the mask using Overlay brushing, this time we’re going to first paint directly on the channel using Overlay mode, then create the mask afterward. But I don’t want to change my Blue copy channel in case I need it again later, so I’m first going to make a copy of that channel, which Photoshop calls Blue copy 2.

Now with that Blue copy 2 channel highlighted in the Channels palette, I’m going to take my Overlay brush and darken all the mountains, trees, and rocks in the midground. I’m going to leave the foreground rocks alone, and I also don’t care about the rocks sticking up out of the water. After all, this mask is going to be for the reflection only, so I don’t care what’s happening outside of that reflection.

Josh Cripps Refining Masks in Photoshop

Ok, this is looking really good, so all that remains is to turn this channel into a mask. Just as before I’m going to click the Load Channel as Selection button in the Channels palette, then head back to my Layers palette. But since we already created a layer mask for our reflection and filled it with black, we can now take a white brush (be sure to switch your brush’s blend mode back to Normal for this) and paint in the areas where we want the clouds to show up. Creating the mask this way gives us even more subtle and selective control over how we want the clouds to appear in the reflection. Here I used a large, soft brush with an opacity of about 30% to gradually paint the clouds into the reflection.

Josh Cripps Refining Masks in Photoshop

And here’s what the image looks like now with the reflective clouds masked in. Note how the water still looks transparent even though we completely replaced the sky reflection. How sweet is that?!

Josh Cripps Refining Masks in Photoshop

Correcting Your Color

Because I didn’t paint the clouds in with 100% opacity, there’s still a little blue leaking through from the underlying layer. But that’s an easy enough fix. In our curves layer above the clouds, select the blue channel and drag it down a little bit, and that’ll pull any excess blue out of the reflection.

Josh Cripps Refining Masks in Photoshop

Now we could stop there and be totally happy with this image, but since we’ve gone through all the trouble of replacing the sky and the reflection, why not add a little more pop and color as well? A quick, easy, and fun way to add color and contrast to your images is with a gradient map. Grab a gradient map adjustment layer and in the properties window, choose a couple of colors. Here I decided to have the gradient go from purple to light yellow. You can add as many colors as you like and change each one, so fiddle around till you get something you like.

Josh Cripps Refining Masks in Photoshop

Now change the blend mode of the gradient map to Soft Light. This will boost the contrast of your image and lessen the impact of the colors.

Josh Cripps Refining Masks in Photoshop

Final Result

The result of that is an image with lots of pop, and a groovy-looking color scheme. And voila, we’re done!

Josh Cripps Refining Masks in Photoshop

Alrighty, guys and gals, thanks for taking the time to read this tutorial, and hope you enjoyed it! Until next time, happy shooting.

22 Comments


user image You
(will not be published)

    • user image
      Guest

      Gawd I’m an idiot! I just re-read the tutorial and you did say that you duplicated the cloud layer AND the Curves layer.

  • user image
    Marco Ruschkowski

    Hm, never thought of using the brush tool for that. Nice idea! I basically do the same using the dodge and burn tools.
    I´m always stunned by your colour work.

    • user image
      Josh Cripps

      Hey Jon, this is Josh, the article author. Yeah, this is a bit complicated, I know! A video would definitely help explain the technique, so maybe Aaron will do a free episode on this. To get to the Channels Palette go to Window -> Channels. Good luck!

      • user image
        Jon Peckham

        Thanks Josh. I did find it, but the tutorial is still confusing me. Im a little slow and just now learning the more intricate details of PS CC. Even layer masks are taking a while
        for me to grasp. Perhaps a video would help? I love the outcome of this tutorial though. I have always wanted to do this. . .

  • user image
    Arturo Yanez

    Josh, thank you very much for the tutorial. The technique is impeccable but I see a problem with the light in the mountains and the stormy, dramatic sky. These two do not go together. Does it make any sense? You cannot have both. The clouds are very tight and do not allow any light to fall through the openings.

  • user image
    True Blue

    Thanks for a great tutorial Josh. I’m a bit confused about duplicating the sky image to create the reflection. When I duplicate the image, the already clipped Curves layer does not duplicate with it. I had to duplicate the Curves layer and clip it to the duplicated and flipped cloud image once I’d finished masking it. That’s no big deal but how do you get the Curves layer to duplicate with the layer it’s clipped to?
    Cheers.

    ETA: I just re-read the tutorial again and you did say to duplicate the cloud layer AND the Curves layer. Sorry about that. I thought there was some secret way of duplicating both layers in one easy click.

  • user image
    Tim Lewis

    i like how there’s no obvious light source for the mountain…. makes it seem like the mountain is illuminated from within, which i dig.

  • user image
    Gursimran Kaur

    Thank you for the very useful tutorial! The results look absolutely stunning and you make it all sound so effortless. Thanks much much!

  • user image
    TomaszProkop

    I think that attaching original files for this edit would be useful for everyone who wants to go through the tutorial to achieve similar results. Please consider it for future posts.

  • user image
    Marcelo Bass

    Excellent tutorial: even when english is not my “lingua mater”, the descriptions and explanations are quite clear. I have heard somewhere about the pen brush in overlay blending mode, to refine masks. However, those indications was -for me- difficult to understand. Now I´m enjoining the technique.
    Thanks Thanks Thanks

  • user image
    Kirk Shultz

    I live just a few miles from from here (as the crow flies) and was just to the south of here a couple weeks ago climbing Clyde minaret. It’s great to see some tips on how to work on nature photography. I can apply what is taught about fashion easily enough, but it’s really cool to see what I can do with photos like my own.

  • user image
    Diogo Costa

    Hi there,
    I have a few problem. I need to remove a white background from a dish, the problem is the dish it’s white and I can’t remove the background without erase the dish or make a very nasty selection. Can someone help me with this problem?

    Thank you

  • user image
    marc.labro

    super tuto !!! in my case i often toggle between brush in overlay mode and dodge/burn tools when the contrast is not enough.
    i am a fan of on1 perfect effect. in th past they had perfect mask with drop/keep colors… they have now improved their brush to paint around trees,… and make “clever” sky removing but in my case it often eats mountains,… and impossible to come back, no other masking tool to refine. the old channel method in photoshop is less moderns but works with everything : mountains, hairs,…
    best regards
    marc