Mortgage rates have been dropping, prompting a huge surge in mortgage refinancing. But in some instances it may not pay to refinance, even if interest rates are near record lows.

After all, when you refinance your mortgage, you have to pay closing costs all over again — which could be in the thousands of dollars, depending on which state you reside in. The loan restarts at the date of the refinance, meaning the life of your loan is also extended. Not to mention that if your property is underwater, and/or you don’t have 20% equity, you may get hit with additional fees. Worse yet, you may not be eligible to refinance.

“The simplest way is to take what you are saving (in a refinancing) and divide it into how much the closing costs are,” said Bob Walters, chief economist at Quicken Loans. That will give you how many months it will take to break even. Weigh that against how long you plot to stay in your home, and you have your answer if it’s worth it to refinance. “A lot of times you will find out it takes 18 months to break even. That’s a excellent deal if you are staying longer than that,” he said.

Another option is to try to do a no-cost refinancing, which is rare these days — or you can pay for points to drive the interest rate down lower, thus reducing your monthly payment.

Currently the interest rate on a 30-year fixed mortgage is at 5.01%. People buying a home usually get a lower rate than those refinancing. Over the holidays there was a flurry of activity as consumers tried to take advantage of the decline in mortgage rates. Economist are mixed as to whether rates will fall even more but many agree that if it saves you money you should jump on it.

“You should do it now because you never know what’s going to happen in this global economy,” said Marc Savage, president of the National Association of Mortgage Brokers. “Interest rates could go up — and when they do, they go up quickly; and when they drop, they drop rather slowly.”

According to Savage, when refinancing you should aim to save $100 a month and make sure you realize enough savings to pay off the closing costs in a decent period of time. What may save you more money over the long run is keeping your monthly payment the same but chipping away at the length of your loan.

“You not only want to save something per month you want to try and knock some years off and shorten the terms. There’s real savings in that,” said Savage.

Orawin Velz, an associate vice president of economic forecasting at the Mortgage Brokers Association, said the goal in refinancing should be to reduce your interest rate by at least one percentage point. She said that while many people refinance to get a lower rate and thus a lower monthly payment, there are others with significant equity in their homes that do it to get access to cash.  “Some people are worried about their job and want to get a small cash out and place it away,” said Velz.

Most mortgage experts agree that when it comes time to refinance, it pays to do your homework and shop around. Sure — your existing relationship with your current mortgage provider may make the process a small simpler, but it doesn’t necessarily mean you will get the best rate. And don’t worry about your credit score taking a hit as a result.

Quicken’s Walters said if your making inquiries in a couple of weeks period the credit scoring agencies know you are simply shopping for a mortgage and don’t hold it against the consumers.

“If you have habitual pattern of looking for all kinds of credit, they differentiate that from someone doing legitimate shopping for a mortgage,” said Walters.